--- title: "STF Testing Frameworks" output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{STF Testing Frameworks} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r setup, include=FALSE} knitr::opts_chunk$set(echo = TRUE) ``` ```{r load, echo=FALSE} library(test.assessr) ``` The `test.assessr` package checks package structure and test drivers to detect and classify testing configurations. If it detects a standard testing framework (STF) (i.e. `testthat` or `testit`), it runs a standard testing workflow involving a number of steps: `Step 1` - run `callr::r_safe` which calls `covr::package_coverage`. - The `r_safe` function has the parameter `error = "error"`. - This means if `package_coverage` encounters an error, then `r_safe` returns an error to the main testing process. - If `Step 1` is successful, then a unit test data coverage object like this is produced: `Step 1 unit test data coverage object` | R object | field | element 1 | element 2 | value | | -------- | ----- | --------- | --------- | ------| | test_package |pkg_name | | | "test.package.0001" | | | pkg_ver | | | "0.1.0" | | | date_time | | | "2026-03-09 14:27:36.673915" | | | executor | | | "xxxxxxx" | | | sysname | | | "Linux" | | | version | | | "#1 SMP PREEMPT_DYNAMIC Mon Dec 1 05:36:50 UTC 2025" | | | release | | | "6.1.158-180.294.amzn2023.x86_64" | | | machine | | | "x86_64" | | | r_version | | |"4.4.1" | | | test_framework_type | | | standard testing framework | | | total_cov | | | 1 | | | res_cov | name| |"test.package.0001" | | | | coverage | filecoverage | R/myscript.R 100 | | | | | totalcoverage | 100 | | | errors | | |NA | | | notes | | |NA | `Step 2` - run `setup_covr_env` which - loads helper files from `tests/testthat`, - loads package namespace objects into `cov_env`, - resolves the project root, - loads package datasets into `cov_env`, - loads Base R datasets and exports into `cov_env`, - optionally loads `nycflights13` datasets if available, - loads dependencies into `cov_env`, and - unloads the package namespace afterward to reduce conflicts. - run `create fail reporter` to find failing tests - run `create test reporter` to create a reporter for all tests - map `exported functions` to test files - create a filter for test status i.e. tests that have `"FAILED", "ERROR", "SKIPPED"` - identify tests to be skipped based on status - comment out tests to be skipped - run `covr::environment_coverage` on remaining test files. - If `Step 2` is successful, then a unit test data coverage object like this is produced: `Step 2 unit test data coverage object` | R object | field | element 1 | element 2 | value | | -------- | ----- | --------- | --------- | ------| | test_package |pkg_name | | | "test.package.0003" | | | pkg_ver | | | "0.1.0" | | | date_time | | | "2026-04-09 10:17:27.71564" | | | executor | | | "xxxxxxx" | | | sysname | | | "Linux" | | | version | | | "#1 SMP PREEMPT_DYNAMIC Mon Dec 1 05:36:50 UTC 2025" | | | release | | | "6.1.158-180.294.amzn2023.x86_64" | | | machine | | | "x86_64" | | | r_version | | |"4.4.1" | | | test_framework_type | | | standard testing framework | | | total_cov | | | 0 | | | res_cov | name| |"test.package.0003" | | | | coverage | filecoverage | num [1(1d)] 0 | | | | | attr(*, "dimnames")=List of 1 chr "myscript.R" | | | | | totalcoverage | 0 | | | errors | | |NA | | | notes | | |NA | | long_summary | file | | | "test-myscript.R#L2_L2" | | | code_script | | | "myscript" | | | context | | | "this works" | | | status | | | "FAILED" | | | n | | | 1 | | | time | | | 0.291 | | test_skip | file | | | "test-myscript.R" | | | code_script | | | "myscript" | | | context | | | "this works" | | | status | | | "FAILED" | | | n | | | 1 | | | time | | | 0.291 | | | expectation: | | | "expect_equal" | | | line1 | | | 2 | | | line2 | | | 2 | This data object differs from `Step 1 data object` in that it gives information about which tests/test blocks were skipped and the status tells the user why they were skipped. `Step 3` - run `withr::with_dir` which calls `covr::package_coverage`. - `covr::package_coverage` has the parameters `type = "tests", quiet = TRUE` - If `Step 3` is successful, then a unit test data coverage object like this is produced: `Step 3 unit test data coverage object` | R object | field | element 1 | element 2 | value | | -------- | ----- | --------- | --------- | ------| | test_package |pkg_name | | | "dplyr" | | | pkg_ver | | | "1.1.4" | | | date_time | | | "2026-04-09 10:17:27.71564" | | | executor | | | "xxxxxxx" | | | sysname | | | "Linux" | | | version | | | "#1 SMP PREEMPT_DYNAMIC Mon Dec 1 05:36:50 UTC 2025" | | | release | | | "6.1.158-180.294.amzn2023.x86_64" | | | machine | | | "x86_64" | | | r_version | | |"4.4.1" | | | test_framework_type | | | standard testing framework | | | total_cov | | | 0 | | | res_cov | name| |"dplyr" | | | | coverage | filecoverage | num [1:109(1d)] 98.3 76.7 97.5 100 100 ... | | | | | attr(*, "dimnames")=List of 1 [1:109] "across.R" "all-equal.R" "arrange.R" "bind-cols.R" ... | | | | | totalcoverage | 0.907 | | | errors | | |NA | | | notes | | |NA | | long_summary | file | | | [1:3432] "test-across.R#L7_L7" "test-across.R#L14_L14" "test-across.R#L20_L20" "test-across.R#L24_L24" ... | | | code_script | | | [1:3432] "across" "across" "across" "across" ... | | | context | | | [1:3432] "across() works on one column data.frame" "across() does not select grouping variables" "across() corre" | | | status | | | [1:3432] "PASS" "PASS" "PASS" "PASS" ... | | | n | | | [1:3432] 1 1 1 1 1 1 1 1 1 1 ... | | | time | | | [1:3432] 0.041 0.014 0.077 0.077 0.077 ...| | test_skip | file | | | [1:448] "test-across.R" "test-across.R" "test-across.R" "test-across.R" ... | | | code_script | | | [1:448] "across" "across" "across" "across" ... | | | context | | | "across() gives meaningful messages" "across() gives meaningful messages" "across() gives meaningful mes" | | | status | | | [1:448] "FAILED" "FAILED" "FAILED" "FAILED" ... | | | n | | | [1:448] 1 1 1 1 1 1 1 1 1 1 ... | | | time | | | [1:448] 2.03 2.03 2.03 2.03 2.03 ... | | | expectation| | | [1:448] "expect_snapshot" "expect_error" "expect_error" "expect_error" ... | | | line1 | | | [1:448] 209 212 216 222 226 230 234 238 243 244 ... | | | line2 | | | [1:448] 280 215 219 225 229 233 237 241 243 244 ... | Both `Step 2 unit test data coverage object` and `Step 3 unit test data coverage object` give the user details about the status of the tests `FAILED`, `PASSED` and `SKIPPED`. They also contain the test block start line (i.e. `line1` and `line2`) and the expectation type in the `expectation` column (e.g. `expect_snapshot` `expect_error`).