JUnit support¶
Nextest can produce output in the JUnit/XUnit XML format. This format is widely understood by test analysis tools and libraries.
To enable JUnit support, add this to your nextest configuration:
Basic JUnit configuration in .config/nextest.toml
[profile.ci.junit] # this can be some other profile, too
path = "junit.xml"
If --profile ci is selected on the command line, a JUnit report will be written out to target/nextest/ci/junit.xml within the workspace root.
Some notes about the JUnit support:
- There are several slightly different formats all called "JUnit" or "XUnit". Nextest adheres to the Jenkins XML format.
- Every test binary forms a single
<testsuite>. Every test forms a single<testcase>. - Standard output and standard error are included for failed and retried tests. (However, invalid XML characters are stripped out.)
Configuration¶
Configuration options supported for JUnit reports, within the junit section:
report-name- The name of the report. Defaults to
"nextest-run". store-success-output- Whether to store output for successful tests in the
<system-out>and<system-err>elements. Defaults to false. store-failure-output- Whether to store output for failing tests in the
<system-out>and<system-err>elements. Defaults to true. report-skipped0.9.143-
Which skipped tests to emit as
<testcase>elements with a<skipped>child."none"(the default) does not emit any skipped tests."ignored"emits only those skipped tests whose ignore status did not match the run's run-ignored selection.- In a default run, emits the tests marked
#[ignore]. - Under
--run-ignored only, emits the tests that are not marked#[ignore].
- In a default run, emits the tests marked
"all"emits all skipped tests except tests skipped because they are not benchmarks. Note that with partitioned runs,"all"causes each partition's report to include the tests skipped in that partition, so a naive merging of reports will produce duplicate skipped entries.
flaky-fail-status0.9.131- Controls how flaky-fail tests are reported.
"failure"(the default) reports them with<failure>and<flakyFailure>/<flakyError>elements."success"reports them as successes, identical to flaky-pass tests. This setting only affects the JUnit representation; the runner will still mark flaky-fail tests as failed.
All of these settings can also be configured on a per-test basis.
Example configuration¶
Per-test JUnit settings
[profile.default.junit]
path = "junit.xml"
# These are the default values, specified for clarity.
store-success-output = false
store-failure-output = true
report-skipped = "none"
flaky-fail-status = "failure"
[[profile.default.overrides]]
filter = 'test(important-test)'
junit.store-success-output = true
[[profile.default.overrides]]
filter = 'test(known_flaky)'
junit.flaky-fail-status = "success"
In this example, the JUnit report will contain the output for all failing tests, and for successful tests that contain "important-test" in the name. Tests matching "known_flaky" that are configured with flaky-result = "fail" will appear as successes in the JUnit report despite being treated as failures by the runner.
Post-processing¶
Some tools that read JUnit files don't follow the Jenkins standard. You can post-process the JUnit file in such cases. Here's some recommendations for post-processing tools written by community members:
- CircleCI:
circleci-junit-fix
Example¶
Here's an example JUnit file generated by cargo-nextest.
<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="nextest-run" tests="3" failures="1" errors="0" uuid="45c50042-482e-477e-88a2-60cfcc3eaf95" timestamp="2024-01-09T07:50:12.664+00:00" time="0.023">
<testsuite name="fixture-project::basic" tests="3" disabled="0" errors="0" failures="1">
<testcase name="test_cwd" classname="fixture-project::basic" timestamp="2024-01-09T07:50:12.665+00:00" time="0.004">
</testcase>
<testcase name="test_failure_assert" classname="fixture-project::basic" timestamp="2024-01-09T07:50:12.665+00:00" time="0.004">
<failure type="test failure">thread 'test_failure_assert' panicked at tests/basic.rs:19:5:
assertion `left == right` failed: this is an assertion
left: 4
right: 5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace</failure>
<rerunFailure timestamp="2024-01-09T07:50:12.670+00:00" time="0.004" type="test failure">thread 'test_failure_assert' panicked at tests/basic.rs:19:5:
assertion `left == right` failed: this is an assertion
left: 4
right: 5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
<system-out>
running 1 test
test test_failure_assert ... FAILED
failures:
failures:
test_failure_assert
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 17 filtered out; finished in 0.00s
</system-out>
<system-err>thread 'test_failure_assert' panicked at tests/basic.rs:19:5:
assertion `left == right` failed: this is an assertion
left: 4
right: 5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
</system-err>
</rerunFailure>
<rerunFailure timestamp="2024-01-09T07:50:12.676+00:00" time="0.004" type="test failure">thread 'test_failure_assert' panicked at tests/basic.rs:19:5:
assertion `left == right` failed: this is an assertion
left: 4
right: 5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
<system-out>
running 1 test
test test_failure_assert ... FAILED
failures:
failures:
test_failure_assert
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 17 filtered out; finished in 0.00s
</system-out>
<system-err>thread 'test_failure_assert' panicked at tests/basic.rs:19:5:
assertion `left == right` failed: this is an assertion
left: 4
right: 5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
</system-err>
</rerunFailure>
<system-out>
running 1 test
test test_failure_assert ... FAILED
failures:
failures:
test_failure_assert
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 17 filtered out; finished in 0.00s
</system-out>
<system-err>thread 'test_failure_assert' panicked at tests/basic.rs:19:5:
assertion `left == right` failed: this is an assertion
left: 4
right: 5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
</system-err>
</testcase>
<testcase name="test_flaky_mod_4" classname="fixture-project::basic" timestamp="2024-01-09T07:50:12.683+00:00" time="0.004">
<flakyFailure timestamp="2024-01-09T07:50:12.665+00:00" time="0.004" type="test failure">thread 'test_flaky_mod_4' panicked at tests/basic.rs:43:9:
Failed because attempt 1 % 4 != 0
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
<system-out>
running 1 test
test test_flaky_mod_4 ... FAILED
failures:
failures:
test_flaky_mod_4
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 17 filtered out; finished in 0.00s
</system-out>
<system-err>thread 'test_flaky_mod_4' panicked at tests/basic.rs:43:9:
Failed because attempt 1 % 4 != 0
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
</system-err>
</flakyFailure>
<flakyFailure timestamp="2024-01-09T07:50:12.671+00:00" time="0.004" type="test failure">thread 'test_flaky_mod_4' panicked at tests/basic.rs:43:9:
Failed because attempt 2 % 4 != 0
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
<system-out>
running 1 test
test test_flaky_mod_4 ... FAILED
failures:
failures:
test_flaky_mod_4
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 17 filtered out; finished in 0.00s
</system-out>
<system-err>thread 'test_flaky_mod_4' panicked at tests/basic.rs:43:9:
Failed because attempt 2 % 4 != 0
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
</system-err>
</flakyFailure>
<flakyFailure timestamp="2024-01-09T07:50:12.676+00:00" time="0.005" type="test failure">thread 'test_flaky_mod_4' panicked at tests/basic.rs:43:9:
Failed because attempt 3 % 4 != 0
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
<system-out>
running 1 test
test test_flaky_mod_4 ... FAILED
failures:
failures:
test_flaky_mod_4
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 17 filtered out; finished in 0.00s
</system-out>
<system-err>thread 'test_flaky_mod_4' panicked at tests/basic.rs:43:9:
Failed because attempt 3 % 4 != 0
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
</system-err>
</flakyFailure>
</testcase>
</testsuite>
</testsuites>