JUnit support¶
Nextest can produce output in the JUnit/XUnit XML format. This format is widely understood by test analysis tools and libraries.
To enable JUnit support, add this to your nextest configuration:
Basic JUnit configuration in <code>.config/nextest.toml</code>
[profile.ci.junit] # this can be some other profile, too
path = "junit.xml"
If --profile ci is selected on the command line, a JUnit report will be written out to target/nextest/ci/junit.xml within the workspace root.
Some notes about the JUnit support:
- There are several slightly different formats all called "JUnit" or "XUnit". Nextest adheres to the Jenkins XML format.
- Every test binary forms a single
<testsuite>. Every test forms a single<testcase>. - Standard output and standard error are included for failed and retried tests. (However, invalid XML characters are stripped out.)
Configuration¶
Configuration options supported for JUnit reports, within the junit section:
report-name- The name of the report. Defaults to
"nextest-run". store-success-output- Whether to store output for successful tests in the
<system-out>and<system-err>elements. Defaults to false. store-failure-output- Whether to store output for failing tests in the
<system-out>and<system-err>elements. Defaults to true. report-skipped0.9.141- Which skipped tests to emit as
<testcase>elements with a<skipped>child.*
"none"(the default) does not emit any skipped tests. *"ignored"emits only tests whose ignore status did not match the run's run-ignored selection (in a default run, the tests marked#[ignore]; under--run-ignored only, the tests that are not marked#[ignore]). *"all"emits all skipped tests except tests skipped because they are not benchmarks. Note that with partitioned runs,"all"causes each partition's report to include the tests skipped in that partition, so a naive merging of reports will produce duplicate skipped entries. flaky-fail-status0.9.131- Controls how flaky-fail tests are reported.
"failure"(the default) reports them with<failure>and<flakyFailure>/<flakyError>elements."success"reports them as successes, identical to flaky-pass tests. This setting only affects the JUnit representation; the runner will still mark flaky-fail tests as failed.
All of these settings can also be configured on a per-test basis.
Example configuration¶
Per-test JUnit settings
[profile.default.junit]
path = "junit.xml"
# These are the default values, specified for clarity.
store-success-output = false
store-failure-output = true
report-skipped = "none"
flaky-fail-status = "failure"
[[profile.default.overrides]]
filter = 'test(important-test)'
junit.store-success-output = true
[[profile.default.overrides]]
filter = 'test(known_flaky)'
junit.flaky-fail-status = "success"
In this example, the JUnit report will contain the output for all failing tests, and for successful tests that contain "important-test" in the name. Tests matching "known_flaky" that are configured with flaky-result = "fail" will appear as successes in the JUnit report despite being treated as failures by the runner.
Post-processing¶
Some tools that read JUnit files don't follow the Jenkins standard. You can post-process the JUnit file in such cases. Here's some recommendations for post-processing tools written by community members:
- CircleCI:
circleci-junit-fix
Example¶
Here's an example JUnit file generated by cargo-nextest.
<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="nextest-run" tests="3" failures="1" errors="0" uuid="45c50042-482e-477e-88a2-60cfcc3eaf95" timestamp="2024-01-09T07:50:12.664+00:00" time="0.023">
<testsuite name="fixture-project::basic" tests="3" disabled="0" errors="0" failures="1">
<testcase name="test_cwd" classname="fixture-project::basic" timestamp="2024-01-09T07:50:12.665+00:00" time="0.004">
</testcase>
<testcase name="test_failure_assert" classname="fixture-project::basic" timestamp="2024-01-09T07:50:12.665+00:00" time="0.004">
<failure type="test failure">thread 'test_failure_assert' panicked at tests/basic.rs:19:5:
assertion `left == right` failed: this is an assertion
left: 4
right: 5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace</failure>
<rerunFailure timestamp="2024-01-09T07:50:12.670+00:00" time="0.004" type="test failure">thread 'test_failure_assert' panicked at tests/basic.rs:19:5:
assertion `left == right` failed: this is an assertion
left: 4
right: 5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
<system-out>
running 1 test
test test_failure_assert ... FAILED
failures:
failures:
test_failure_assert
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 17 filtered out; finished in 0.00s
</system-out>
<system-err>thread 'test_failure_assert' panicked at tests/basic.rs:19:5:
assertion `left == right` failed: this is an assertion
left: 4
right: 5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
</system-err>
</rerunFailure>
<rerunFailure timestamp="2024-01-09T07:50:12.676+00:00" time="0.004" type="test failure">thread 'test_failure_assert' panicked at tests/basic.rs:19:5:
assertion `left == right` failed: this is an assertion
left: 4
right: 5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
<system-out>
running 1 test
test test_failure_assert ... FAILED
failures:
failures:
test_failure_assert
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 17 filtered out; finished in 0.00s
</system-out>
<system-err>thread 'test_failure_assert' panicked at tests/basic.rs:19:5:
assertion `left == right` failed: this is an assertion
left: 4
right: 5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
</system-err>
</rerunFailure>
<system-out>
running 1 test
test test_failure_assert ... FAILED
failures:
failures:
test_failure_assert
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 17 filtered out; finished in 0.00s
</system-out>
<system-err>thread 'test_failure_assert' panicked at tests/basic.rs:19:5:
assertion `left == right` failed: this is an assertion
left: 4
right: 5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
</system-err>
</testcase>
<testcase name="test_flaky_mod_4" classname="fixture-project::basic" timestamp="2024-01-09T07:50:12.683+00:00" time="0.004">
<flakyFailure timestamp="2024-01-09T07:50:12.665+00:00" time="0.004" type="test failure">thread 'test_flaky_mod_4' panicked at tests/basic.rs:43:9:
Failed because attempt 1 % 4 != 0
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
<system-out>
running 1 test
test test_flaky_mod_4 ... FAILED
failures:
failures:
test_flaky_mod_4
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 17 filtered out; finished in 0.00s
</system-out>
<system-err>thread 'test_flaky_mod_4' panicked at tests/basic.rs:43:9:
Failed because attempt 1 % 4 != 0
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
</system-err>
</flakyFailure>
<flakyFailure timestamp="2024-01-09T07:50:12.671+00:00" time="0.004" type="test failure">thread 'test_flaky_mod_4' panicked at tests/basic.rs:43:9:
Failed because attempt 2 % 4 != 0
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
<system-out>
running 1 test
test test_flaky_mod_4 ... FAILED
failures:
failures:
test_flaky_mod_4
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 17 filtered out; finished in 0.00s
</system-out>
<system-err>thread 'test_flaky_mod_4' panicked at tests/basic.rs:43:9:
Failed because attempt 2 % 4 != 0
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
</system-err>
</flakyFailure>
<flakyFailure timestamp="2024-01-09T07:50:12.676+00:00" time="0.005" type="test failure">thread 'test_flaky_mod_4' panicked at tests/basic.rs:43:9:
Failed because attempt 3 % 4 != 0
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
<system-out>
running 1 test
test test_flaky_mod_4 ... FAILED
failures:
failures:
test_flaky_mod_4
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 17 filtered out; finished in 0.00s
</system-out>
<system-err>thread 'test_flaky_mod_4' panicked at tests/basic.rs:43:9:
Failed because attempt 3 % 4 != 0
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
</system-err>
</flakyFailure>
</testcase>
</testsuite>
</testsuites>