Criterion benchmarks

Nextest supports running benchmarks in "test mode" with Criterion.rs.

What is test mode?

Many benchmarks depend on the system that's running them being quiescent. In other words, while benchmarks are being run there shouldn't be any other user or system activity. This can make benchmarks hard or even unsuitable to run in CI systems like GitHub Actions, where the capabilities of individual runners vary or are too noisy to produce useful results.

However, it can still be good to verify in CI that benchmarks compile correctly, and don't panic when run. To support this use case, libraries like Criterion support running benchmarks in "test mode".

For criterion and nextest, benchmarks are run with the following settings:

  • With the test Cargo profile. This is typically the same as the dev profile, and can be overridden with --cargo-profile.
  • With one iteration of the benchmark.

Requirements

  • Criterion 0.5.0 or above; previous versions are not compatible with nextest.
  • Any recent version of cargo-nextest.

Running benchmarks

By default, cargo nextest run does not include benchmarks as part of the test run. (This matches cargo test.)

To include benchmarks in your test run, use cargo nextest run --all-targets.

This will produce output that looks like:

% cargo nextest run --all-targets
    Finished test [unoptimized + debuginfo] target(s) in 0.05s
    Starting 7 tests across 1 binaries
        PASS [   0.368s] my-benchmarks::bench/my_bench depends_on_cache
        PASS [   0.404s] my-benchmarks::bench/my_bench depends_on
        PASS [   0.443s] my-benchmarks::bench/my_bench into_ids
        PASS [   0.520s] my-benchmarks::bench/my_bench make_graph
        PASS [   0.546s] my-benchmarks::bench/my_bench resolve_package
        PASS [   0.588s] my-benchmarks::bench/my_bench make_cycles
        PASS [   0.625s] my-benchmarks::bench/my_bench make_package_name
------------
     Summary [   0.626s] 7 tests run: 7 passed, 0 skipped

To run just benchmarks in test mode, use cargo nextest run --benches.