Skip to content

Signal handling

Nextest's signal handling uses Tokio's native support for signals. Signals are received by a multiplexer called SignalHandler, which generates a stream of events. The runner loop's dispatcher then selects over this stream. On receiving a signal, the dispatcher is responsible for broadcasting a message to all units.

Here's a simple sequence diagram of how signals are handled:

sequenceDiagram
  autonumber
  participant tokio as Tokio
  participant muxer as signal muxer
  participant dispatcher as dispatcher
  muxer->>tokio: install signal handlers
  tokio->>+muxer: stream signals
  muxer->>-dispatcher: stream event
  activate dispatcher
  dispatcher->>+units: broadcast signal
  units-->>-dispatcher: respond if needed
  deactivate dispatcher

Signal handling on Unix

Unix platforms have a rich set of signals, and nextest interacts with them in a few different ways.

Process groups

On Unix platforms, nextest creates a new process group for each unit of work. Process groups do not form a tree; this directly impacts signal handling in interactive use.

  • For example, when the user presses Ctrl-C in a terminal, the terminal sends SIGINT to the foreground process group.
  • If nextest did not create a new process group for each unit, the terminal would have sent SIGINT to both nextest and all unit processes.
  • Since nextest creates a new process group for each unit, the terminal only sends SIGINT to nextest. It is then nextest's responsibility to send the signal to all child process groups.

For more about process groups, see Rain's RustConf 2023 talk: Beyond Ctrl-C: the dark corners of Unix signal handling. There's an edited transcript of the talk on Rain's blog. (Rain is nextest's primary author and maintainer.)

Shutdown signal handling

On receiving a signal like SIGINT or SIGTERM, the dispatcher sends a message to all units to terminate with the same signal. Units then follow a flow similar to timeouts, where they have a grace period to exit cleanly before being killed.

Currently, nextest sends SIGINT on receiving SIGINT, SIGTERM on receiving SIGTERM, and so on. There is no inherent reason the two have to be the same, other than a general expectation of "abstraction coherence" (a kind of least astonishment): if you set up process groups, you should behave similarly to the world in which you don't set up process groups. This is a good principle to follow, but it's not a hard requirement.

Job control

On Unix platforms, nextest supports job control via the SIGTSTP and SIGCONT signals.

SIGTSTP vs SIGSTOP

There are two closely-related signals in Unix: SIGTSTP and SIGSTOP. The main difference between the two is that SIGTSTP can be caught and handled by a process (as it is by nextest), while SIGSTOP cannot.

When the user presses Ctrl-Z in a terminal, the signal that's sent is SIGTSTP, not SIGSTOP. We are lucky that is the case, because it gives nextest an opportunity to do some bookkeeping before stopping itself.

When a SIGTSTP is received, the dispatcher sends a message to all units to stop. At that point, units do two things:

  • Send SIGTSTP to their associated process groups. Since nextest creates a separate process group for each test, it must forward the signal to the process group of the test.
  • Pause all running timers. This is very important! If the run is paused in the middle of a test being executed, the time spent in the pause window must not be included in the test's total run time. This is particularly important for timeouts imposed by nextest.

Similarly, when a SIGCONT is received, the dispatcher sends a message to all units to resume. Units then send SIGCONT to their associated process groups, and resume all paused timers.

Pausable timers

Tokio itself doesn't have great support for pausing timers, other than a global pause in single-threaded runtimes that's mostly geared towards #[tokio::test]. So nextest manages its own pausable timers. This is done via two structs inside the nextest_runner::time module:

  • Stopwatch to measure time. A Stopwatch is a pair of (start time, instant) which also tracks a pause state, along with a duration accounting for previous pauses.
  • PausableSleep to wait until a duration has elapsed. PausableSleep is a wrapper around a Tokio sleep which also tracks a pause state.

Each step in the executor is responsible for calling pause and resume on all timers it has access to. Unfortunately, we haven't quite figured out an automatic way of ensuring that everything that should be paused is paused, so this process requires manual review.

Both of these types are useful in general, and could be extracted into a library if there's interest.

Double-spawning processes

On Unix platforms, when spawning a child process, nextest does not directly spawn the child. Instead, it spawns a copy of itself, which then spawns the process using exec.

(Note: This is done by calling posix_spawn on the current cargo-nextest executable with a hidden __double-spawn command, not via the fork system call. fork is quite messy at best and actively dangerous at worst, and is best to be avoided.)

This double-spawn approach works around a gnarly race with SIGTSTP handling. If a child process receives SIGTSTP at exactly the wrong time (a window of around 5ms on 2022-era hardware under load), it can get stuck in a "half-born" paused state, and the parent process can get stuck in an uninterruptible sleep state waiting for the child to finish spawning.

This bug is universal

This race exists with all posix_spawn invocations on Linux, and likely on most other Unix platforms. It appears to be a flaw with the posix_spawn specification, and any implementations that don't specifically account for this issue are likely to have this bug.

With nextest, users just hit it more often because nextest spawns a lot of processes very quickly.

You can reproduce the issue yourself by setting NEXTEST_DOUBLE_SPAWN=0, then running cargo nextest run -j64 against a repository; the clap repository is a good candidate because it has many small tests. Now try hitting Ctrl-Z a few times, running fg to resume nextest after each pause. You'll likely see one run in ten or so where nextest gets stuck.

With the double-spawn approach:

  • Nextest first uses a signal mask to temporarily block SIGTSTP.
  • It then spawns a copy of itself, which inherits the signal mask.
  • At this point, the spawn is complete, and the parent process can carry on.

  • The child copy then unblocks SIGTSTP.

    A queued up SIGTSTP may be received at this point. If that is so, the process is paused. But, importantly, the parent does not get stuck waiting for the child to finish spawning.

  • Finally, the spawned child uses Command::exec to replace itself with the test or script process.

The double-spawn approach completely addresses this race, and no instances of a stuck runner have been observed since it was implemented.

Signal handling on Windows

Windows has a much simpler signal model than Unix.

  • For console applications, Windows supports pressing Ctrl-C, which resembles SIGINT.
  • Windows also has the notion of console process groups. However, the way in which they work is quite different from Unix process groups. For example, Ctrl-C events cannot be limited to a single process group.
  • For graphical applications, Windows supports the WM_CLOSE message.

In many ways, the WM_CLOSE message is more reasonable than Unix signals, because it's a message that can be handled or ignored, and ties nicely into an event-driven model such as that of nextest. It is geared towards GUI applications, but can be used in console applications as well by creating a hidden window. See this StackOverflow post for more.

Currently, nextest does not put tests into separate process groups on Windows, nor does it send WM_CLOSE messages. This is an area where nextest could benefit from Windows expertise.

Job objects

On Windows, nextest uses job objects to manage the lifetime of all child processes. Job objects don't support graceful termination like Unix signals do. The only method available is TerminateJobObject, which is equivalent to a Unix SIGKILL.

Unlike process groups, job objects form a tree. If something else runs nextest within a job object and then calls TerminateJobObject, both nextest and all its child processes are terminated.

Last major revision: 2024-12-06