---
Id: 50fbf663-cbee-4a7b-86bd-7fae7a050552
Urls: []
Language: en
---
# Benchmarking .NET test frameworks

Compares xUnit.net v3, NUnit, MSTest and TUnit on execution time, compilation time, Native AOT, and the
cost of the usual lifecycle features.

## Running it

```
cd TestFrameworkBench
dotnet run -c Release -- --sizes 1,1000
```

The harness generates a test project for every cell of the matrix, restores it, builds it, runs it several
times, and merges the timings into `results/results.json`. `results/report.txt` is regenerated from the whole
file after every run, so the matrix can be filled in over several invocations.

Options:

| Option | Default | Meaning |
| --- | --- | --- |
| `--frameworks` | all | `xunit`, `nunit`, `mstest`, `tunit` |
| `--sizes` | `1,1000,10000` | how many tests the generated suite contains |
| `--scenarios` | all | `Bare`, `PerTestHooks`, `ClassFixture`, `AssemblyFixture`, `DataDriven`, `Async`, `Failing`, `ParallelismCheck` |
| `--runners` | all | `Mtp`, `MtpAot`, `VsTest` |
| `--repetitions` | `10` | timed runs per cell, plus one discarded warm-up |
| `--work` | temp directory | where the generated projects go |
| `--report-only` | | regenerate `report.txt` without measuring anything |

A full sweep takes a few hours, mostly in the 10000 test builds and the Native AOT publishes.

## How the measurement works

Every framework is driven the same way. All four support Microsoft.Testing.Platform, which builds the test
project into a self contained executable, so the run is a plain process launch and the timing is the wall
clock of that process. The legacy path goes through `dotnet test` instead, and TUnit has no VSTest adapter,
so that row is empty for it.

Restore is timed separately and never counted as build time. The cold build deletes the build outputs but
keeps the restore assets, so a restore never happens inside a timed build. The incremental build appends a
line to one test file and rebuilds, which is what a developer actually waits for.

Assertions are written by hand rather than with each framework's assertion library, because the comparison
is about discovery and execution machinery. TUnit's assertions are asynchronous, which would otherwise force
its tests to be async while the other three stayed synchronous.

Tests are spread over classes of 100. Several parallelism models key off the class, so putting everything
into one class would silently serialize some frameworks.

## Two things that make this kind of benchmark wrong

**Tests that do not run.** A misconfigured test project compiles, runs, reports success, and executes
nothing. The harness passes `--minimum-expected-tests` so the runner itself fails when too few tests ran,
and separately parses the executed count out of the summary and refuses to record a measurement that does
not match. An MSTest Native AOT project without the source generator hits exactly this: it builds and runs
and executes zero tests.

**Parallelism that is not on.** Three of the four frameworks run tests sequentially out of the box, and
each turns it on differently. The `ParallelismCheck` scenario generates tests that only sleep, so the total
wall clock says whether they really ran concurrently:

```
dotnet run -c Release -- --sizes 300 --scenarios ParallelismCheck --runners Mtp --repetitions 1
```

300 tests sleeping 20 ms each take 6 seconds if they are serialized. Anything near that means the
configuration did not take effect.
