There is no thread: OS async I/O and the .NET thread pool

This post is part of the series 'Async/await in C#'. Be sure to check out the rest of the blog posts of the series!

• Why async exists: the cost of blocking a thread
• There is no thread: OS async I/O and the .NET thread pool (this post)

The previous post showed that blocking a thread while waiting for I/O throttles everything else the process wants to do, and that awaiting instead does not. That raises an obvious question: if no thread is waiting, who is waiting?

The answer is that nobody is. There is no thread. This post is about what that actually means, because it is not a figure of speech.

#Proving it first

The sample starts 100 000 concurrent one second waits and prints how many threads the process is using while they are all pending:

var tasks = new Task[100_000];
for (var i = 0; i < tasks.Length; i++)
{
    tasks[i] = Task.Delay(TimeSpan.FromSeconds(1));
}

await Task.WhenAll(tasks);

Processors:                 15
Threads before:             9
Thread pool threads before: 0

100000 operations started in 14 ms
Threads while waiting:      10
Thread pool threads:        0

All completed in 1105 ms
Threads after:              27
Thread pool threads after:  16
Completed work items:       99999

One hundred thousand operations in flight, and the process is using ten OS threads. Not ten thousand, ten. The thread pool has not created a single worker thread yet, because there is no work to run: the waits are pending, and pending is not work.

Then everything completes at once and the pool grows to 16 threads to run the 100 000 continuations. Total elapsed: 1105 ms, for an operation that takes 1000 ms.

#So where does the wait live

It lives in the operating system, and for a network read it lives in the network card and its driver.

When you call a synchronous read, the kernel puts your thread on a wait queue attached to the file descriptor, and the scheduler stops giving it CPU time. The thread exists only so that the kernel has something to wake up. It is a bookmark, and an expensive one.

Every operating system offers a way to avoid that bookmark:

• Windows has I/O Completion Ports. You associate a handle with a port, issue an overlapped read that returns immediately, and later the kernel posts a completion packet to the port. A small pool of threads calls GetQueuedCompletionStatus and picks packets up.
• Linux has epoll (and now io_uring). One thread waits on many file descriptors at once and is told which ones are ready.
• macOS and the BSDs have kqueue, which works the same way.

The shape is identical everywhere: one thread waits for many operations, rather than one thread per operation. That inversion is the whole trick, and it is much older than async/await.

.NET wraps all of this. Socket uses completion ports on Windows and an epoll or kqueue based SocketAsyncEngine on Unix. FileStream uses overlapped I/O on Windows when you pass FileOptions.Asynchronous. Task.Delay does not even need I/O: it registers a timer, and the timer queue posts the completion when it fires.

#The thread pool is the other half

The operating system tells you an operation finished. Something still has to run the code that comes after the await. That something is the thread pool.

It maintains two things you should know about:

Worker threads run queued work items. The minimum is the processor count. Above that, the pool adds threads slowly, driven by a hill-climbing algorithm that measures throughput and only keeps threads that improve it. This is why blocked pool threads are so damaging: they do not consume CPU, so the algorithm sees no reason to add more, and the queue just backs up. ThreadPool.SetMinThreads raises the floor and is the usual emergency fix for a starving service, but it treats the symptom.

A global queue and per-thread local queues. Work queued from a pool thread goes into that thread's local queue and is picked up last-in-first-out, which keeps the data it touches warm in cache. Work queued from outside goes to the global queue. Other threads can steal from a local queue when they run out. This is why ThreadPool.UnsafeQueueUserWorkItem takes a preferLocal parameter.

#What queueing one item costs

Every await that suspends ends up queueing something to the pool, so the cost of that operation sets a floor under all async code. The benchmark queues 1000 items four ways and waits for them all:

Roughly 200 nanoseconds per item, and the three low-level variants are within noise of each other. The interesting column is the last one.

IThreadPoolWorkItem is the cheapest: the object you queue is the work item, so there is no separate delegate and no closure. Passing a delegate and a state object costs a third more memory. Task.Run costs three times as much, because it also allocates a Task so that the caller has something to await, plus the delegate.

That is the price of the convenience, and it is a price worth paying when you need the Task. It is not worth paying when you do not: fire-and-forget work that nobody awaits does not need a Task at all.

#Where the thread pool shows up in async code

Putting the two halves together, this is what an await on a pending network read costs:

1. The read is issued and returns immediately. No thread is involved from here on.
2. The calling thread returns to the pool and runs other work.
3. The kernel signals completion through the completion port, epoll or kqueue.
4. .NET queues the continuation to the thread pool.
5. A pool thread, not necessarily the original one, runs the code after the await.

Step 5 is why the thread identity changes across an await, and why thread-affine state such as [ThreadStatic] fields or a thread's CurrentCulture cannot be relied on afterwards. The mechanism that carries the state you do want across that boundary is ExecutionContext, and it gets its own post later.

#What to take away

• Nothing waits during an await. The operating system tracks the pending operation and reports completion.
• 100 000 concurrent waits need about as many threads as an idle process.
• Every OS provides the same inversion: one thread learns about many completed operations, instead of one thread per operation.
• The thread pool runs the continuations. It grows slowly on purpose, which is what makes blocked pool threads so damaging.
• Queueing work costs roughly 200 ns. Task.Run costs three times the memory of the raw pool APIs, because it also allocates the Task.
• The thread that resumes after an await is usually not the thread that started the method.

The next post looks at the object that ties all of this together: Task, what it actually holds, and how a completion turns into a continuation running.

#Additional resources

• Code sample
• Benchmark

Do you have a question or a suggestion about this post? Contact me!