Diagnosing thread pool starvation from a .NET dump with an AI agent

 
 
  • Gérald Barré

This post is part of the series 'Debugging .NET with an AI agent'. Be sure to check out the rest of the blog posts of the series!

Thread pool starvation does not hang the application, it makes it slow, and it makes it slow in a way that no profiler flags: the CPU is idle, no query is slow, and every individual operation looks fine. What is wrong is that the pool has no thread left to run the next work item, because the previous ones are all blocked waiting for something.

This is also the case where an agent saves the most time, because the evidence is hundreds of identical stacks that a human has to scroll through. The sample contains the application, the collection script, and the skill. Every command output below comes from it.

#The application that stops making progress

500 work items, each blocking on an async call:

C#
for (var i = 0; i < 500; i++)
{
    var id = i;
    ThreadPool.QueueUserWorkItem(_ => _ = GetCustomerAsync(id).Result);
}

static async Task<string> GetCustomerAsync(int id)
{
    await Task.Delay(TimeSpan.FromMinutes(5)); // Stands for a slow database call
    return "customer-" + id;
}

This is sync-over-async on a thread pool thread. In a web application the same pattern hides behind an MVC action that calls .Result or .Wait(), or a constructor that resolves something asynchronous.

#Collecting the data

Two things are worth collecting: a timeline, because starvation is a behaviour over time, and a dump, because the stacks name the culprit.

Shell
dotnet tool install --global dotnet-dump
dotnet tool install --global dotnet-counters

dotnet-counters collect --process-id 1234 --counters System.Runtime --refresh-interval 2 \
    --duration 00:00:00:20 --format json --output ./counters.json

dotnet-dump collect --process-id 1234 --type Heap --output ./starvation.dmp

--type Heap rather than Full here: with hundreds of blocked threads, a Full dump of this trivial application is already 1.6 GB. Field values are not needed for this investigation, only stacks and thread pool state.

#What the skill brings to a slow application

Same setup as the first post of this series, same SKILL.md. Two of its lines matter here, one to pick the commands and one to read the counters correctly:

| Symptom                        | Commands                                       |
| ------------------------------ | ---------------------------------------------- |
| Slow under load, low CPU usage | threadpool, parallelstacks, dumpasync --stats  |

counters*.json: values whose name ends with "/ N sec" are the delta over the interval, not an absolute value

The second one looks like a detail and decides the answer: read as absolute values, the same numbers describe a perfectly healthy thread pool.

Shell
dotnet-dump analyze ./starvation.dmp -c "threadpool" -c "parallelstacks" -c "exit"

#The investigation

The prompt:

The service answers slowly and the CPU is idle. artifacts/counters.json is a 20 second timeline and
artifacts/starvation.dmp is a dump taken at the end of it. Find out why.

##The timeline says "threads", not "work"

dotnet-counters with --format json produces a file an agent can parse directly. The counters that matter:

19:38:17  thread_pool.thread.count  +15   queue.length  -15   cpu.time  0.009
19:38:19  thread_pool.thread.count  +14   queue.length  -13   cpu.time  0.008
19:38:21  thread_pool.thread.count  +11   queue.length  -12   cpu.time  0.005
19:38:23  thread_pool.thread.count  +10   queue.length  -10   cpu.time  0.006
19:38:25  thread_pool.thread.count  +10   queue.length  -10   cpu.time  0.007

Those values are deltas over each 2 second interval, not absolute values, which is the one thing to tell the model up front. Read that way: the pool is adding 5 to 7 threads per second, continuously, while the process consumes about 0.007 s of CPU every 2 seconds, roughly 0.3 % of one core.

A pool that keeps injecting threads is a pool whose existing threads are not coming back. Combined with an idle CPU, that rules out "the work is expensive" and points at "the work is blocked". This is the fingerprint of starvation, and it is visible before any dump is opened.

##The dump confirms it in six lines

Shell
dotnet-dump analyze artifacts/starvation.dmp -c "threadpool" -c "exit"
Using the Portable thread pool.

CPU utilization:  0%
Workers Total:    184
Workers Running:  184
Workers Idle:     0
Worker Min Limit: 15
Worker Max Limit: 32767

184 worker threads, all running, none idle, 0 % CPU. "Running" here means "not parked waiting for work", which is precisely the point: they are all busy doing nothing.

##What all those threads are waiting for

clrstack -all on this dump prints 184 nearly identical stacks. parallelstacks merges them:

Shell
dotnet-dump analyze artifacts/starvation.dmp -c "parallelstacks" -c "exit"
           ~~~~ 132,133,134,135...
             88 System.Threading.Monitor.Wait(Object, Int32)
             88 System.Threading.ManualResetEventSlim.Wait(Int32, CancellationToken)
             88 System.Threading.Tasks.Task.InternalWaitCore(Int32, CancellationToken)
             88 System.Threading.Tasks.Task.InternalWait(Int32, CancellationToken)
             88 System.Threading.QueueUserWorkItemCallbackDefaultContext.Execute()
           ~~~~ 100,101,102,10a...
                ~~~~ bc,be,bf,c0...
                  66 System.Threading.Monitor.Wait(Object, Int32)
             96 System.Threading.ManualResetEventSlim.Wait(Int32, CancellationToken)
             96 System.Threading.Tasks.Task.SpinThenBlockingWait(Int32, CancellationToken)
             96 System.Threading.Tasks.Task.InternalWaitCore(Int32, CancellationToken)
             96 System.Threading.Tasks.Task.InternalWait(Int32, CancellationToken)
             96 System.Threading.Tasks.Task<System.__Canon>.GetResultCore(Boolean)
             96 Program+<>c__DisplayClass0_0.<<Main>$>b__1(Object)
       184 System.Threading.ThreadPoolWorkQueue.Dispatch()
       184 System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart()

==> 188 threads with 2 roots

Everything is there:

  • PortableThreadPool+WorkerThread.WorkerThreadStart at the bottom of 184 stacks: these are pool threads, not application threads. Waits on dedicated threads are irrelevant to a starvation investigation, and this frame is how to tell them apart.
  • Task<T>.GetResultCore under ThreadPoolWorkQueue.Dispatch: a pool thread blocked waiting for a task. That is sync-over-async, and it is the definition of the problem.
  • The frame above it, Program+<>c__DisplayClass0_0.<<Main>$>b__1(Object), names the code: the lambda passed to QueueUserWorkItem. In a real application this is the controller action or the service method to fix.

For a model, this output is also 30 lines instead of several thousand, which is the difference between an answer and a context window full of duplicated frames. Reducing before the data reaches the model is the single most useful habit in this workflow.

##What the blocked tasks are waiting for

Shell
dotnet-dump analyze artifacts/starvation.dmp -c "dumpasync --stats" -c "exit"
MT               Count    Type
0000ffff5c47d830 184      Program+<<<Main>$>g__GetCustomerAsync|0_0>d

184 pending GetCustomerAsync state machines, one per blocked thread. The numbers match, which closes the loop: each blocked pool thread is waiting for one pending async operation, and each of those operations needs a pool thread to resume. The pool is chasing its own tail.

#The fix

await the call instead of reading .Result:

C#
ThreadPool.QueueUserWorkItem(async _ => _ = await GetCustomerAsync(id)); // or a proper async pipeline

In an ASP.NET Core application it means making the action asynchronous all the way down. Raising ThreadPool.SetMinThreads is the mitigation everyone tries first; it buys time under a load spike and does not remove the cause.

#What the agent gets wrong, and how to fix it

Every mistake below has a fix, and it is always one of three things: a line in the skill, a better prompt, or one command you run yourself.

It reads the counters as absolute values. thread.count at +15 is not "15 threads", it is "15 more threads than 2 seconds ago". Read the other way, the timeline describes a healthy pool.

Fix: the counters line quoted at the beginning of this post. You can also check it yourself: threadpool prints an absolute Workers Total: 184, which a pool of "15 threads" cannot have.

It confuses this with the async deadlock from the previous post. Both have Task.GetResultCore at the top.

Fix: the same skill line as in that post, which uses the number of threads and Workers Idle to separate them. Here the answer is on the first screen: 184 workers, 0 idle.

It blames the slow dependency. Task.Delay stands for a database call here, and a model will often conclude "the database is slow" and propose a timeout or a cache. The database being slow is what exposes the bug; the bug is blocking a pool thread while waiting for it.

Fix: two lines in the skill, one for the diagnosis and one for the fix:

In thread pool starvation, a slow dependency exposes the problem but does not cause it. The cause is the
call that blocks a pool thread while it waits

7. The fix must remove the cause. Never propose GC.Collect, ThreadPool.SetMinThreads or a longer timeout.

Rule 7 matters because SetMinThreads is the answer most blog posts give, so it is the answer a model gives too.

It picks a random thread. Given clrstack -all, an agent tends to analyze whichever of the 184 stacks comes first, sometimes a runtime thread.

Fix: the parallelstacks line in the skill. Merged stacks come with counts, and a count of 96 is hard to ignore.

The next post changes symptom entirely: nothing hangs, nothing is slow, memory just keeps growing.

Do you have a question or a suggestion about this post? Contact me!

Follow me:
Enjoy this blog?