Finding an async deadlock caused by a SynchronizationContext with an AI agent

 
 
  • Gérald Barré

This post is part of the series 'Debugging .NET with an AI agent'. Be sure to check out the rest of the blog posts of the series!

The previous post found a deadlock by asking which thread owned which monitor. This one has the same symptom, a process that hangs and uses no CPU, and none of that evidence: no monitor is held, no thread is contended, and the thread pool is idle. It is the classic .Result deadlock, and finding it from a dump means following a completely different trail.

The sample contains the application, the collection script, and the skill. Every command output below comes from it.

#The application that hangs

A console application that installs a single-threaded SynchronizationContext, the way WinForms, WPF, MAUI and legacy ASP.NET do, and then blocks on an async call:

C#
var context = new SingleThreadedSynchronizationContext();
SynchronizationContext.SetSynchronizationContext(context);

var customer = GetCustomerAsync(1001).Result; // never returns

static async Task<string> GetCustomerAsync(int id)
{
    await Task.Delay(TimeSpan.FromMilliseconds(200)); // Stands for a database or HTTP call
    return "customer-" + id;
}

sealed class SingleThreadedSynchronizationContext : SynchronizationContext
{
    private readonly BlockingCollection<(SendOrPostCallback Callback, object? State)> _queue = [];

    public override void Post(SendOrPostCallback d, object? state) => _queue.Add((d, state));

    // The main thread never gets back here to pump the queue: it is blocked on .Result
    public void RunMessageLoop()
    {
        foreach (var (callback, state) in _queue.GetConsumingEnumerable())
        {
            callback(state);
        }
    }
}

The 200 ms delay completes almost immediately. What never happens is the continuation after the await: it is posted to the synchronization context, and the only thread that runs the callbacks posted to that context is the thread blocked on .Result. SynchronizationContext, ConfigureAwait and deadlocks explains the mechanism in detail.

#Collecting the dump

Shell
dotnet tool install --global dotnet-dump

dotnet-dump ps
dotnet-dump collect --process-id 1234 --type Full --output ./asyncdeadlock.dmp

Full and not Heap: the last step of this investigation reads the fields of the synchronization context, and dumpobj only prints field values on a Full dump. In a container, add --cap-add=SYS_PTRACE, and analyze the dump on the operating system that produced it.

#The same skill, the same starting point

The skill built in the first post of this series is reused here unchanged, and its symptom table sends the agent to the same three commands as the previous post, because the symptom is the same:

| Symptom             | Commands                             |
| ------------------- | ------------------------------------ |
| Hang, no CPU usage  | parallelstacks, syncblk, dumpasync   |

Two of the three say nothing useful this time. What makes the difference is the rule that an empty output is a result to report rather than a reason to start guessing:

3. If a command fails or prints nothing, say so. Do not fall back to guessing from the source code

As always, commands go on the command line and the last one is exit:

Shell
dotnet-dump analyze ./asyncdeadlock.dmp -c "parallelstacks" -c "syncblk" -c "dumpasync" -c "exit"

#The investigation

The prompt is the same as for any hang:

./AsyncDeadlockDemo hangs and uses no CPU. artifacts/asyncdeadlock.dmp is a dump taken while it was hung.
Find out why and propose a fix.

##The stacks: one blocked thread, an idle pool

Shell
dotnet-dump analyze artifacts/asyncdeadlock.dmp -c "parallelstacks" -c "exit"
 ~~~~ 68
    1 System.Threading.Monitor.Wait(Object, Int32)
    1 System.Threading.ManualResetEventSlim.Wait(Int32, CancellationToken)
    1 System.Threading.Tasks.Task.SpinThenBlockingWait(Int32, CancellationToken)
    1 System.Threading.Tasks.Task.InternalWaitCore(Int32, CancellationToken)
    1 System.Threading.Tasks.Task.InternalWait(Int32, CancellationToken)
    1 System.Threading.Tasks.Task<System.__Canon>.GetResultCore(Boolean)
    1 Program.<Main>$(String[])

      ~~~~ 71
         1 System.Threading.WaitHandle.WaitOneNoCheck(Int32, Boolean, Object, WaitHandleWaitSourceMap)
         1 System.Threading.TimerQueue.TimerThread()
      ~~~~ 73
         1 System.Threading.WaitHandle.WaitOneNoCheck(Int32, Boolean, Object, WaitHandleWaitSourceMap)
         1 System.Threading.PortableThreadPool+GateThread.GateThreadStart()
      ~~~~ 72,74
         2 System.Threading.LowLevelLifoSemaphore.WaitNative(SafeWaitHandle, Int32)
         2 System.Threading.LowLevelLifoSemaphore.WaitForSignal(Int32)
         2 System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart()
    4 System.Threading.Thread.StartCallback()

==> 5 threads with 2 roots

Five threads in total. One is blocked in Task.GetResultCore called from Main. The other four are runtime threads doing nothing: the timer thread, the gate thread, and two worker threads parked in LowLevelLifoSemaphore.WaitForSignal, which is what an idle pool thread waits on.

That last detail is the one worth teaching an agent, because it separates this bug from thread pool starvation, which produces a nearly identical top frame. Here: 5 threads, 2 idle workers, and a single blocking call. Starvation looks like 181 threads with none idle.

##No monitor is involved

Shell
dotnet-dump analyze artifacts/asyncdeadlock.dmp -c "syncblk" -c "exit"
Index         SyncBlock MonitorHeld Recursion Owning Thread Info          SyncBlock Owner
-----------------------------
Total           1
Free            0

Empty. Whatever this is, it is not two threads fighting over lock statements. At this point a model that reasons from memory tends to give up on the dump and start reading source code, which is exactly what the rule quoted above prevents: an empty syncblk is a fact to record, not a dead end.

##The pending state machine

dumpasync walks the async state machines that are alive on the heap and shows what each one is waiting for:

Shell
dotnet-dump analyze artifacts/asyncdeadlock.dmp -c "dumpasync" -c "exit"
STACK 1
<< Awaiting: 0000fff79a80c568 0000ffff7cc6c2c0 System.Runtime.CompilerServices.TaskAwaiter >>
  0000fff79a80c518 0000ffff7cd1a1c0 (0) Program+<<<Main>$>g__GetCustomerAsync|0_0>d
    0000fff79a80c608 0000ffff7cd1c228 () System.Threading.Tasks.Task+SetOnInvokeMres

One state machine, GetCustomerAsync, still at state (0), which is the first await. The last line is the interesting one: the continuation object attached to its task is a SetOnInvokeMres, the event that Task.Wait and .Result use to block a thread. So the task the thread is waiting for and the state machine that never resumed are the same object, and the state machine is waiting on a TaskAwaiter.

That already contradicts the naive reading of the source ("the delay is 200 ms, it must have finished"). It has finished. Its continuation has not run.

##Where the continuation went

The last step is to find the queue it was posted to. Find the synchronization context on the heap:

Shell
dotnet-dump analyze artifacts/asyncdeadlock.dmp -c "dumpheap -type SingleThreadedSynchronizationContext" -c "exit"
         Address               MT               Size
    fff79a80acb0     ffff7ca968f8                 32

Statistics:
          MT Count TotalSize Class Name
ffff7ca968f8     1        32 SingleThreadedSynchronizationContext

Read its fields, then the fields of the collection it holds:

Shell
dotnet-dump analyze artifacts/asyncdeadlock.dmp -c "dumpobj fff79a80acb0" -c "exit"
              MT    Field   Offset                 Type VT     Attr            Value Name
0000ffff7c9a30e0  4000bf2        8       System.Boolean  1 instance                0 _requireWaitNotification
0000ffff7cc677c8  4000001       10 ...Private.CoreLib]]  0 instance 0000fff79a80acd0 _queue
Shell
dotnet-dump analyze artifacts/asyncdeadlock.dmp -c "dumpobj 0000fff79a80acd0" -c "exit"
              MT    Field   Offset                 Type VT     Attr            Value Name
0000ffff7cc68b20  4000005        8 ...Private.CoreLib]]  0 instance 0000fff79a80ad18 _collection
0000ffff7c9a74e8  4000006       30         System.Int32  1 instance               -1 _boundedCapacity

_collection is the ConcurrentQueue behind the BlockingCollection, and SOS has a command that prints its content:

Shell
dotnet-dump analyze artifacts/asyncdeadlock.dmp -c "dumpconcurrentqueue 0000fff79a80ad18" -c "exit"
System.Collections.Concurrent.ConcurrentQueue<System.ValueTuple<System.Threading.SendOrPostCallback, System.Object>>
   1 - dumparray 0000fff79a80af00
---------------------------------------------
1 items

One item, a SendOrPostCallback, sitting in the queue of the synchronization context. Nobody is pumping it, because the only thread that would is thread 68, blocked in .Result.

That is the complete chain, and every link is a command output: the thread blocked in GetResultCore, the state machine still at its first await, and the continuation queued in a context that is never pumped. Diagnosis: blocking on an async call from a thread that owns a SynchronizationContext.

#The fix

await instead of .Result, which is the only fix that removes the cause:

C#
var customer = await GetCustomerAsync(1001);

When the blocking call cannot be removed, ConfigureAwait(false) on the awaits inside the library prevents the continuation from being posted back to the context. It makes this specific deadlock disappear without making the code correct; the thread is still blocked while waiting.

Two remarks for an agent's report:

  • The bug is at the call site, but the fix often belongs to the library being called. The stacks show both.
  • This is a two-line bug that a code review catches instantly. The dump is what tells you this hang is that bug, rather than the ten other things that can hang a process.

#What the agent gets wrong, and how to fix it

Every mistake below has a fix, and it is always one of three things: a line in the skill, a better prompt, or one command you run yourself.

It confuses this with thread pool starvation. Both show a thread blocked in Task.GetResultCore, and the fix a model proposes for starvation, more threads, does nothing here.

Fix: a line in the skill that tells the two apart with something visible in the dump:

A thread blocked in Task.GetResultCore has two common causes. A handful of threads with idle pool workers
(threadpool, or workers parked in LowLevelLifoSemaphore.WaitForSignal) is a SynchronizationContext
deadlock. Hundreds of pool threads with Workers Idle: 0 is thread pool starvation

On this dump, the parallelstacks output from the beginning already shows the two parked workers.

It trusts the source over the dump. "The task is a 200 ms delay, so it completed" is true and irrelevant: the state machine never resumed.

Fix: rule 2, and reading the evidence section. Every sentence in the report must come with a command, and "it completed" has none. When you see a claim like that, send it back with one question: which command shows it? Here the only answer is dumpasync, and it says the opposite.

It stops at "no monitor is held".

Fix: rule 3, quoted at the beginning of this post, plus the symptom table: it lists dumpasync right after syncblk, so an empty syncblk still leaves the agent a next command to run.

It asserts the queue without looking. "The continuation is stuck in the context queue" is the correct conclusion, and models tend to write it after one command instead of four.

Fix: the same question as above. If the evidence section has no dumpconcurrentqueue output, the claim is a guess, and asking for the command that prints the queued item is enough for the agent to go and run the three missing steps.

The next post takes the third variation of the same top frame: hundreds of threads blocked in Task.GetResultCore, which is thread pool starvation.

Do you have a question or a suggestion about this post? Contact me!

Follow me:
Enjoy this blog?