Finding a deadlock in a .NET dump with an AI agent

 
 
  • Gérald Barré

This post is part of the series 'Debugging .NET with an AI agent'. Be sure to check out the rest of the blog posts of the series!

A hang is the easiest problem to diagnose from a dump and the hardest to diagnose any other way. There is no exception, no log line, and attaching a debugger to a production process is rarely an option. But the state that explains it, which thread holds what and which thread waits for what, is entirely present in a dump.

This post takes a deadlocked process and lets an agent find both locks. The sample contains the application, the collection script, and the skill. Every command output below comes from it.

#The application that hangs

Two threads take the same two locks in the opposite order. The Barrier only makes the race deterministic:

C#
static class MonitorDeadlock
{
    private static readonly object BalanceLock = new();
    private static readonly object HistoryLock = new();

    public static void Run()
    {
        var ready = new Barrier(2);

        var transfer = new Thread(() =>
        {
            lock (BalanceLock)
            {
                ready.SignalAndWait();
                lock (HistoryLock) { Console.WriteLine("never reached"); }
            }
        })
        { Name = "Transfer" };

        var audit = new Thread(() =>
        {
            lock (HistoryLock)
            {
                ready.SignalAndWait();
                lock (BalanceLock) { Console.WriteLine("never reached"); }
            }
        })
        { Name = "Audit" };

        transfer.Start();
        audit.Start();
        transfer.Join();
    }
}

#Collecting the dump

The process is alive, so dotnet-dump attaches to it:

Shell
dotnet tool install --global dotnet-dump

dotnet-dump ps
dotnet-dump collect --process-id 1234 --type Full --output ./deadlock.dmp

--type Heap is enough to read stacks and locks; Full is what makes dumpobj able to print field values later, and it is the type I default to for a hang because the file stays small when the heap is small.

Two constraints: in a container createdump needs SYS_PTRACE (docker run --cap-add=SYS_PTRACE ...), and a dump is analyzed on the operating system that produced it. The sample's run-in-docker.sh does both.

#What the skill brings to a hang

The setup is the one built step by step in the first post of this series: the diagnostic tools, a dump, and a SKILL.md file that teaches the agent to run SOS commands non-interactively. The same file is reused here unchanged, and two of its lines do all the work for this scenario:

| Symptom             | Commands                             |
| ------------------- | ------------------------------------ |
| Hang, no CPU usage  | parallelstacks, syncblk, dumpasync   |

- syncblk only reports Monitor/lock(object) owners. It is blind to System.Threading.Lock, SemaphoreSlim,
  ReaderWriterLockSlim and async locks: read the stacks for those

The table row is the only reason the agent runs syncblk at all instead of opening the source code. The note below it is what keeps it going when syncblk prints nothing, which is what happens in the second half of this post.

Everything the agent does below is this command, with different arguments, and the final exit that stops it from waiting for input forever:

Shell
dotnet-dump analyze ./deadlock.dmp -c "syncblk" -c "parallelstacks" -c "exit"

#The investigation

The prompt:

./DeadlockDemo hangs and uses no CPU. artifacts/deadlock.dmp is a dump taken while it was hung.
Find out why and propose a fix.

##Who holds what

syncblk lists the objects that are used as monitors and are currently owned:

Shell
dotnet-dump analyze artifacts/deadlock.dmp -c "syncblk" -c "exit"
Index         SyncBlock MonitorHeld Recursion Owning Thread Info          SyncBlock Owner
    2 0000FFF768003CF0            3         1 0000AAAB05BB1C80 10   8   0000fff79280c1b0 System.Object
    3 0000FFF768003D48            3         1 0000AAAB05C60910 11   9   0000fff79280c1c8 System.Object

Two objects are locked, by two different threads. The second value of Owning Thread Info is the operating system thread id: 10 and 11 here (hexadecimal). MonitorHeld is greater than 1 on both, which means each monitor is owned and has a thread waiting for it. Two monitors, each owned by one thread and wanted by another: that is the shape of a deadlock, before looking at a single stack.

##What each thread is doing

Shell
dotnet-dump analyze artifacts/deadlock.dmp -c "parallelstacks" -c "exit"
 ~~~~ 7
    1 System.Threading.Thread.Join(Int32)
    1 System.Threading.Thread.Join()
    1 MonitorDeadlock.Run()
    1 Program.<Main>$(String[])

      ~~~~ 10
         1 System.Threading.Monitor.Enter_Slowpath(Object)
         1 System.Threading.Monitor.Enter(Object, Boolean ByRef)
         1 MonitorDeadlock+<>c__DisplayClass2_0.<Run>b__0()
      ~~~~ 11
         1 System.Threading.Monitor.Enter_Slowpath(Object)
         1 System.Threading.Monitor.Enter(Object, Boolean ByRef)
         1 MonitorDeadlock+<>c__DisplayClass2_0.<Run>b__1()
    2 System.Threading.Thread.StartCallback()

==> 3 threads with 2 roots

parallelstacks merges identical stacks and prefixes each group with the thread ids it covers. Thread 10 owns one monitor, per syncblk, and is blocked in Monitor.Enter; thread 11 does the reverse. The main thread 7 is in Thread.Join, waiting for a thread that will never finish.

The frames also name the code: MonitorDeadlock.Run closure methods b__0 and b__1, which are the two lambdas passed to new Thread(...). That is enough for the agent to point at the two lock statements and propose the fix: take BalanceLock and HistoryLock in the same order everywhere, or hold a single lock.

clrstack -all shows the same information with the full frames, one thread at a time. On a three-thread process both work; on a process with hundreds of threads, parallelstacks is the one to give a model, because it turns hundreds of nearly identical stacks into a few lines with counts.

#The blind spot: System.Threading.Lock

Run the same investigation on the second scenario of the sample, which uses the Lock type introduced in .NET 9 instead of locking on an object:

C#
private static readonly Lock BalanceLock = new();
private static readonly Lock HistoryLock = new();

The bug is identical. syncblk says nothing at all:

Shell
dotnet-dump analyze artifacts/deadlock.dmp -c "syncblk" -c "exit"
Index         SyncBlock MonitorHeld Recursion Owning Thread Info          SyncBlock Owner
-----------------------------
Total           1
Free            0

No rows. A Lock does not use a sync block, so the command that identifies lock owners has nothing to report. The same is true of SemaphoreSlim, ReaderWriterLockSlim, and every async lock.

The stacks still tell the story:

      ~~~~ 50
         1 System.Threading.WaitHandle.WaitOneNoCheck(Int32, Boolean, Object, WaitHandleWaitSourceMap)
         1 System.Threading.Lock.TryEnterSlow(Int32, ThreadId)
         1 System.Threading.Lock.EnterAndGetCurrentThreadId()
         1 System.Threading.Lock.EnterScope()
         1 LockTypeDeadlock+<>c__DisplayClass2_0.<Run>b__0()
      ~~~~ 51
         1 System.Threading.WaitHandle.WaitOneNoCheck(Int32, Boolean, Object, WaitHandleWaitSourceMap)
         1 System.Threading.Lock.TryEnterSlow(Int32, ThreadId)
         1 System.Threading.Lock.EnterAndGetCurrentThreadId()
         1 System.Threading.Lock.EnterScope()
         1 LockTypeDeadlock+<>c__DisplayClass2_0.<Run>b__1()

Two threads inside Lock.TryEnterSlow, in two different lambdas of the same method. The conclusion is the same, it just requires reading the stacks instead of a table. What is lost is the ownership information: syncblk said which thread held which object, the stacks only say who is waiting.

This matters more with an agent than with a human. Most of what has been written about SOS predates .NET 9, so a model that answers from memory will treat an empty syncblk as "no lock contention, look somewhere else". Stating the limit explicitly in the skill is what prevents that:

syncblk only reports Monitor/lock(object) owners. It is blind to System.Threading.Lock, SemaphoreSlim,
ReaderWriterLockSlim and async locks: read the stacks for those

#What the agent gets wrong, and how to fix it

Every mistake below has a fix, and it is always one of three things: a line in the skill, a better prompt, or one command you run yourself.

It calls two owned monitors a deadlock. Two locked objects are just two locks. They become a deadlock when the stacks show each owner blocked on the object the other one holds, and a model that has read syncblk tends to skip that check.

Fix: a line in the skill that requires the pair:

syncblk lists owners, not waiters. Before calling it a deadlock, show from the stacks that each owner is
blocked on a lock held by the other

It reads an empty syncblk as "no lock". Then it moves on to another hypothesis.

Fix: the System.Threading.Lock line quoted in the previous section. With it, an empty table sends the agent to the stacks instead of away from them.

It names the lock. The SyncBlock Owner column gives the address and the type of the locked object, System.Object, never the field that holds it. A model will still write BalanceLock in its report, because that is what the source suggests, and it will be right half the time.

Fix: a skill line that says where field names do not come from:

SOS prints object addresses and types, never the name of the field that holds them [...]. Do not name a
field unless a stack frame or the source at that frame shows it; otherwise identify the object by its
address and type

Here the stacks are enough: <Run>b__0 blocks on its second lock, and the source of that lambda says which field that is. A report that says so, citing the frame, is correct. A report that just says BalanceLock is a guess that happens to look the same.

The next post keeps the same symptom, a process that hangs and uses no CPU, but with no lock involved at all: an await whose continuation can never run.

Do you have a question or a suggestion about this post? Contact me!

Follow me:
Enjoy this blog?