Finding a leak rooted by an event handler or a timer with an AI agent

 
 
  • Gérald Barré

This post is part of the series 'Debugging .NET with an AI agent'. Be sure to check out the rest of the blog posts of the series!

The previous post found a leak whose root was a static cache, which is the case everybody recognises. This one has no cache, no static collection, and nothing that looks like storage. It leaks because objects subscribe to an event and because they schedule a timer.

Same three commands, but the gcroot chain looks completely different, and the output is large enough that reading it becomes a problem of its own. The sample contains the application, the collection script, and the skill. Every command output below comes from it.

#The application that grows

Two per-request objects. One subscribes to a static event, the other schedules a timer. Neither unsubscribes nor disposes:

C#
static class ConfigurationMonitor
{
    // A static event holds a delegate whose invocation list references every subscriber
    public static event EventHandler? Changed;

    public static void NotifyChanged() => Changed?.Invoke(null, EventArgs.Empty);
}

sealed class ReportPage
{
    private readonly byte[] _renderedHtml = new byte[16 * 1024];

    public ReportPage(int id)
    {
        // Never removed: this page stays alive as long as the application runs
        ConfigurationMonitor.Changed += OnConfigurationChanged;
    }
}

sealed class RefreshJob
{
    private readonly byte[] _buffer = new byte[16 * 1024];
    private readonly Timer _timer;

    public RefreshJob(int id)
    {
        // A scheduled timer is rooted by the runtime, and it roots its state and its callback target
        _timer = new Timer(Refresh, this, TimeSpan.FromMinutes(10), TimeSpan.FromMinutes(10));
    }
}

Both bugs are ordinary. The event one is the classic .NET leak; the timer one shows up whenever a component schedules periodic work and forgets Dispose, which is easy in code that has no explicit lifetime.

#Collecting the data

Shell
dotnet-dump collect --process-id 1234 --type Full --output ./eventleak.dmp

dotnet-gcdump collect --process-id 1234 --output ./heap-t1.gcdump
sleep 15
dotnet-gcdump collect --process-id 1234 --output ./heap-t2.gcdump

Full this time rather than Heap: the interesting objects are delegates, and reading their fields with dumpobj only works on a Full dump.

#The investigation

The prompt:

The memory of ./EventLeakDemo grows and there is no cache in this code. artifacts/ has a dump and two
gcdumps. Find what keeps the objects alive.

##Growth first, this time

The two snapshots come first, because they are cheap and they say whether there is anything to investigate:

# heap-t1.txt              # heap-t2.txt, 15 seconds later
     39,867,668  bytes          139,633,112  bytes
             40     1,200  RefreshJob            40     4,200  RefreshJob
             32     1,200  ReportPage            32     4,200  ReportPage

1,200 to 4,200 instances of two application types, 40 MB to 139 MB. The leak is real and it is proportional to the number of requests processed.

##What is on the heap

Shell
dotnet-dump analyze artifacts/eventleak.dmp -c "dumpheap -stat" -c "exit"
ffff785ddd58 1,238      93,292 System.String
ffff788758d0 4,200     100,800 System.Threading.Timer
ffff78875f38 4,200     100,800 System.Threading.TimerHolder
ffff786a6078 4,200     134,400 ReportPage
ffff786a6178 4,200     168,000 RefreshJob
ffff78874600 4,200     268,800 System.Threading.TimerCallback
ffff78874388 4,201     268,864 System.EventHandler
ffff78875d48 4,200     403,200 System.Threading.TimerQueueTimer
ffff78874108 8,402 137,827,249 System.Byte[]
Total 50,596 objects, 140,007,495 bytes

This table is already the diagnosis, for a reader who knows what to look for: 4,200 of each application type, and next to them 4,200 EventHandler, 4,200 TimerCallback and 4,200 TimerQueueTimer. Matching counts mean one delegate and one timer per instance, and delegates and timers are exactly the two things that keep objects alive without looking like storage.

##The event handler chain

Shell
dotnet-dump analyze artifacts/eventleak.dmp -c "dumpheap -mt ffff786a6078" -c "exit"
dotnet-dump analyze artifacts/eventleak.dmp -c "gcroot fff79600a910" -c "exit"
HandleTable:
    0000ffffb75b13e8 (strong handle)
          -> fff794000028     System.Object[]
          -> fff7a11f7a18     System.EventHandler (static variable: System.AppDomain.s_domain)
          -> fff7a0e9e280     System.Object[]
          -> fff79600e948     System.EventHandler
          -> fff79600a910     ReportPage

Found 1 unique roots.

Five links, and the middle three are what a delegate looks like on the heap:

  • a System.EventHandler held by a static: the backing field of ConfigurationMonitor.Changed,
  • an Object[]: the invocation list of that multicast delegate,
  • one EventHandler per subscription, each holding its target.

dumpobj on the static delegate makes the scale explicit:

              MT    Field   Offset                 Type VT     Attr            Value Name
0000ffff774447b0  400025e       28        System.Object  0 instance 0000fff7a0e9e280 _invocationList
0000ffff785d73b0  400025f       30        System.IntPtr  1 instance 0000000000001068 _invocationCount

_invocationCount is 0x1068, that is 4,200 subscribers on a single event, in an application that is supposed to have a handful of live pages. The fix is -= in a Dispose, or a weak event pattern.

##The timer chain, and an output problem

The same command on a RefreshJob produces something much less pleasant:

Shell
dotnet-dump analyze artifacts/eventleak.dmp -c "gcroot fff79600e988" -c "exit"
HandleTable:
    0000ffffb75b13e8 (strong handle)
          -> fff796012b10     System.Threading.TimerQueue[] (static variable: ...)
          -> fff796012ba0     System.Threading.TimerQueue
          -> fff7a11fbb08     System.Threading.TimerQueueTimer
          -> fff7a11f3910     System.Threading.TimerQueueTimer
          -> fff7a11eb718     System.Threading.TimerQueueTimer
          [ ... 4,190 more TimerQueueTimer links ... ]
          -> fff796012a20     System.Threading.TimerQueueTimer
          -> fff79600e988     RefreshJob

Found 3 unique roots.

12,628 lines, because scheduled timers form a linked list and gcroot walks it link by link. The information content is two lines: the root is the runtime's TimerQueue, and the object at the end is a RefreshJob. Everything in between is the same type repeated four thousand times.

This is the shape of output that ruins an agent run: pasted whole, it fills the context window with one repeated frame and pushes out the evidence collected earlier. Two lines of the skill built in the first post fix it, one about this command in particular:

gcroot can print tens of thousands of lines when the root is a linked list, for example a timer queue.
What matters is the first line, the root, and the last one, the object

and one that applies to every command, in the section on running the tools:

Some of these commands print thousands of nearly identical lines. Reduce the output before reading it:
pipe through head, tail, grep or grep -c.

timerinfo answers the same question without the linked list:

Shell
dotnet-dump analyze artifacts/eventleak.dmp -c "timerinfo" -c "exit"
(L) 0x0000FFF7A11FBB08 @  600000 ms every   600000 ms |  0000FFF7A11F7A58 (RefreshJob) ->
(L) 0x0000FFF7A11F3910 @  600000 ms every   600000 ms |  0000FFF7A11EF860 (RefreshJob) ->
(L) 0x0000FFF7A11EB718 @  600000 ms every   600000 ms |  0000FFF7A11E7650 (RefreshJob) ->

4,200 distinct timers, every one of them with a RefreshJob as its state, all with the same 10 minute period. Here too the useful summary is a count and a type name, not the 8,400 lines the command prints. dotnet-dump analyze ... -c "timerinfo" -c "exit" | grep -c RefreshJob is a perfectly good tool call for an agent to make, and one it will make on its own once the skill says that reducing output is expected.

#The fix

Both objects need an owner and a lifetime:

C#
sealed class ReportPage : IDisposable
{
    public void Dispose() => ConfigurationMonitor.Changed -= OnConfigurationChanged;
}

sealed class RefreshJob : IDisposable
{
    public void Dispose() => _timer.Dispose();
}

The general rule that both cases illustrate: a subscription and a scheduled timer are references from something long-lived to something short-lived. The short-lived object cannot decide to die; only the long-lived one can let go.

#What the agent gets wrong, and how to fix it

Every mistake below has a fix, and it is always one of three things: a line in the skill, a better prompt, or one command you run yourself.

It looks for a collection. Asked to find a leak, models look for a dictionary or a list. Here the roots are a delegate and a timer queue, and the word "cache" appears nowhere in the code.

Fix: a skill line that points at the pattern visible in dumpheap -stat:

Objects are not only kept alive by collections. When an application type has the same count as
System.EventHandler, System.Threading.TimerCallback or System.Threading.TimerQueueTimer, look for an
event subscription or a timer. timerinfo lists the timers

It drowns in gcroot output. A single gcroot on a timer-rooted object can fill the context and end the investigation.

Fix: the two lines quoted in the timer section: one about gcroot on linked lists, one about reducing output with head, tail and grep -c.

It reads the static annotation literally. (static variable: System.AppDomain.s_domain) is how SOS labels any static. A model that reports "the leak is in AppDomain.s_domain" is quoting the tool, not the code.

Fix: the same line as in the deadlock post, which says field names never come from SOS output and names this label explicitly. The type of the rooted object, System.EventHandler, is what identifies the event.

It proposes the wrong fix. GC.Collect, or a weak reference added to ReportPage, the object that cannot do anything about it. The reference belongs to the static event, so only the code that subscribed can remove it.

Fix: rule 7:

7. The fix must remove the cause. Never propose GC.Collect, ThreadPool.SetMinThreads or a longer timeout.
   For a leak, remove the reference held by the long-lived object: unsubscribe, dispose the timer, evict the entry

The six posts so far used the same skill from beginning to end. That is the point worth keeping: the setup is one Markdown file, written once, that grows by a line or two every time a command surprises you. The rest of the pattern does not change either, collect the right artifact, run commands that reduce rather than dump everything, and require that every claim in the final report quotes the output that supports it. The agent is good at the mechanical middle part, which is most of the work; deciding what to collect and checking the conclusion is still yours.

Some questions have no answer in any file you can collect, such as which code keeps writing a wrong value. The next post attaches a debugger to the running process instead.

Do you have a question or a suggestion about this post? Contact me!

Follow me:
Enjoy this blog?