This post is part of the series 'Debugging .NET with an AI agent'. Be sure to check out the rest of the blog posts of the series!
The previous post found a leak whose root was a static cache, which is the case everybody recognises. This one has no cache, no static collection, and nothing that looks like storage. It leaks because objects subscribe to an event and because they schedule a timer.
Same three commands, but the gcroot chain looks completely different, and the output is large enough that reading it becomes a problem of its own. The sample contains the application, the collection script, and the skill. Every command output below comes from it.
#The application that grows
Two per-request objects. One subscribes to a static event, the other schedules a timer. Neither unsubscribes nor disposes:
C#
static class ConfigurationMonitor
{
// A static event holds a delegate whose invocation list references every subscriber
public static event EventHandler? Changed;
public static void NotifyChanged() => Changed?.Invoke(null, EventArgs.Empty);
}
sealed class ReportPage
{
private readonly byte[] _renderedHtml = new byte[16 * 1024];
public ReportPage(int id)
{
// Never removed: this page stays alive as long as the application runs
ConfigurationMonitor.Changed += OnConfigurationChanged;
}
}
sealed class RefreshJob
{
private readonly byte[] _buffer = new byte[16 * 1024];
private readonly Timer _timer;
public RefreshJob(int id)
{
// A scheduled timer is rooted by the runtime, and it roots its state and its callback target
_timer = new Timer(Refresh, this, TimeSpan.FromMinutes(10), TimeSpan.FromMinutes(10));
}
}
Both bugs are ordinary. The event one is the classic .NET leak; the timer one shows up whenever a component schedules periodic work and forgets Dispose, which is easy in code that has no explicit lifetime.
#Collecting the data
Shell
dotnet-dump collect --process-id 1234 --type Full --output ./eventleak.dmp
dotnet-gcdump collect --process-id 1234 --output ./heap-t1.gcdump
sleep 15
dotnet-gcdump collect --process-id 1234 --output ./heap-t2.gcdump
Full this time rather than Heap: the interesting objects are delegates, and reading their fields with dumpobj only works on a Full dump.
#The investigation
The prompt:
The memory of ./EventLeakDemo grows and there is no cache in this code. artifacts/ has a dump and two
gcdumps. Find what keeps the objects alive.
##Growth first, this time
The two snapshots come first, because they are cheap and they say whether there is anything to investigate:
# heap-t1.txt # heap-t2.txt, 15 seconds later
39,867,668 bytes 139,633,112 bytes
40 1,200 RefreshJob 40 4,200 RefreshJob
32 1,200 ReportPage 32 4,200 ReportPage
1,200 to 4,200 instances of two application types, 40 MB to 139 MB. The leak is real and it is proportional to the number of requests processed.
##What is on the heap
Shell
dotnet-dump analyze artifacts/eventleak.dmp -c "dumpheap -stat" -c "exit"
ffff785ddd58 1,238 93,292 System.String
ffff788758d0 4,200 100,800 System.Threading.Timer
ffff78875f38 4,200 100,800 System.Threading.TimerHolder
ffff786a6078 4,200 134,400 ReportPage
ffff786a6178 4,200 168,000 RefreshJob
ffff78874600 4,200 268,800 System.Threading.TimerCallback
ffff78874388 4,201 268,864 System.EventHandler
ffff78875d48 4,200 403,200 System.Threading.TimerQueueTimer
ffff78874108 8,402 137,827,249 System.Byte[]
Total 50,596 objects, 140,007,495 bytes
This table is already the diagnosis, for a reader who knows what to look for: 4,200 of each application type, and next to them 4,200 EventHandler, 4,200 TimerCallback and 4,200 TimerQueueTimer. Matching counts mean one delegate and one timer per instance, and delegates and timers are exactly the two things that keep objects alive without looking like storage.
##The event handler chain
Shell
dotnet-dump analyze artifacts/eventleak.dmp -c "dumpheap -mt ffff786a6078" -c "exit"
dotnet-dump analyze artifacts/eventleak.dmp -c "gcroot fff79600a910" -c "exit"
HandleTable:
0000ffffb75b13e8 (strong handle)
-> fff794000028 System.Object[]
-> fff7a11f7a18 System.EventHandler (static variable: System.AppDomain.s_domain)
-> fff7a0e9e280 System.Object[]
-> fff79600e948 System.EventHandler
-> fff79600a910 ReportPage
Found 1 unique roots.
Five links, and the middle three are what a delegate looks like on the heap:
- a
System.EventHandler held by a static: the backing field of ConfigurationMonitor.Changed, - an
Object[]: the invocation list of that multicast delegate, - one
EventHandler per subscription, each holding its target.
dumpobj on the static delegate makes the scale explicit:
MT Field Offset Type VT Attr Value Name
0000ffff774447b0 400025e 28 System.Object 0 instance 0000fff7a0e9e280 _invocationList
0000ffff785d73b0 400025f 30 System.IntPtr 1 instance 0000000000001068 _invocationCount
_invocationCount is 0x1068, that is 4,200 subscribers on a single event, in an application that is supposed to have a handful of live pages. The fix is -= in a Dispose, or a weak event pattern.
##The timer chain, and an output problem
The same command on a RefreshJob produces something much less pleasant:
Shell
dotnet-dump analyze artifacts/eventleak.dmp -c "gcroot fff79600e988" -c "exit"
HandleTable:
0000ffffb75b13e8 (strong handle)
-> fff796012b10 System.Threading.TimerQueue[] (static variable: ...)
-> fff796012ba0 System.Threading.TimerQueue
-> fff7a11fbb08 System.Threading.TimerQueueTimer
-> fff7a11f3910 System.Threading.TimerQueueTimer
-> fff7a11eb718 System.Threading.TimerQueueTimer
[ ... 4,190 more TimerQueueTimer links ... ]
-> fff796012a20 System.Threading.TimerQueueTimer
-> fff79600e988 RefreshJob
Found 3 unique roots.
12,628 lines, because scheduled timers form a linked list and gcroot walks it link by link. The information content is two lines: the root is the runtime's TimerQueue, and the object at the end is a RefreshJob. Everything in between is the same type repeated four thousand times.
This is the shape of output that ruins an agent run: pasted whole, it fills the context window with one repeated frame and pushes out the evidence collected earlier. Two lines of the skill built in the first post fix it, one about this command in particular:
gcroot can print tens of thousands of lines when the root is a linked list, for example a timer queue.
What matters is the first line, the root, and the last one, the object
and one that applies to every command, in the section on running the tools:
Some of these commands print thousands of nearly identical lines. Reduce the output before reading it:
pipe through head, tail, grep or grep -c.
timerinfo answers the same question without the linked list:
Shell
dotnet-dump analyze artifacts/eventleak.dmp -c "timerinfo" -c "exit"
(L) 0x0000FFF7A11FBB08 @ 600000 ms every 600000 ms | 0000FFF7A11F7A58 (RefreshJob) ->
(L) 0x0000FFF7A11F3910 @ 600000 ms every 600000 ms | 0000FFF7A11EF860 (RefreshJob) ->
(L) 0x0000FFF7A11EB718 @ 600000 ms every 600000 ms | 0000FFF7A11E7650 (RefreshJob) ->
4,200 distinct timers, every one of them with a RefreshJob as its state, all with the same 10 minute period. Here too the useful summary is a count and a type name, not the 8,400 lines the command prints. dotnet-dump analyze ... -c "timerinfo" -c "exit" | grep -c RefreshJob is a perfectly good tool call for an agent to make, and one it will make on its own once the skill says that reducing output is expected.
#The fix
Both objects need an owner and a lifetime:
C#
sealed class ReportPage : IDisposable
{
public void Dispose() => ConfigurationMonitor.Changed -= OnConfigurationChanged;
}
sealed class RefreshJob : IDisposable
{
public void Dispose() => _timer.Dispose();
}
The general rule that both cases illustrate: a subscription and a scheduled timer are references from something long-lived to something short-lived. The short-lived object cannot decide to die; only the long-lived one can let go.
#What the agent gets wrong, and how to fix it
Every mistake below has a fix, and it is always one of three things: a line in the skill, a better prompt, or one command you run yourself.
It looks for a collection. Asked to find a leak, models look for a dictionary or a list. Here the roots are a delegate and a timer queue, and the word "cache" appears nowhere in the code.
Fix: a skill line that points at the pattern visible in dumpheap -stat:
Objects are not only kept alive by collections. When an application type has the same count as
System.EventHandler, System.Threading.TimerCallback or System.Threading.TimerQueueTimer, look for an
event subscription or a timer. timerinfo lists the timers
It drowns in gcroot output. A single gcroot on a timer-rooted object can fill the context and end the investigation.
Fix: the two lines quoted in the timer section: one about gcroot on linked lists, one about reducing output with head, tail and grep -c.
It reads the static annotation literally. (static variable: System.AppDomain.s_domain) is how SOS labels any static. A model that reports "the leak is in AppDomain.s_domain" is quoting the tool, not the code.
Fix: the same line as in the deadlock post, which says field names never come from SOS output and names this label explicitly. The type of the rooted object, System.EventHandler, is what identifies the event.
It proposes the wrong fix. GC.Collect, or a weak reference added to ReportPage, the object that cannot do anything about it. The reference belongs to the static event, so only the code that subscribed can remove it.
Fix: rule 7:
7. The fix must remove the cause. Never propose GC.Collect, ThreadPool.SetMinThreads or a longer timeout.
For a leak, remove the reference held by the long-lived object: unsubscribe, dispose the timer, evict the entry
The six posts so far used the same skill from beginning to end. That is the point worth keeping: the setup is one Markdown file, written once, that grows by a line or two every time a command surprises you. The rest of the pattern does not change either, collect the right artifact, run commands that reduce rather than dump everything, and require that every claim in the final report quotes the output that supports it. The agent is good at the mechanical middle part, which is most of the work; deciding what to collect and checking the conclusion is still yours.
Some questions have no answer in any file you can collect, such as which code keeps writing a wrong value. The next post attaches a debugger to the running process instead.
Do you have a question or a suggestion about this post? Contact me!