A container restarted during the night, and this is everything the log has:
Unhandled exception. System.NullReferenceException: Object reference not set to an instance of an object.
at Program.<<Main>$>g__GetCustomerDomain|0_2(Order order) in /work/series/CrashDemo/Program.cs:line 26
at Program.<<Main>$>g__Process|0_1(Order order) in /work/series/CrashDemo/Program.cs:line 20
at Program.<Main>$(String[] args) in /work/series/CrashDemo/Program.cs:line 6
Which line failed is known. Which order failed is not, and in a real application that is the only question that matters: is it one bad record, or is the customer missing for a whole category of orders? That data was in memory when the process died, so it is in the crash dump, and nowhere else.
Getting it out is mechanical: run a command, read a table of hexadecimal addresses, use one of them as the argument of the next command, repeat until the picture makes sense. There is nothing clever in it, only the mapping from a symptom to the right command and the patience to cross-reference addresses. That is exactly the kind of work a coding agent does well, provided it can run the commands instead of guessing what they would print.
This post does the whole thing, step by step:
- install the diagnostic tools,
- collect a dump of the crash,
- run one command yourself, before delegating anything,
- write a skill, the file that teaches the agent how to read a dump,
- ask the question in one sentence,
- follow the investigation,
- check the answer.
There is no MCP server, no plugin and no integration to set up. The agent needs a shell and one Markdown file. The rest of this series reuses that same file on a different problem in each post.
The sample contains the application, the collection script, and the skill. Every command output below comes from it.
#The application that crashes
CrashDemo processes orders. The third one has no customer:
C#
var orders = LoadOrders();
foreach (var order in orders)
{
Process(order);
}
static Order[] LoadOrders() =>
[
new Order { Id = 1001, Reference = "REF-1001", Customer = new Customer { Email = "alice@example.com" } },
new Order { Id = 1002, Reference = "REF-1002", Customer = new Customer { Email = "bob@example.com" } },
new Order { Id = 1003, Reference = "REF-1003", Customer = null },
];
static string GetCustomerDomain(Order order)
{
var email = order.Customer!.Email;
return email[(email.IndexOf('@') + 1)..];
}
The bug is visible here because the whole application fits on a screen. Pretend it does not: the orders come from a database, there are forty thousand of them, and the only thing you have is the stack trace above.
#How an agent can analyze a dump
A dump is a binary file of a few hundred megabytes. The agent never opens it, and none of it is uploaded anywhere. What reads a dump is dotnet-dump analyze, a command line tool that knows how to walk the memory of a dead .NET process and prints text.
That tool is usually presented as an interactive shell, which an agent cannot use. But it also accepts -c several times and runs the commands in order:
Shell
dotnet-dump analyze ./crash.dmp -c "clrthreads" -c "printexception" -c "exit"
This is the single detail the whole workflow rests on. From there the agent works the way it works on any other command line tool: it runs a command, reads the text that comes back, and decides what to run next. There is nothing to connect and nothing to install beyond the tool itself.
Two consequences worth understanding before going further:
- The dump file stays on the machine. Only the text output of each command goes into the model context, which is a much smaller amount of data, and a much more sensitive one. The Dumps contain secrets section below covers what that means.
- The agent can only know what a command printed. Everything it says that is not in some command output is a guess, and step 4 is mostly about making guesses visible.
#Step 1: Install the diagnostic tools
Shell
dotnet tool install --global dotnet-dump
That is the only tool this post needs. Later posts in the series also use these two:
Shell
dotnet tool install --global dotnet-gcdump
dotnet tool install --global dotnet-counters
An agent with a shell can install them on its own, but install them yourself once anyway: it keeps the first run from starting with three minutes of tool installation, and it tells you immediately whether they work on this machine.
#Step 2: Collect a dump of the crash
Nothing has to be watching the process. The runtime writes the dump itself when a crash happens, if you set three environment variables before starting the application. This is the only reliable way to get a crash dump in a container, where the process is gone before anyone can react:
Shell
export DOTNET_DbgEnableMiniDump=1
export DOTNET_DbgMiniDumpType=4
export DOTNET_DbgMiniDumpName=/dumps/crash.dmp
DOTNET_DbgMiniDumpType decides what is captured:
| Value | Type | Contains |
|---|
| 1 | Mini | Modules, threads, stacks, exceptions |
| 2 | Heap | The above plus the managed heap |
| 3 | Triage | Like Mini, with personal information removed |
| 4 | Full | All memory, including module images |
Use 4. The reason is the dumpobj needs a Full dump section below, and it costs a bigger file: 118 MB instead of 72 MB for this tiny application.
If the process is still running, dotnet-dump collects a dump on demand instead. How to generate a dump file of a .NET application covers the other ways.
Shell
dotnet-dump ps
dotnet-dump collect --process-id 1234 --type Full --output ./crash.dmp
Two constraints will bite before the agent is even involved:
- In a container,
createdump needs SYS_PTRACE to read the memory of the target process: docker run --cap-add=SYS_PTRACE .... - A dump is analyzed on the operating system that produced it. Copying a Linux dump to a Mac gives
Analyzing Windows or Linux dumps not supported when running on MacOS. The sample's run-in-docker.sh runs the collection, and later the analysis, inside a container based on the same image as production.
In the sample, this whole step is one script:
Shell
./collect.sh # or ./run-in-docker.sh on macOS
It leaves artifacts/crash.dmp next to the application.
#Step 3: Run one command yourself
Before handing anything over, run one command by hand. It takes ten seconds, it proves the tool works on this dump on this machine, and it gives you a piece of real output to compare against later:
Shell
dotnet-dump analyze artifacts/crash.dmp -c "clrthreads" -c "exit"
DBG ID OSID ThreadOBJ State GC Mode ... Apt Exception
0 1 7 0000AAAAF7324E10 20020 Cooperative ... Ukn System.NullReferenceException 0000fff75e80ae00
5 2 c 0000AAAAF7326E50 21220 Preemptive ... Ukn (Finalizer)
6 3 d 0000AAAAF732C9D0 21220 Preemptive ... Ukn
Three managed threads, and thread 0 carries a System.NullReferenceException whose address is right there in the last column.
This is worth doing every time, and not only the first time. A model that has seen thousands of SOS transcripts can produce output that looks exactly like the real thing for a command it never ran. Having seen one genuine table means you know what the real one looks like, and that this dump has three threads and not eighteen.
#Step 4: Write the skill
Left alone with a dump, an agent does two annoying things: it invents SOS commands that do not exist, and it reads the source code until it finds something that looks like the bug, then presents it as the conclusion. Both are fixed by giving it the knowledge it is missing, in a file.
##What a skill is
A skill is a folder containing a file named SKILL.md. The file is a short YAML header followed by Markdown instructions:
Markdown
---
name: analyzing-dotnet-dumps
description: Diagnose a .NET process post-mortem from a dump, gcdump, counters or trace file. Use when a .dmp, .gcdump or .nettrace file is mentioned, or when asked why a .NET process crashed, hung, answers slowly, or grows in memory.
---
# Analyzing .NET dumps
You are debugging a .NET application post-mortem, from a dump file. You cannot run the application, you
cannot add logs. The dump is the only source of truth.
...
The mechanism is worth understanding because it is the whole reason to use a skill here. The agent knows the name and description of every skill available to it, permanently, and that costs almost nothing. It reads the body only when the description matches what you asked. A file such as AGENTS.md or CLAUDE.md works the other way around: its entire content is loaded into every conversation in that repository.
Sixty lines of SOS trivia are exactly what you want when you hand over a dump, and pure noise in every other conversation about that code base. That is a skill.
##Where the file goes
The folder name is the skill name, and the file must be called SKILL.md. Where the folder goes depends on the agent:
| Agent | In a repository | For all your projects |
|---|
| Claude Code | .claude/skills/analyzing-dotnet-dumps/SKILL.md | ~/.claude/skills/ |
| GitHub Copilot | .github/skills/ or .claude/skills/ | ~/.copilot/skills/ |
| Codex | .agents/skills/analyzing-dotnet-dumps/SKILL.md | ~/.agents/skills/ |
Dump analysis is not specific to one repository, so the personal folder is the better place for it in real life: write it once, and it is available in every project you debug. The sample ships it inside the project, at .claude/skills/analyzing-dotnet-dumps/SKILL.md, so that unzipping the sample and opening an agent in that folder is enough to reproduce the post.
##What goes in the body
The body is ordinary Markdown, and the agent reads it as instructions. Four things belong in it, and every one of them corresponds to something that goes wrong without it.
How to run the tool. Without this, the agent runs dotnet-dump analyze crash.dmp, and waits forever:
## Running the tools
Run SOS commands non-interactively. Every invocation reloads the dump, so batch related commands, and
always end with "exit" or the process waits for input forever:
dotnet-dump analyze <dump> -c "<command>" -c "<command>" -c "exit"
Some of these commands print thousands of nearly identical lines. Reduce the output before reading it:
pipe through `head`, `tail`, `grep` or `grep -c`.
A map from symptom to command. This is the part that takes a human years to acquire, and it fits in a five row table. Without it, a model reaches for commands it half remembers, and some of them do not exist:
## Where to start
| Symptom | Commands |
| ------------------------------------ | ------------------------------------------------------------------ |
| Crash | `clrthreads`, `printexception`, `clrstack -a`, `dso`, `dumpobj` |
| Hang, no CPU usage | `parallelstacks`, `syncblk`, `dumpasync` |
| Slow under load, low CPU usage | `threadpool`, `parallelstacks`, `dumpasync --stats` |
| Memory grows | `dumpheap -stat` (biggest types last), then `gcroot <address>` |
| High CPU usage | a dump is the wrong tool, ask for a `dotnet-trace` trace instead |
The limits of each command. Most of what has been written about SOS is old, and a model answering from memory will happily retry a command that cannot work on this dump. This is the section that grows: every time the agent gets something wrong, the fix is usually one more line here. The first one comes from this post:
## Known limits
- `dumpobj` only lists field values when the dump was collected with `--type Full`. On a `Heap` or `Mini`
dump it prints three lines and the tool exits: ask for a Full dump instead of retrying
Rules that keep the answer honest. These matter more than the rest, because a wrong dump analysis is expensive: it sends someone to rewrite code that was never the problem.
## Rules
1. Only report what a command printed. Never invent an address, a type name, a thread id, or a stack frame
2. Quote the command and the relevant output lines as evidence for every claim
3. If a command fails or prints nothing, say so. Do not fall back to guessing from the source code
Rule 2 is the one that makes the report checkable in step 7: an answer that quotes the command and its output can be verified by running that command again, and an answer that does not cannot be verified at all.
Four more rules follow in the file: do not claim a memory leak from a single dump, report what you have after about ten commands rather than running twenty, do not touch the source code until the dump supports the diagnosis, and do not propose a fix that only hides the cause. The complete file is in the sample. It also has a section that tells apart problems that look alike, and an output format asking for symptom, evidence, root cause, fix and confidence. It already contains the lines every post in this series relies on, and each post explains the ones it needs.
#Step 5: Ask the question
With the skill in place, the prompt is a single sentence, and it does not mention SOS, dotnet-dump, or the skill itself:
artifacts/crash.dmp is a dump of the crash in ./CrashDemo. Find which input caused it.
The word .dmp in that sentence is what matches the description written in step 4, so the agent loads the skill on its own and starts from the symptom table. Every agent also lets you name a skill explicitly when you would rather not rely on that, which is worth doing the first few times to check that the file is where you think it is.
#Step 6: Follow the investigation
What follows is the agent's run. Read it as the sequence of commands it chose, because the same sequence works when you do it by hand.
##Which thread, which exception
It starts with clrthreads, the command from step 3, and gets the same three threads and the same exception address. printexception formats that exception:
Shell
dotnet-dump analyze artifacts/crash.dmp -c "printexception" -c "exit"
Exception object: 0000fff75e80ae00
Exception type: System.NullReferenceException
Message: Object reference not set to an instance of an object.
InnerException: <none>
StackTrace (generated):
SP IP Function
0000FFFFECE27990 0000FFFF40FF3344 CrashDemo.dll!Program.<<Main>$>g__GetCustomerDomain|0_2(Order)+0x44
0000FFFFECE279C0 0000FFFF40FF31E8 CrashDemo.dll!Program.<<Main>$>g__Process|0_1(Order)+0x30
0000FFFFECE27A10 0000FFFF40FF1D7C CrashDemo.dll!Program.<Main>$(System.String[])+0x12c
So far this is the log, reformatted, and if the investigation stopped here it would have added nothing. InnerException: <none> is worth noting anyway: on a wrapped exception, printexception follows the chain, which is often where the real error is.
##The part the log does not have
dso (dump stack objects) lists every managed object referenced from the current thread's stack:
Shell
dotnet-dump analyze artifacts/crash.dmp -c "dso" -c "exit"
SP/REG Object Name
ffffece25da0 fff75e80ae00 System.NullReferenceException
ffffece27948 fff75e80a8f0 System.IO.TextWriter+SyncTextWriter
ffffece27968 fff75e80adb8 System.String
ffffece27988 fff75e80ace8 Order
ffffece279b8 fff75e80ace8 Order
ffffece27a08 fff75e80ace8 Order
ffffece27a38 fff75e80ac38 Order[]
ffffece27a68 fff75e8096d0 System.String[]
There is the Order being processed, and the Order[] it came from. clrstack -a reaches the same object through the method parameters, which is useful when several objects of the same type are alive:
Program.<<Main>$>g__GetCustomerDomain|0_2(Order) [/work/series/CrashDemo/Program.cs @ 26]
PARAMETERS:
order (0x0000FFFFECE279B8) = 0x0000fff75e80ace8
Now read the object:
Shell
dotnet-dump analyze artifacts/crash.dmp -c "dumpobj fff75e80ace8" -c "exit"
Name: Order
MethodTable: 0000ffff410a4af0
Size: 40(0x28) bytes
File: /work/bin/CrashDemo/CrashDemo.dll
Fields:
MT Field Offset Type VT Attr Value Name
0000ffff40fb74e8 4000001 18 System.Int32 1 instance 1003 <Id>k__BackingField
0000ffff40fddd58 4000002 8 System.String 0 instance 0000fff749600d50 <Reference>k__BackingField
0000ffff41264150 4000003 10 Customer 0 instance 0000000000000000 <Customer>k__BackingField
Id = 1003, and Customer = 0000000000000000, which is null. Strings are one more dumpobj away:
Shell
dotnet-dump analyze artifacts/crash.dmp -c "dumpobj 0000fff749600d50" -c "exit"
Name: System.String
String: REF-1003
The answer to "which input caused it" is order 1003, reference REF-1003, and the reason is a Customer that is null. dumparray on the Order[] from dso shows the whole batch, so the agent can also answer "how many of them are affected":
Shell
dotnet-dump analyze artifacts/crash.dmp -c "dumparray fff75e80ac38" -c "exit"
Name: Order[]
Array: Rank 1, Number of elements 3, Type CLASS
[0] 0000fff75e80ac68
[1] 0000fff75e80aca8
[2] 0000fff75e80ace8
That is the loop worth automating: dso or clrstack -a to find the objects, dumpobj to read them, dumparray to walk collections. It is entirely mechanical, it is where a human gets bored and stops, and it produces the only information the log did not have.
#Step 7: Check the answer
The report comes back in the shape the skill asked for: symptom, evidence, root cause, fix, confidence. The evidence section is the one to read, and checking it takes one command.
Pick the claim the conclusion rests on, here "the Order at fff75e80ace8 has Id = 1003 and a null Customer", and run that one command yourself:
Shell
dotnet-dump analyze artifacts/crash.dmp -c "dumpobj fff75e80ace8" -c "exit"
If the output matches what the report quoted, the chain holds, because every earlier step only produced that address. If it does not, you have learned something much more important than the answer to this crash.
The two things to look at specifically:
- Does every address in the report appear in the output of an earlier command? Addresses are the one thing a model cannot plausibly invent twice in a row, so a mismatch is obvious.
- Is the conclusion from the dump, or from the source? "The third order has no customer" is in
Program.cs. "The crash happened on order 1003" is in the dump. Only the second is evidence, and on a real application only the second is true.
That is the trade this workflow makes: you go from "run twenty commands and cross-reference addresses" to "read a report and verify one command".
#dumpobj needs a Full dump
Collect the same crash with DOTNET_DbgMiniDumpType=2 (Heap) and the investigation stops halfway:
Name: Order
MethodTable: 0000ffff5a884af0
Canonical MethodTable: 0000ffff5a884af0
Three lines, then the tool exits with a segmentation fault, on every object I tried, including System.String. Pointing it at the application binaries with setsymbolserver -directory does not help. dumpheap -stat, clrstack, dso and gcroot all work on that same Heap dump; only reading field values needs the module images that a Full dump contains.
Tested with dotnet-dump 9.0.661903 and dotnet-debug 0.0.731901 on .NET 10 dumps. The practical consequence is the line already quoted in step 4, because otherwise a model retries the same command with variations for a while:
dumpobj only lists field values on a Full dump. On a Heap or Mini dump it prints three lines and the tool
exits: ask for a Full dump instead of retrying
#Dumps contain secrets
This post is the one where it matters most, because dumpobj is exactly the command that prints field values. A Full dump contains everything the process had in memory: connection strings, tokens, decrypted configuration, customer data, and the content of every request in flight.
- Treat a dump as production data. It should not be committed, uploaded to a file-sharing service, or attached to an issue.
--type Triage (or DOTNET_DbgMiniDumpType=3) removes personal information when the dump has to travel, at the cost of the field values this post relies on. - The dump file never leaves the machine with this workflow, but the output of the commands does. An agent that reads a
HttpRequest object can send a bearer token to a model provider. Decide which commands you let it run on a dump that holds real data, and use a local model when nothing may leave your infrastructure.
#What the agent gets wrong, and how to fix it
Every mistake below has a fix, and it is always one of three things: a line in the skill, a better prompt, or one command you run yourself.
It invents output. A model that has seen thousands of SOS transcripts can write a table that looks exactly right for a command it never ran, with plausible addresses in it.
Fix: rules 1 and 2 of the skill make it put the command next to every claim, and step 7 is you running one of those commands again. An address that does not come back is the tell.
It answers from the source code. Given a dump and a repository, an agent often finds a plausible bug in Program.cs and presents it as the explanation, without checking that the dump agrees.
Fix: rule 3 in the skill, and a prompt that asks something only the dump knows. "Why does it crash?" can be answered from the source, so the agent will. "Find which input caused it" cannot, so it has to run dumpobj.
It retries dumpobj on a Heap dump, with small variations, for a while.
Fix: the Known limits line quoted in step 4 tells it to stop and ask for a Full dump. The fix on your side is to collect one in the first place, with DOTNET_DbgMiniDumpType=4.
The next post in this series reuses the same skill, unchanged, on a process that does not crash at all, and does not do anything either: a deadlock.
Do you have a question or a suggestion about this post? Contact me!