When building distributed systems, maintaining observability across process boundaries is crucial for understanding the flow of requests through your application. OpenTelemetry provides a standard way to propagate tracing context, but implementing it correctly requires understanding several key concepts. This post explains how to propagate telemetry context in .NET applications, particularly when using messaging systems, background jobs, or other asynchronous processing patterns.
#Understanding the problem
In a typical web application, OpenTelemetry automatically propagates tracing context through HTTP headers. When Service A calls Service B via HTTP, the trace context flows seamlessly. However, this automatic propagation doesn't work when you:
- Send messages through a queue (Azure Service Bus, RabbitMQ, Kafka)
- Schedule background jobs (Hangfire, Quartz.NET)
- Store work items in a database for later processing
- Use any other asynchronous communication pattern
Without proper propagation, each process starts a new, disconnected trace, making it impossible to see the complete request flow in your observability platform.
#Setting up OpenTelemetry
First, ensure you have the necessary NuGet package:
XML
<PackageReference Include="OpenTelemetry.Api" />
This package provides the core APIs for context propagation without requiring you to set up the full OpenTelemetry SDK in every part of your application.
#Key concepts
##Activity and ActivityContext
In .NET, Activity represents a single operation in your application. The ActivityContext contains the trace ID, span ID, and trace flags needed to link distributed operations. You access the current activity through Activity.Current.
##Propagation Context
PropagationContext is OpenTelemetry's representation of the context to propagate. It contains both the ActivityContext (for tracing) and Baggage (for cross-cutting concerns).
##Baggage vs. Context
Understanding the difference between these two concepts is essential:
- ActivityContext: Contains tracing information (trace ID, span ID, trace flags). This is what links operations together in your distributed trace. It's always propagated.
- Baggage: Key-value pairs that travel with the request. Unlike trace context, baggage is optional and should be used sparingly because it increases payload size and can expose sensitive information. Examples include correlation IDs, user IDs, or feature flags.
For security and performance reasons, you should be very cautious about propagating baggage. In most cases, propagating just the trace context is sufficient.
##ActivityLink
An ActivityLink creates a relationship between spans in different traces. This is particularly useful for scenarios like:
- Processing messages from a queue where multiple inputs contribute to one output
- Batch processing where you want to link individual items to the batch operation
- Fan-out scenarios where one request triggers multiple parallel operations
Unlike parent-child relationships, links allow you to reference spans from multiple traces or create non-hierarchical relationships.
#When to propagate telemetry
You should propagate telemetry context in these scenarios:
- Message queues: When sending messages to Azure Service Bus, RabbitMQ, Kafka, or similar systems
- Background jobs: When scheduling work with Hangfire, Quartz.NET, or similar frameworks
- Database-backed queues: When storing work items for later processing
- Cache-based communication: When using Redis or other caches for inter-process communication
- File-based communication: When writing files that another process will consume
The key indicator is any time you cross a process boundary asynchronously, breaking automatic context propagation.
#Implementing telemetry propagation
Here's a reusable helper class that handles context propagation:
C#
using System.Diagnostics;
using System.Text.Json;
using OpenTelemetry;
using OpenTelemetry.Context.Propagation;
public sealed class TelemetryPropagator
{
private static readonly Dictionary<string, HashSet<string>> Empty = [];
private readonly Dictionary<string, HashSet<string>> _data;
private TelemetryPropagator(Dictionary<string, HashSet<string>> data)
{
_data = data;
}
public string ToJson()
{
return JsonSerializer.Serialize(_data);
}
public override string ToString()
{
return ToJson();
}
public static TelemetryPropagator FromJson(string? json)
{
if (json is null)
return new(Empty);
var data = JsonSerializer.Deserialize<Dictionary<string, HashSet<string>>>(json);
return new(data ?? Empty);
}
public static TelemetryPropagator? Create()
{
var activityContext = Activity.Current?.Context ?? default;
if (activityContext == default)
return null;
var data = new Dictionary<string, HashSet<string>>(StringComparer.Ordinal);
var baggageIgnoredForPayloadSizeAndSecurityConsiderations = default(Baggage);
var propagationContext = new PropagationContext(activityContext, baggageIgnoredForPayloadSizeAndSecurityConsiderations);
Propagators.DefaultTextMapPropagator.Inject(propagationContext, data, (item, name, value) =>
{
if (data.TryGetValue(name, out var node))
{
node.Add(value);
}
else
{
data[name] = new HashSet<string>(capacity: 1, StringComparer.Ordinal) { value };
}
});
return new TelemetryPropagator(data);
}
public PropagationContext ExtractContext()
{
return Propagators.DefaultTextMapPropagator.Extract(default, this, (item, name) =>
{
if (item._data.TryGetValue(name, out var node))
return node;
return null;
});
}
public IEnumerable<ActivityLink>? CreateActivityLink()
{
var propagationContext = ExtractContext();
return propagationContext == default ? null : [new ActivityLink(propagationContext.ActivityContext)];
}
public static IEnumerable<ActivityLink>? CreateActivityLinkFromJson(string? json)
{
if (json is null)
return null;
var telemetry = FromJson(json);
return telemetry.CreateActivityLink();
}
}
#Using the propagator
##Sending side (producer)
When sending a message or scheduling a job, capture the context:
C#
// When sending a message
var telemetryContext = TelemetryPropagator.Create();
var message = new MyMessage
{
// Your message properties
Foo = "Bar",
// Propagate the telemetry context
TelemetryContext = telemetryContext?.ToJson()
};
await messageQueue.SendAsync(message);
##Receiving side (consumer)
When processing the message, restore the context:
C#
// Option 1: Continue the trace (parent-child relationship)
var propagationContext = TelemetryPropagator.FromJson(message.TelemetryContext).ExtractContext();
using var activity = ActivitySource.StartActivity(
"ProcessMessage",
ActivityKind.Consumer,
propagationContext.ActivityContext);
// Your processing logic
// Option 2: Create a link (for non-hierarchical relationships)
var links = TelemetryPropagator.CreateActivityLinkFromJson(message.TelemetryContext);
using var activity = ActivitySource.StartActivity(
"ProcessMessage",
ActivityKind.Consumer,
parentContext: default,
links: links);
// Your processing logic
##Choosing between parent-child and links
Use parent-child relationships (Option 1) when:
- The consumer is a direct continuation of the producer's work
- You want a single, linear trace
- The relationship is 1:1 or 1:many
Use activity links (Option 2) when:
- Processing multiple messages into one operation (many:1)
- Creating non-hierarchical relationships
- You want separate traces but still show the relationship
- Processing can happen in any order
#Additional resources
Do you have a question or a suggestion about this post? Contact me!
