Filtering Telemetry in .NET

Filtering Telemetry in .NET

Background

Telemetry is crucial to observability, but too much telemetry can drown out important signals & can lead to noisy logs & spending more $$$ to store them in Azure Log Analytics, Splunk, DynaTrace, DataDog, etc.

This post details some guidelines to reduce the amount of telemetry generated by a .NET app using OpenTelemetry. The guidance here is based on an ASP.NET Core Web API project using .NET Aspire deployed to Azure Container Apps, but most of the ideas are generally applicable.

1) Disable Console Logging

By default, when using Host.CreateApplicationBuilder or WebApplication.CreateBuilder which most of the built-in .NET templates use, the Console Logging Provider is registered to output logs to stdout. When using Azure Container Apps & you're a good observability citizen, you likely have configured Container Apps to send console logs to Log Analytics. When the app is also instrumented using OpenTelemetry & configured to export logs to Application Insights/Azure Monitor - as is configured in the .NET Aspire ServiceDefaults project - these logs are duplicated as they are exported from both the Container App console logs as well as directly from your app to Application Insights & ultimately into Log Analytics.

To address this duplication, add the following before adding the OpenTelemetry logger (builder.Logging.AddOpenTelemetry()):

builder.Logging.ClearProviders();

This will remove the Console logging provider & only export logs using OpenTelemetry. Be aware that this will prevent logs from being output to the console & showing up in the Container App’s Log Stream, so errors which occur on startup before the OpenTelemetry logger is configured will be lost & can make diagnosing container startup crashes more difficult.

In terms of other methods for collecting telemetry from Azure Container Apps specifically, it’s worthwhile to call out that there is currently a preview feature for using OpenTelemetry agents in Container Apps.

2) Filtering Traces

A common requirement when it comes to filtering telemetry is excluding telemetry which is generated as the result of calling a health check endpoint. Usually, these health check signals add noise/cost & are not all that useful/valuable.

OpenTelemetry supports filtering Traces by using a Processor on the Activity type. We can combine a Processor with IHttpContextAccessor to filter out traces which originate from known health check request paths by setting the Activity's ActivityTraceFlags to prevent the trace from being exported:

internal sealed class HealthCheckTraceFilterProcessor(IHttpContextAccessor httpContextAccessor)
    : BaseProcessor<Activity>
{
    private readonly IHttpContextAccessor _httpContextAccessor = httpContextAccessor;

    public override void OnEnd(Activity activity)
    {
        if (!OKtoSend(activity))
        {
            activity.ActivityTraceFlags &= ~ActivityTraceFlags.Recorded;
        }
    }

    private bool OKtoSend(Activity activity)
    {
        var excludedHealthCheckPaths = new[]
        {
            "/health/live",
            "/health/ready"
        };

        return !excludedHealthCheckPaths.Any(path =>
            _httpContextAccessor.HttpContext?.Request.Path.Value?.Equals(path,
                StringComparison.OrdinalIgnoreCase) ?? false);
    }
}

And add the processor to the tracing configuration:

builder.Services.AddHttpContextAccessor();
builder.Services.AddSingleton<HealthCheckTraceFilterProcessor>();

builder.Services.AddOpenTelemetry()
    .WithMetrics(metrics =>
    {
        ...
    })
    .WithTracing(tracing =>
    {
        using var scope = sp.CreateScope();
        tracing.AddProcessor(scope.ServiceProvider.GetRequiredService<HealthCheckTraceFilterProcessor>());

        ...
    });

When the app uses a SQL database, it's common to have instrumentation for SQL queries using OpenTelemetry.Instrumentation.SqlClient. This instrumentation is included when using .NET Aspire integrations such as the .NET Aspire SQL Server integration or the .NET Aspire SQL Server Entity Framework Core integration.

In one of our applications, we use MassTransit with the Transactional Outbox enabled & this requires periodically polling specific tables in the database for messages to send to the broker. These SQL queries generate traces which are not useful unless they fail. We can use a combination of Enrichment & a Processor to identify these traces & filter them accordingly.

First, we need to disable the tracing configuration implemented by the .NET Aspire integration as we will be customising it ourselves (the example below is for the .NET Aspire SQL Server Entity Framework Core integration):

builder.EnrichSqlServerDbContext<AppDbContext>(configure =>
{
    configure.DisableTracing = true;
});

Secondly, when configuring tracing, the OpenTelemetry.Instrumentation.SqlClient package needs to be installed to allow customising the traces it generates. Then we enrich the traces to set a custom tag for queries to known MassTransit outbox tables:

tracing.AddSource(builder.Environment.ApplicationName)
    ...
    ...
    .AddSqlClientInstrumentation(opt =>
    {
        opt.Enrich = (Activity activity, string eventName, object rawObject) =>
        {
            if (!eventName.Equals("OnCustom")) return;
            if (rawObject is not SqlCommand cmd) return;
            if (cmd.CommandText.Contains("InboxState") || cmd.CommandText.Contains("OutboxState"))
            {
                activity.SetTag("masstransit.outbox.query", true);
            }
        }
        ;
    });

Then we can write a Processor which filters based on our custom tag:

internal sealed class MassTransitOutboxDbOperationTraceFilterProcessor : BaseProcessor<Activity>
{
    public override void OnEnd(Activity activity)
    {
        if (activity.TagObjects.Any(kv => kv.Key == "masstransit.outbox.query"))
            activity.ActivityTraceFlags &= ~ActivityTraceFlags.Recorded;
    }
}

And lastly, add it to our tracing configuration:

.WithTracing(tracing =>
{
    tracing.AddProcessor(new MassTransitOutboxDbOperationTraceFilterProcessor());
   ...
})

3) Filtering Logs

First, some background: OpenTelemetry does support filtering logs, but the mechanism for doing so is not as straightforward as trace filtering due to the way that OpenTelemetry integrates with the built-in ILogger. There are many GitHub issues (1, 2, 3) I referenced when going down this rabbit hole. Long story short, log filtering involves wrapping an OpenTelemetry exporter with a filtering Processor to prevent exporting the logs. This is what it would look like (taken from one of the above issues):

    configure.AddProcessor(new FilteringBatchLogRecordProcessor(ShouldUpload, exporter));

    internal sealed class FilteringBatchLogRecordProcessor : BatchLogRecordExportProcessor
    {
        private readonly Func<LogRecord, bool> filter;

        public FilteringBatchLogRecordProcessor(Func<LogRecord, bool> filter, BaseExporter<LogRecord> exporter)
            : base(exporter)
        {
            this.filter = filter;
        }

        public override void OnEnd(LogRecord logRecord)
        {
            // Call the underlying processor
            // only if the Filter returns true.
            if (this.filter(logRecord))
            {
                base.OnEnd(logRecord);
            }
        }
    }

Testing this approach in a .NET Aspire project, I could filter logs when exporting to the local OTLP Endpoint configured by .NET Aspire:

var customOtlpExporter = new OtlpLogExporter(new OtlpExporterOptions
{
    Endpoint = new Uri(builder.Configuration["OTEL_EXPORTER_OTLP_ENDPOINT"])
});

...

builder.Logging
    .AddOpenTelemetry(logging =>
{
    logging.AddProcessor(new FilteringBatchLogRecordProcessor((log) => false, customOtlpExporter));

    ...
});

The key thing to note in the above snippet is that in order to wrap an exporter with the filtering processor, one has to create an instance of an exporter - in this case the OtlpLogExporter to pass it to the FilteringBatchLogRecordProcessor. However, when trying to configure this for the Azure Monitor exporter, I found that it simply wasn't possible because one can't initialize a new AzureMonitorLogExporter because its constructor is internal. This was highlighted in one of the above GitHub issues & in fact, a pull request was created to make the constructor public but was closed without reason.

With this setback, I decided to explore alternative filtering mechanisms to the native OpenTelemetry processors.

The built-in .NET ILogger is limited in terms of log filtering in that it can only filter based on log category. This makes it difficult to filter logs based on some other criteria such as request path or the content of the log message. There is currently an open issue on GitHub to extend ILogger to make this easier in future.

To support more advanced filtering capabilities, we can use Serilog in order to achieve our health check endpoint filtering use case.

Serilog has an ILogEventFilter interface which can be used to implement a filter which operates on health check requests:

public class HealthCheckLogsFilter(IHttpContextAccessor httpContextAccessor) : ILogEventFilter
{
    public bool IsEnabled(LogEvent logEvent)
    {
        if (httpContextAccessor.HttpContext is null)
            return true;

        var healthCheckPaths = new[]
        {
            "/health/live",
            "/health/ready"
        };

        var matchesHealthCheckPath = healthCheckPaths.Any(path =>
            httpContextAccessor.HttpContext?.Request.Path.Value?.Equals(path,
                StringComparison.OrdinalIgnoreCase) ?? true);

        return !(matchesHealthCheckPath && logEvent.Level == LogEventLevel.Information);
    }
}

Which can be wired up to the Serilog log configuration:

builder.Services.AddSingleton<HealthCheckLogsFilter>();

builder.Services.AddSerilog((services, lc) =>
{
    ...
    lc.Filter.With(services.GetRequiredService<HealthCheckLogsFilter>())
    ...
});

The same MassTransit outbox SQL queries mentioned above also generate logs for each query in addition to traces. These logs all have the category Microsoft.EntityFrameworkCore.Database.Command, but we don't necessarily want to filter out all logs from this category as this will also exclude logs from other processes which we care about. To specifically target only these MassTransit query logs, we can use an Excluding log filter with Serilog.

Firstly, we create the exclusion method which checks for logs which contain the SQL operation on known MassTransit outbox tables & only if the Log Level is Information so that when one of these queries throw an error, we don't filter them out:

private static bool MassTransitOutboxDbCommandLogExclusion(LogEvent logEvent)
{
    const string logCategory = """
                               "Microsoft.EntityFrameworkCore.Database.Command"
                               """;

    var sourceContext = logEvent.Properties.FirstOrDefault(l => l.Key == "SourceContext").Value.ToString();

    if (sourceContext != logCategory) return false;

    var commandText = logEvent.Properties.FirstOrDefault(l => l.Key == "commandText").Value?.ToString();

    if (commandText is null) return false;

    string[] massTransitOutboxTableNames = ["InboxState", "OutboxState"];

    var isMassTransitDbCommand = !string.IsNullOrWhiteSpace(commandText) && massTransitOutboxTableNames.Any(commandText.Contains);

    return isMassTransitDbCommand && logEvent is { Level: LogEventLevel.Information };
}

Then we register this method with Serilog using .Filter.ByExcluding():

builder.Services.AddSerilog((services, lc) =>
{
    ...
    .Filter.ByExcluding(MassTransitOutboxDbCommandLogExclusion);

});

4) Filtering Metrics

For some interesting history around metric filtering, see this GitHub issue.

To satisfy the requirement of filtering metrics for health check endpoints, this PR introduced the capability to opt-out of metric collection on a per-endpoint basis.

This can be added to our health check endpoints to disable metrics:

healthChecks.MapHealthChecks("/health/ready")
    .DisableHttpMetrics();

healthChecks.MapHealthChecks("/health/live", new HealthCheckOptions
{
    Predicate = static r => r.Tags.Contains("live")
})
.DisableHttpMetrics();

It's important to note that the DisableHttpMetrics() method is only available in .NET 9 & above.

Other Considerations

During the wire-up of all these various filters, there were some finer points of configuration needed to address the following:

Sending Serilog logs to the local OTEL endpoint when running on a local machine so that logs appear in the .NET Aspire dashboard.
Using the Azure.Monitor.OpenTelemetry.Exporter package instead of the Azure.Monitor.OpenTelemetry.AspNetCore package as the latter overrides some tracing & logging configuration which un-does a lot of our hard work.

I've enclosed the full configuration below to show the full setup of all the telemetry configuration:

public static class TelemetryConfigurationExtensions
{
    public static IHostApplicationBuilder AddTelemetryConfiguration(this IHostApplicationBuilder builder)
    {
        // Required so that traces for health checks are not recorded
        builder.Services.AddHttpContextAccessor();

        builder.Services.AddSingleton<HealthCheckLogsFilter>();
        builder.Services.AddSingleton<HealthCheckTraceFilterProcessor>();

        builder.Services.AddSerilog((services, lc) =>
        {
            lc.MinimumLevel.Override("Microsoft", LogEventLevel.Information)
                .MinimumLevel.Override("Microsoft.AspNetCore", LogEventLevel.Warning)
                .Filter.With(services.GetRequiredService<HealthCheckLogsFilter>())
                .Enrich.FromLogContext()
                .Filter.ByExcluding(MassTransitOutboxDbCommandLogExclusion);

        }, writeToProviders: true);

        builder.Services.AddOpenTelemetry()
            .ConfigureResource(resourceBuilder =>
            {
                resourceBuilder
                    .AddAzureContainerAppsDetector(); // Used to set the Cloud Role Name and the Cloud Role Instance properties
            })
            .WithMetrics(metrics =>
            {
                metrics.AddAspNetCoreInstrumentation()
                    .AddHttpClientInstrumentation()
                    .AddRuntimeInstrumentation();
            })
            .WithTracing(tracing =>
            {
                tracing.AddProcessor(new MassTransitOutboxDbOperationTraceFilterProcessor());

                tracing.AddSource(builder.Environment.ApplicationName)
                    .AddAspNetCoreInstrumentation()
                    .AddHttpClientInstrumentation()
                    .AddSqlClientInstrumentation(opt =>
                    {
                        opt.Enrich = SqlClientMassTransitOutboxQueryEnrichment;
                    });
            });

        var applicationInsightsConnectionString = builder.Configuration["APPLICATIONINSIGHTS_CONNECTION_STRING"];

        builder.Services.ConfigureOpenTelemetryTracerProvider((sp, tracerProviderBuilder) =>
        {
            using var scope = sp.CreateScope();

            tracerProviderBuilder.AddProcessor(scope.ServiceProvider.GetRequiredService<HealthCheckTraceFilterProcessor>());

            if (builder.Environment.IsProduction())
            {
                tracerProviderBuilder.AddAzureMonitorTraceExporter(options =>
                {
                    options.ConnectionString = applicationInsightsConnectionString;
                });
            }
        });

        builder.Services.ConfigureOpenTelemetryMeterProvider((_, meterProviderBuilder) =>
        {
            if (builder.Environment.IsProduction())
            {
                meterProviderBuilder.AddAzureMonitorMetricExporter(options =>
                {
                    options.ConnectionString = applicationInsightsConnectionString;
                });
            }
        });

        // Clear all existing logging providers so that logs are only sent to App Insights/Log Analytics
        // This prevents duplicate logs from logging to the console & also being forwarded to Log Analytics
        builder.Logging.ClearProviders();

        builder.Logging.AddOpenTelemetry(logging =>
        {
            logging.IncludeFormattedMessage = true;
            logging.IncludeScopes = true;

            if (builder.Environment.IsProduction())
            {
                logging.AddAzureMonitorLogExporter(x =>
                {
                    x.ConnectionString = applicationInsightsConnectionString;
                });
            }
        });

        var useOtlpExporter = !string.IsNullOrWhiteSpace(builder.Configuration["OTEL_EXPORTER_OTLP_ENDPOINT"]);

        if (useOtlpExporter)
        {
            builder.Services.AddOpenTelemetry()
                .UseOtlpExporter();
        }

        return builder;
    }

    private static void SqlClientMassTransitOutboxQueryEnrichment(Activity activity, string eventName, object rawObject)
    {
        if (!eventName.Equals("OnCustom")) return;
        if (rawObject is not SqlCommand cmd) return;
        if (cmd.CommandText.Contains("InboxState") || cmd.CommandText.Contains("OutboxState"))
        {
            activity.SetTag("masstransit.outbox.query", true);
        }
    }

    private static bool MassTransitOutboxDbCommandLogExclusion(LogEvent logEvent)
    {
        const string logCategory = """
                                   "Microsoft.EntityFrameworkCore.Database.Command"
                                   """;

        var sourceContext = logEvent.Properties.FirstOrDefault(l => l.Key == "SourceContext").Value.ToString();

        if (sourceContext != logCategory) return false;

        var commandText = logEvent.Properties.FirstOrDefault(l => l.Key == "commandText").Value?.ToString();

        if (commandText is null) return false;

        string[] massTransitOutboxTableNames = ["InboxState", "OutboxState"];

        var isMassTransitDbCommand = !string.IsNullOrWhiteSpace(commandText) && massTransitOutboxTableNames.Any(commandText.Contains);

        return isMassTransitDbCommand && logEvent is { Level: LogEventLevel.Information };
    }
}

internal sealed class MassTransitOutboxDbOperationTraceFilterProcessor : BaseProcessor<Activity>
{
    public override void OnEnd(Activity activity)
    {
        if (activity.TagObjects.Any(kv => kv.Key == "masstransit.outbox.query"))
            activity.ActivityTraceFlags &= ~ActivityTraceFlags.Recorded;
    }
}

internal sealed class HealthCheckTraceFilterProcessor(IHttpContextAccessor httpContextAccessor) : BaseProcessor<Activity>
{
    public override void OnEnd(Activity activity)
    {
        var excludedHealthCheckPaths = new[]
        {
            "/health/live",
            "/health/ready"
        };

        if (excludedHealthCheckPaths.Any(path =>
                httpContextAccessor.HttpContext?.Request.Path.Value?.Equals(path,
                    StringComparison.OrdinalIgnoreCase) ?? false))
        {
            activity.ActivityTraceFlags &= ~ActivityTraceFlags.Recorded;
        }
    }
}

public class HealthCheckLogsFilter(IHttpContextAccessor httpContextAccessor) : ILogEventFilter
{
    public bool IsEnabled(LogEvent logEvent)
    {
        if (httpContextAccessor.HttpContext is null)
            return true;

        var healthCheckPaths = new[]
        {
            "/health/live",
            "/health/ready"
        };

        var matchesHealthCheckPath = healthCheckPaths.Any(path =>
            httpContextAccessor.HttpContext?.Request.Path.Value?.Equals(path,
                StringComparison.OrdinalIgnoreCase) ?? true);

        return !(matchesHealthCheckPath && logEvent.Level == LogEventLevel.Information);
    }
}

Conclusion

These techniques may not necessarily be the most optimal way to achieve the desired outcome, or quite the OpenTelemetry-native way, so if there are better approaches, please let me know!

OpenTelemetry is indeed enormously powerful & quite customisable in how you can adapt it to your use case. Though it’s worth noting that there are some gotchas with how it integrates into the broader .NET ecosystem which may be unintuitive. This post documents some learnings in my telemetry journey & hopefully these learnings can help cut down on unnecessary telemetry & hopefully save some $$$ in the process.