# Filtering Telemetry in .NET

## Background

Telemetry is crucial to observability, but too much telemetry can drown out important signals & can lead to noisy logs & spending more $$$ to store them in Azure Log Analytics, Splunk, DynaTrace, DataDog, etc.

This post details some guidelines to reduce the amount of telemetry generated by a .NET app using OpenTelemetry. The guidance here is based on an ASP.NET Core Web API project using .NET Aspire deployed to Azure Container Apps, but most of the ideas are generally applicable.

## 1) Disable Console Logging

By default, when using [Host.CreateApplicationBuilder](https://learn.microsoft.com/en-us/dotnet/api/microsoft.extensions.hosting.host.createapplicationbuilder) or [WebApplication.CreateBuilder](https://learn.microsoft.com/en-us/dotnet/api/microsoft.aspnetcore.builder.webapplication.createbuilder) which most of the built-in .NET templates use, the [Console Logging Provider](https://learn.microsoft.com/en-us/dotnet/core/extensions/logging-providers#console) is registered to output logs to `stdout`. When using Azure Container Apps & you're a good observability citizen, you likely have configured Container Apps to [send console logs to Log Analytics](https://learn.microsoft.com/en-us/azure/container-apps/log-monitoring?tabs=bash#console-logs). When the app is also instrumented using OpenTelemetry & configured to export logs to Application Insights/Azure Monitor - as is configured in the .NET Aspire `ServiceDefaults` project - these logs are duplicated as they are exported from both the Container App console logs as well as directly from your app to Application Insights & ultimately into Log Analytics.

To address this duplication, add the following before adding the OpenTelemetry logger (`builder.Logging.AddOpenTelemetry()`):

```csharp
builder.Logging.ClearProviders();
```

This will remove the Console logging provider & only export logs using OpenTelemetry. Be aware that this will prevent logs from being output to the console & showing up in the Container App’s Log Stream, so errors which occur on startup before the OpenTelemetry logger is configured will be lost & can make diagnosing container startup crashes more difficult.

In terms of other methods for collecting telemetry from Azure Container Apps specifically, it’s worthwhile to call out that there is currently a [preview feature for using OpenTelemetry agents](https://learn.microsoft.com/en-us/azure/container-apps/opentelemetry-agents) in Container Apps.

## 2) Filtering Traces

A common requirement when it comes to filtering telemetry is excluding telemetry which is generated as the result of calling a health check endpoint. Usually, these health check signals add noise/cost & are not all that useful/valuable.

OpenTelemetry [supports filtering Traces](https://github.com/open-telemetry/opentelemetry-dotnet/blob/main/docs/trace/customizing-the-sdk/README.md#processors--exporters) by using a `Processor` on the `Activity` type. We can combine a Processor with `IHttpContextAccessor` to filter out traces which originate from known health check request paths by setting the Activity's `ActivityTraceFlags` to prevent the trace from being exported:

```csharp
internal sealed class HealthCheckTraceFilterProcessor(IHttpContextAccessor httpContextAccessor)
    : BaseProcessor<Activity>
{
    private readonly IHttpContextAccessor _httpContextAccessor = httpContextAccessor;

    public override void OnEnd(Activity activity)
    {
        if (!OKtoSend(activity))
        {
            activity.ActivityTraceFlags &= ~ActivityTraceFlags.Recorded;
        }
    }

    private bool OKtoSend(Activity activity)
    {
        var excludedHealthCheckPaths = new[]
        {
            "/health/live",
            "/health/ready"
        };

        return !excludedHealthCheckPaths.Any(path =>
            _httpContextAccessor.HttpContext?.Request.Path.Value?.Equals(path,
                StringComparison.OrdinalIgnoreCase) ?? false);
    }
}
```

And add the processor to the tracing configuration:

```csharp
builder.Services.AddHttpContextAccessor();
builder.Services.AddSingleton<HealthCheckTraceFilterProcessor>();

builder.Services.AddOpenTelemetry()
    .WithMetrics(metrics =>
    {
        ...
    })
    .WithTracing(tracing =>
    {
        using var scope = sp.CreateScope();
        tracing.AddProcessor(scope.ServiceProvider.GetRequiredService<HealthCheckTraceFilterProcessor>());

        ...
    });
```

When the app uses a SQL database, it's common to have instrumentation for SQL queries using [OpenTelemetry.Instrumentation.SqlClient](https://github.com/open-telemetry/opentelemetry-dotnet-contrib/blob/main/src/OpenTelemetry.Instrumentation.SqlClient/README.md). This instrumentation is included when using .NET Aspire integrations such as the [.NET Aspire SQL Server integration](https://learn.microsoft.com/en-us/dotnet/aspire/database/sql-server-integration) or the [.NET Aspire SQL Server Entity Framework Core integration](https://learn.microsoft.com/en-us/dotnet/aspire/database/sql-server-entity-framework-integration).

In one of our applications, we use [MassTransit with the Transactional Outbox](https://masstransit.io/documentation/patterns/transactional-outbox) enabled & this requires periodically polling specific tables in the database for messages to send to the broker. These SQL queries generate traces which are not useful unless they fail. We can use a combination of **Enrichment** & a `Processor` to identify these traces & filter them accordingly.

First, we need to disable the tracing configuration implemented by the .NET Aspire integration as we will be customising it ourselves (the example below is for the .NET Aspire SQL Server Entity Framework Core integration):

```csharp
builder.EnrichSqlServerDbContext<AppDbContext>(configure =>
{
    configure.DisableTracing = true;
});
```

Secondly, when configuring tracing, the `OpenTelemetry.Instrumentation.SqlClient` package needs to be installed to allow customising the traces it generates. Then we enrich the traces to set a custom tag for queries to known MassTransit outbox tables:

```csharp
tracing.AddSource(builder.Environment.ApplicationName)
    ...
    ...
    .AddSqlClientInstrumentation(opt =>
    {
        opt.Enrich = (Activity activity, string eventName, object rawObject) =>
        {
            if (!eventName.Equals("OnCustom")) return;
            if (rawObject is not SqlCommand cmd) return;
            if (cmd.CommandText.Contains("InboxState") || cmd.CommandText.Contains("OutboxState"))
            {
                activity.SetTag("masstransit.outbox.query", true);
            }
        }
        ;
    });
```

Then we can write a `Processor` which filters based on our custom tag:

```csharp
internal sealed class MassTransitOutboxDbOperationTraceFilterProcessor : BaseProcessor<Activity>
{
    public override void OnEnd(Activity activity)
    {
        if (activity.TagObjects.Any(kv => kv.Key == "masstransit.outbox.query"))
            activity.ActivityTraceFlags &= ~ActivityTraceFlags.Recorded;
    }
}
```

And lastly, add it to our tracing configuration:

```csharp
.WithTracing(tracing =>
{
    tracing.AddProcessor(new MassTransitOutboxDbOperationTraceFilterProcessor());
   ...
})
```

## 3) Filtering Logs

First, some background: OpenTelemetry does support filtering logs, but the mechanism for doing so is not as straightforward as trace filtering due to the way that OpenTelemetry integrates with the built-in `ILogger`. There are many GitHub issues ([1](https://github.com/open-telemetry/opentelemetry-dotnet/issues/5924), [2](https://github.com/open-telemetry/opentelemetry-dotnet/issues/5250), [3](https://github.com/Azure/azure-sdk-for-net/issues/32276)) I referenced when going down this rabbit hole. Long story short, log filtering involves wrapping an OpenTelemetry exporter with a filtering `Processor` to prevent exporting the logs. This is what it would look like (taken from one of the above issues):

```csharp
    configure.AddProcessor(new FilteringBatchLogRecordProcessor(ShouldUpload, exporter));

    internal sealed class FilteringBatchLogRecordProcessor : BatchLogRecordExportProcessor
    {
        private readonly Func<LogRecord, bool> filter;

        public FilteringBatchLogRecordProcessor(Func<LogRecord, bool> filter, BaseExporter<LogRecord> exporter)
            : base(exporter)
        {
            this.filter = filter;
        }

        public override void OnEnd(LogRecord logRecord)
        {
            // Call the underlying processor
            // only if the Filter returns true.
            if (this.filter(logRecord))
            {
                base.OnEnd(logRecord);
            }
        }
    }
```

Testing this approach in a .NET Aspire project, I could filter logs when exporting to the local OTLP Endpoint configured by .NET Aspire:

```csharp
var customOtlpExporter = new OtlpLogExporter(new OtlpExporterOptions
{
    Endpoint = new Uri(builder.Configuration["OTEL_EXPORTER_OTLP_ENDPOINT"])
});

...

builder.Logging
    .AddOpenTelemetry(logging =>
{
    logging.AddProcessor(new FilteringBatchLogRecordProcessor((log) => false, customOtlpExporter));

    ...
});
```

The key thing to note in the above snippet is that in order to wrap an exporter with the filtering processor, one has to create an instance of an exporter - in this case the `OtlpLogExporter` to pass it to the `FilteringBatchLogRecordProcessor`. However, when trying to configure this for the Azure Monitor exporter, I found that it simply wasn't possible because one can't initialize a new `AzureMonitorLogExporter` because its constructor is `internal`. This was highlighted in one of the above GitHub issues & in fact, a [pull request was created to make the constructor public](https://github.com/Azure/azure-sdk-for-net/pull/35568) but was closed without reason.

With this setback, I decided to explore alternative filtering mechanisms to the native OpenTelemetry processors.

The built-in .NET `ILogger` is limited in terms of log filtering in that it can only filter based on log category. This makes it difficult to filter logs based on some other criteria such as request path or the content of the log message. There is currently an [open issue on GitHub](https://github.com/dotnet/runtime/issues/82465) to extend `ILogger` to make this easier in future.

To support more advanced filtering capabilities, we can use [Serilog](https://serilog.net/) in order to achieve our health check endpoint filtering use case.

Serilog has an `ILogEventFilter` interface which can be used to implement a filter which operates on health check requests:

```csharp
public class HealthCheckLogsFilter(IHttpContextAccessor httpContextAccessor) : ILogEventFilter
{
    public bool IsEnabled(LogEvent logEvent)
    {
        if (httpContextAccessor.HttpContext is null)
            return true;

        var healthCheckPaths = new[]
        {
            "/health/live",
            "/health/ready"
        };

        var matchesHealthCheckPath = healthCheckPaths.Any(path =>
            httpContextAccessor.HttpContext?.Request.Path.Value?.Equals(path,
                StringComparison.OrdinalIgnoreCase) ?? true);

        return !(matchesHealthCheckPath && logEvent.Level == LogEventLevel.Information);
    }
}
```

Which can be wired up to the Serilog log configuration:

```csharp
builder.Services.AddSingleton<HealthCheckLogsFilter>();

builder.Services.AddSerilog((services, lc) =>
{
    ...
    lc.Filter.With(services.GetRequiredService<HealthCheckLogsFilter>())
    ...
});
```

The same MassTransit outbox SQL queries mentioned above also generate logs for each query in addition to traces. These logs all have the category `Microsoft.EntityFrameworkCore.Database.Command`, but we don't necessarily want to filter out all logs from this category as this will also exclude logs from other processes which we care about. To specifically target only these MassTransit query logs, we can use an `Excluding` log filter with Serilog.

Firstly, we create the exclusion method which checks for logs which contain the SQL operation on known MassTransit outbox tables & only if the Log Level is `Information` so that when one of these queries throw an error, we don't filter them out:

```csharp
private static bool MassTransitOutboxDbCommandLogExclusion(LogEvent logEvent)
{
    const string logCategory = """
                               "Microsoft.EntityFrameworkCore.Database.Command"
                               """;

    var sourceContext = logEvent.Properties.FirstOrDefault(l => l.Key == "SourceContext").Value.ToString();

    if (sourceContext != logCategory) return false;

    var commandText = logEvent.Properties.FirstOrDefault(l => l.Key == "commandText").Value?.ToString();

    if (commandText is null) return false;

    string[] massTransitOutboxTableNames = ["InboxState", "OutboxState"];

    var isMassTransitDbCommand = !string.IsNullOrWhiteSpace(commandText) && massTransitOutboxTableNames.Any(commandText.Contains);

    return isMassTransitDbCommand && logEvent is { Level: LogEventLevel.Information };
}
```

Then we register this method with Serilog using `.Filter.ByExcluding()`:

```csharp
builder.Services.AddSerilog((services, lc) =>
{
    ...
    .Filter.ByExcluding(MassTransitOutboxDbCommandLogExclusion);

});
```

## 4) Filtering Metrics

For some interesting history around metric filtering, [see this GitHub issue](https://github.com/open-telemetry/opentelemetry-dotnet-contrib/issues/1765).

To satisfy the requirement of filtering metrics for health check endpoints, [this PR](https://github.com/dotnet/aspnetcore/pull/56036) introduced the capability to opt-out of metric collection on a per-endpoint basis.

This can be added to our health check endpoints to disable metrics:

```csharp
healthChecks.MapHealthChecks("/health/ready")
    .DisableHttpMetrics();

healthChecks.MapHealthChecks("/health/live", new HealthCheckOptions
{
    Predicate = static r => r.Tags.Contains("live")
})
.DisableHttpMetrics();
```

It's important to note that the `DisableHttpMetrics()` method is only available in .NET 9 & above.

## Other Considerations

During the wire-up of all these various filters, there were some finer points of configuration needed to address the following:

1. Sending Serilog logs to the local OTEL endpoint when running on a local machine so that logs appear in the .NET Aspire dashboard.
    
2. Using the `Azure.Monitor.OpenTelemetry.Exporter` package instead of the `Azure.Monitor.OpenTelemetry.AspNetCore` package as the latter overrides some tracing & logging configuration which un-does a lot of our hard work.
    

I've enclosed the full configuration below to show the full setup of all the telemetry configuration:

```csharp
public static class TelemetryConfigurationExtensions
{
    public static IHostApplicationBuilder AddTelemetryConfiguration(this IHostApplicationBuilder builder)
    {
        // Required so that traces for health checks are not recorded
        builder.Services.AddHttpContextAccessor();

        builder.Services.AddSingleton<HealthCheckLogsFilter>();
        builder.Services.AddSingleton<HealthCheckTraceFilterProcessor>();

        builder.Services.AddSerilog((services, lc) =>
        {
            lc.MinimumLevel.Override("Microsoft", LogEventLevel.Information)
                .MinimumLevel.Override("Microsoft.AspNetCore", LogEventLevel.Warning)
                .Filter.With(services.GetRequiredService<HealthCheckLogsFilter>())
                .Enrich.FromLogContext()
                .Filter.ByExcluding(MassTransitOutboxDbCommandLogExclusion);

        }, writeToProviders: true);

        builder.Services.AddOpenTelemetry()
            .ConfigureResource(resourceBuilder =>
            {
                resourceBuilder
                    .AddAzureContainerAppsDetector(); // Used to set the Cloud Role Name and the Cloud Role Instance properties
            })
            .WithMetrics(metrics =>
            {
                metrics.AddAspNetCoreInstrumentation()
                    .AddHttpClientInstrumentation()
                    .AddRuntimeInstrumentation();
            })
            .WithTracing(tracing =>
            {
                tracing.AddProcessor(new MassTransitOutboxDbOperationTraceFilterProcessor());

                tracing.AddSource(builder.Environment.ApplicationName)
                    .AddAspNetCoreInstrumentation()
                    .AddHttpClientInstrumentation()
                    .AddSqlClientInstrumentation(opt =>
                    {
                        opt.Enrich = SqlClientMassTransitOutboxQueryEnrichment;
                    });
            });

        var applicationInsightsConnectionString = builder.Configuration["APPLICATIONINSIGHTS_CONNECTION_STRING"];

        builder.Services.ConfigureOpenTelemetryTracerProvider((sp, tracerProviderBuilder) =>
        {
            using var scope = sp.CreateScope();

            tracerProviderBuilder.AddProcessor(scope.ServiceProvider.GetRequiredService<HealthCheckTraceFilterProcessor>());

            if (builder.Environment.IsProduction())
            {
                tracerProviderBuilder.AddAzureMonitorTraceExporter(options =>
                {
                    options.ConnectionString = applicationInsightsConnectionString;
                });
            }
        });

        builder.Services.ConfigureOpenTelemetryMeterProvider((_, meterProviderBuilder) =>
        {
            if (builder.Environment.IsProduction())
            {
                meterProviderBuilder.AddAzureMonitorMetricExporter(options =>
                {
                    options.ConnectionString = applicationInsightsConnectionString;
                });
            }
        });

        // Clear all existing logging providers so that logs are only sent to App Insights/Log Analytics
        // This prevents duplicate logs from logging to the console & also being forwarded to Log Analytics
        builder.Logging.ClearProviders();

        builder.Logging.AddOpenTelemetry(logging =>
        {
            logging.IncludeFormattedMessage = true;
            logging.IncludeScopes = true;

            if (builder.Environment.IsProduction())
            {
                logging.AddAzureMonitorLogExporter(x =>
                {
                    x.ConnectionString = applicationInsightsConnectionString;
                });
            }
        });

        var useOtlpExporter = !string.IsNullOrWhiteSpace(builder.Configuration["OTEL_EXPORTER_OTLP_ENDPOINT"]);

        if (useOtlpExporter)
        {
            builder.Services.AddOpenTelemetry()
                .UseOtlpExporter();
        }

        return builder;
    }

    private static void SqlClientMassTransitOutboxQueryEnrichment(Activity activity, string eventName, object rawObject)
    {
        if (!eventName.Equals("OnCustom")) return;
        if (rawObject is not SqlCommand cmd) return;
        if (cmd.CommandText.Contains("InboxState") || cmd.CommandText.Contains("OutboxState"))
        {
            activity.SetTag("masstransit.outbox.query", true);
        }
    }

    private static bool MassTransitOutboxDbCommandLogExclusion(LogEvent logEvent)
    {
        const string logCategory = """
                                   "Microsoft.EntityFrameworkCore.Database.Command"
                                   """;

        var sourceContext = logEvent.Properties.FirstOrDefault(l => l.Key == "SourceContext").Value.ToString();

        if (sourceContext != logCategory) return false;

        var commandText = logEvent.Properties.FirstOrDefault(l => l.Key == "commandText").Value?.ToString();

        if (commandText is null) return false;

        string[] massTransitOutboxTableNames = ["InboxState", "OutboxState"];

        var isMassTransitDbCommand = !string.IsNullOrWhiteSpace(commandText) && massTransitOutboxTableNames.Any(commandText.Contains);

        return isMassTransitDbCommand && logEvent is { Level: LogEventLevel.Information };
    }
}

internal sealed class MassTransitOutboxDbOperationTraceFilterProcessor : BaseProcessor<Activity>
{
    public override void OnEnd(Activity activity)
    {
        if (activity.TagObjects.Any(kv => kv.Key == "masstransit.outbox.query"))
            activity.ActivityTraceFlags &= ~ActivityTraceFlags.Recorded;
    }
}

internal sealed class HealthCheckTraceFilterProcessor(IHttpContextAccessor httpContextAccessor) : BaseProcessor<Activity>
{
    public override void OnEnd(Activity activity)
    {
        var excludedHealthCheckPaths = new[]
        {
            "/health/live",
            "/health/ready"
        };

        if (excludedHealthCheckPaths.Any(path =>
                httpContextAccessor.HttpContext?.Request.Path.Value?.Equals(path,
                    StringComparison.OrdinalIgnoreCase) ?? false))
        {
            activity.ActivityTraceFlags &= ~ActivityTraceFlags.Recorded;
        }
    }
}

public class HealthCheckLogsFilter(IHttpContextAccessor httpContextAccessor) : ILogEventFilter
{
    public bool IsEnabled(LogEvent logEvent)
    {
        if (httpContextAccessor.HttpContext is null)
            return true;

        var healthCheckPaths = new[]
        {
            "/health/live",
            "/health/ready"
        };

        var matchesHealthCheckPath = healthCheckPaths.Any(path =>
            httpContextAccessor.HttpContext?.Request.Path.Value?.Equals(path,
                StringComparison.OrdinalIgnoreCase) ?? true);

        return !(matchesHealthCheckPath && logEvent.Level == LogEventLevel.Information);
    }
}
```

## Conclusion

These techniques may not necessarily be the most optimal way to achieve the desired outcome, or quite the *OpenTelemetry-native* way, so if there are better approaches, please let me know!

OpenTelemetry is indeed enormously powerful & quite customisable in how you can adapt it to your use case. Though it’s worth noting that there are some gotchas with how it integrates into the broader .NET ecosystem which may be unintuitive. This post documents some learnings in my telemetry journey & hopefully these learnings can help cut down on unnecessary telemetry & hopefully save some $$$ in the process.