Telemetry is crucial to observability, but too much telemetry can drown out important signals & can lead to noisy logs & spending more $$$ to store them in Azure Log Analytics, Splunk, DynaTrace, DataDog, etc.

This post details some guidelines to reduce the amount of telemetry generated by a .NET app using OpenTelemetry. The guidance here is based on an ASP.NET Core Web API project using .NET Aspire deployed to Azure Container Apps, but most of the ideas are generally applicable.

1) Disable Console Logging

By default, when using Host.CreateApplicationBuilder or WebApplication.CreateBuilder which most of the built-in .NET templates use, the Console Logging Provider is registered to output logs to stdout. When using Azure Container Apps & you're a good observability citizen, you likely have configured Container Apps to send console logs to Log Analytics. When the app is also instrumented using OpenTelemetry & configured to export logs to Application Insights/Azure Monitor - as is configured in the .NET Aspire ServiceDefaults project - these logs are duplicated as they are exported from both the Container App console logs as well as directly from your app to Application Insights & ultimately into Log Analytics.

To address this duplication, add the following before adding the OpenTelemetry logger (builder.Logging.AddOpenTelemetry()):

builder.Logging.ClearProviders();

This will remove the Console logging provider & only export logs using OpenTelemetry. Be aware that this will prevent logs from being output to the console & showing up in the Container App’s Log Stream, so errors which occur on startup before the OpenTelemetry logger is configured will be lost & can make diagnosing container startup crashes more difficult.

In terms of other methods for collecting telemetry from Azure Container Apps specifically, it’s worthwhile to call out that there is currently a preview feature for using OpenTelemetry agents in Container Apps.

2) Filtering Traces

A common requirement when it comes to filtering telemetry is excluding telemetry which is generated as the result of calling a health check endpoint. Usually, these health check signals add noise/cost & are not all that useful/valuable.

OpenTelemetry supports filtering Traces by using a Processor on the Activity type. We can combine a Processor with IHttpContextAccessor to filter out traces which originate from known health check request paths by setting the Activity's ActivityTraceFlags to prevent the trace from being exported:

internal sealed class HealthCheckTraceFilterProcessor(IHttpContextAccessor httpContextAccessor)
    : BaseProcessor<Activity>
{
    private readonly IHttpContextAccessor _httpContextAccessor = httpContextAccessor;

    public override void OnEnd(Activity activity)
    {
        if (!OKtoSend(activity))
        {
            activity.ActivityTraceFlags &= ~ActivityTraceFlags.Recorded;
        }
    }

    private bool OKtoSend(Activity activity)
    {
        var excludedHealthCheckPaths = new[]
        {
            "/health/live",
            "/health/ready"
        };

        return !excludedHealthCheckPaths.Any(path =>
            _httpContextAccessor.HttpContext?.Request.Path.Value?.Equals(path,
                StringComparison.OrdinalIgnoreCase) ?? false);
    }
}

And add the processor to the tracing configuration:

builder.Services.AddHttpContextAccessor();
builder.Services.AddSingleton<HealthCheckTraceFilterProcessor>();

builder.Services.AddOpenTelemetry()
    .WithMetrics(metrics =>
    {
        ...
    })
    .WithTracing(tracing =>
    {
        using var scope = sp.CreateScope();
        tracing.AddProcessor(scope.ServiceProvider.GetRequiredService<HealthCheckTraceFilterProcessor>());

        ...
    });

When the app uses a SQL database, it's common to have instrumentation for SQL queries using OpenTelemetry.Instrumentation.SqlClient. This instrumentation is included when using .NET Aspire integrations such as the .NET Aspire SQL Server integration or the .NET Aspire SQL Server Entity Framework Core integration.

In one of our applications, we use MassTransit with the Transactional Outbox enabled & this requires periodically polling specific tables in the database for messages to send to the broker. These SQL queries generate traces which are not useful unless they fail. We can use a combination of Enrichment & a Processor to identify these traces & filter them accordingly.

First, we need to disable the tracing configuration implemented by the .NET Aspire integration as we will be customising it ourselves (the example below is for the .NET Aspire SQL Server Entity Framework Core integration):

builder.EnrichSqlServerDbContext<AppDbContext>(configure =>
{
    configure.DisableTracing = true;
});

Secondly, when configuring tracing, the OpenTelemetry.Instrumentation.SqlClient package needs to be installed to allow customising the traces it generates. Then we enrich the traces to set a custom tag for queries to known MassTransit outbox tables:

tracing.AddSource(builder.Environment.ApplicationName)
    ...
    ...
    .AddSqlClientInstrumentation(opt =>
    {
        opt.Enrich = (Activity activity, string eventName, object rawObject) =>
        {
            if (!eventName.Equals("OnCustom")) return;
            if (rawObject is not SqlCommand cmd) return;
            if (cmd.CommandText.Contains("InboxState") || cmd.CommandText.Contains("OutboxState"))
            {
                activity.SetTag("masstransit.outbox.query", true);
            }
        }
        ;
    });

Then we can write a Processor which filters based on our custom tag:

internal sealed class MassTransitOutboxDbOperationTraceFilterProcessor : BaseProcessor<Activity>
{
    public override void OnEnd(Activity activity)
    {
        if (activity.TagObjects.Any(kv => kv.Key == "masstransit.outbox.query"))
            activity.ActivityTraceFlags &= ~ActivityTraceFlags.Recorded;
    }
}

And lastly, add it to our tracing configuration:

.WithTracing(tracing =>
{
    tracing.AddProcessor(new MassTransitOutboxDbOperationTraceFilterProcessor());
   ...
})

3) Filtering Logs

First, some background: OpenTelemetry does support filtering logs, but the mechanism for doing so is not as straightforward as trace filtering due to the way that OpenTelemetry integrates with the built-in ILogger. There are many GitHub issues (1, 2, 3) I referenced when going down this rabbit hole. Long story short, log filtering involves wrapping an OpenTelemetry exporter with a filtering Processor to prevent exporting the logs. This is what it would look like (taken from one of the above issues):

    configure.AddProcessor(new FilteringBatchLogRecordProcessor(ShouldUpload, exporter));

    internal sealed class FilteringBatchLogRecordProcessor : BatchLogRecordExportProcessor
    {
        private readonly Func<LogRecord, bool> filter;

        public FilteringBatchLogRecordProcessor(Func<LogRecord, bool> filter, BaseExporter<LogRecord> exporter)
            : base(exporter)
        {
            this.filter = filter;
        }

        public override void OnEnd(LogRecord logRecord)
        {
            // Call the underlying processor
            // only if the Filter returns true.
            if (this.filter(logRecord))
            {
                base.OnEnd(logRecord);
            }
        }
    }

Testing this approach in a .NET Aspire project, I could filter logs when exporting to the local OTLP Endpoint configured by .NET Aspire:

var customOtlpExporter = new OtlpLogExporter(new OtlpExporterOptions
{
    Endpoint = new Uri(builder.Configuration["OTEL_EXPORTER_OTLP_ENDPOINT"])
});

...

builder.Logging
    .AddOpenTelemetry(logging =>
{
    logging.AddProcessor(new FilteringBatchLogRecordProcessor((log) => false, customOtlpExporter));

    ...
});

The key thing to note in the above snippet is that in order to wrap an exporter with the filtering processor, one has to create an instance of an exporter - in this case the OtlpLogExporter to pass it to the FilteringBatchLogRecordProcessor. However, when trying to configure this for the Azure Monitor exporter, I found that it simply wasn't possible because one can't initialize a new AzureMonitorLogExporter because its constructor is internal. This was highlighted in one of the above GitHub issues & in fact, a pull request was created to make the constructor public but was closed without reason.

With this setback, I decided to explore alternative filtering mechanisms to the native OpenTelemetry processors.

The built-in .NET ILogger is limited in terms of log filtering in that it can only filter based on log category. This makes it difficult to filter logs based on some other criteria such as request path or the content of the log message. There is currently an open issue on GitHub to extend ILogger to make this easier in future.

To support more advanced filtering capabilities, we can use Serilog in order to achieve our health check endpoint filtering use case.

Serilog has an ILogEventFilter interface which can be used to implement a filter which operates on health check requests:

public class HealthCheckLogsFilter(IHttpContextAccessor httpContextAccessor) : ILogEventFilter
{
    public bool IsEnabled(LogEvent logEvent)
    {
        if (httpContextAccessor.HttpContext is null)
            return true;

        var healthCheckPaths = new[]
        {
            "/health/live",
            "/health/ready"
        };

        var matchesHealthCheckPath = healthCheckPaths.Any(path =>
            httpContextAccessor.HttpContext?.Request.Path.Value?.Equals(path,
                StringComparison.OrdinalIgnoreCase) ?? true);

        return !(matchesHealthCheckPath && logEvent.Level == LogEventLevel.Information);
    }
}

Which can be wired up to the Serilog log configuration:

builder.Services.AddSingleton<HealthCheckLogsFilter>();

builder.Services.AddSerilog((services, lc) =>
{
    ...
    lc.Filter.With(services.GetRequiredService<HealthCheckLogsFilter>())
    ...
});

The same MassTransit outbox SQL queries mentioned above also generate logs for each query in addition to traces. These logs all have the category Microsoft.EntityFrameworkCore.Database.Command, but we don't necessarily want to filter out all logs from this category as this will also exclude logs from other processes which we care about. To specifically target only these MassTransit query logs, we can use an Excluding log filter with Serilog.

Firstly, we create the exclusion method which checks for logs which contain the SQL operation on known MassTransit outbox tables & only if the Log Level is Information so that when one of these queries throw an error, we don't filter them out:

private static bool MassTransitOutboxDbCommandLogExclusion(LogEvent logEvent)
{
    const string logCategory = """
                               "Microsoft.EntityFrameworkCore.Database.Command"
                               """;

    var sourceContext = logEvent.Properties.FirstOrDefault(l => l.Key == "SourceContext").Value.ToString();

    if (sourceContext != logCategory) return false;

    var commandText = logEvent.Properties.FirstOrDefault(l => l.Key == "commandText").Value?.ToString();

    if (commandText is null) return false;

    string[] massTransitOutboxTableNames = ["InboxState", "OutboxState"];

    var isMassTransitDbCommand = !string.IsNullOrWhiteSpace(commandText) && massTransitOutboxTableNames.Any(commandText.Contains);

    return isMassTransitDbCommand && logEvent is { Level: LogEventLevel.Information };
}

Then we register this method with Serilog using .Filter.ByExcluding():

builder.Services.AddSerilog((services, lc) =>
{
    ...
    .Filter.ByExcluding(MassTransitOutboxDbCommandLogExclusion);

});

4) Filtering Metrics

For some interesting history around metric filtering, see this GitHub issue.

To satisfy the requirement of filtering metrics for health check endpoints, this PR introduced the capability to opt-out of metric collection on a per-endpoint basis.

This can be added to our health check endpoints to disable metrics:

healthChecks.MapHealthChecks("/health/ready")
    .DisableHttpMetrics();

healthChecks.MapHealthChecks("/health/live", new HealthCheckOptions
{
    Predicate = static r => r.Tags.Contains("live")
})
.DisableHttpMetrics();

It's important to note that the DisableHttpMetrics() method is only available in .NET 9 & above.

Other Considerations

During the wire-up of all these various filters, there were some finer points of configuration needed to address the following:

1. Sending Serilog logs to the local OTEL endpoint when running on a local machine so that logs appear in the .NET Aspire dashboard.

2. Using the Azure.Monitor.OpenTelemetry.Exporter package instead of the Azure.Monitor.OpenTelemetry.AspNetCore package as the latter overrides some tracing & logging configuration which un-does a lot of our hard work.

I've enclosed the full configuration below to show the full setup of all the telemetry configuration:

public static class TelemetryConfigurationExtensions
{
    public static IHostApplicationBuilder AddTelemetryConfiguration(this IHostApplicationBuilder builder)
    {
        // Required so that traces for health checks are not recorded
        builder.Services.AddHttpContextAccessor();

        builder.Services.AddSingleton<HealthCheckLogsFilter>();
        builder.Services.AddSingleton<HealthCheckTraceFilterProcessor>();

        builder.Services.AddSerilog((services, lc) =>
        {
            lc.MinimumLevel.Override("Microsoft", LogEventLevel.Information)
                .MinimumLevel.Override("Microsoft.AspNetCore", LogEventLevel.Warning)
                .Filter.With(services.GetRequiredService<HealthCheckLogsFilter>())
                .Enrich.FromLogContext()
                .Filter.ByExcluding(MassTransitOutboxDbCommandLogExclusion);

        }, writeToProviders: true);

        builder.Services.AddOpenTelemetry()
            .ConfigureResource(resourceBuilder =>
            {
                resourceBuilder
                    .AddAzureContainerAppsDetector(); // Used to set the Cloud Role Name and the Cloud Role Instance properties
            })
            .WithMetrics(metrics =>
            {
                metrics.AddAspNetCoreInstrumentation()
                    .AddHttpClientInstrumentation()
                    .AddRuntimeInstrumentation();
            })
            .WithTracing(tracing =>
            {
                tracing.AddProcessor(new MassTransitOutboxDbOperationTraceFilterProcessor());

                tracing.AddSource(builder.Environment.ApplicationName)
                    .AddAspNetCoreInstrumentation()
                    .AddHttpClientInstrumentation()
                    .AddSqlClientInstrumentation(opt =>
                    {
                        opt.Enrich = SqlClientMassTransitOutboxQueryEnrichment;
                    });
            });

        var applicationInsightsConnectionString = builder.Configuration["APPLICATIONINSIGHTS_CONNECTION_STRING"];

        builder.Services.ConfigureOpenTelemetryTracerProvider((sp, tracerProviderBuilder) =>
        {
            using var scope = sp.CreateScope();

            tracerProviderBuilder.AddProcessor(scope.ServiceProvider.GetRequiredService<HealthCheckTraceFilterProcessor>());

            if (builder.Environment.IsProduction())
            {
                tracerProviderBuilder.AddAzureMonitorTraceExporter(options =>
                {
                    options.ConnectionString = applicationInsightsConnectionString;
                });
            }
        });

        builder.Services.ConfigureOpenTelemetryMeterProvider((_, meterProviderBuilder) =>
        {
            if (builder.Environment.IsProduction())
            {
                meterProviderBuilder.AddAzureMonitorMetricExporter(options =>
                {
                    options.ConnectionString = applicationInsightsConnectionString;
                });
            }
        });

        // Clear all existing logging providers so that logs are only sent to App Insights/Log Analytics
        // This prevents duplicate logs from logging to the console & also being forwarded to Log Analytics
        builder.Logging.ClearProviders();

        builder.Logging.AddOpenTelemetry(logging =>
        {
            logging.IncludeFormattedMessage = true;
            logging.IncludeScopes = true;

            if (builder.Environment.IsProduction())
            {
                logging.AddAzureMonitorLogExporter(x =>
                {
                    x.ConnectionString = applicationInsightsConnectionString;
                });
            }
        });

        var useOtlpExporter = !string.IsNullOrWhiteSpace(builder.Configuration["OTEL_EXPORTER_OTLP_ENDPOINT"]);

        if (useOtlpExporter)
        {
            builder.Services.AddOpenTelemetry()
                .UseOtlpExporter();
        }

        return builder;
    }

    private static void SqlClientMassTransitOutboxQueryEnrichment(Activity activity, string eventName, object rawObject)
    {
        if (!eventName.Equals("OnCustom")) return;
        if (rawObject is not SqlCommand cmd) return;
        if (cmd.CommandText.Contains("InboxState") || cmd.CommandText.Contains("OutboxState"))
        {
            activity.SetTag("masstransit.outbox.query", true);
        }
    }

    private static bool MassTransitOutboxDbCommandLogExclusion(LogEvent logEvent)
    {
        const string logCategory = """
                                   "Microsoft.EntityFrameworkCore.Database.Command"
                                   """;

        var sourceContext = logEvent.Properties.FirstOrDefault(l => l.Key == "SourceContext").Value.ToString();

        if (sourceContext != logCategory) return false;

        var commandText = logEvent.Properties.FirstOrDefault(l => l.Key == "commandText").Value?.ToString();

        if (commandText is null) return false;

        string[] massTransitOutboxTableNames = ["InboxState", "OutboxState"];

        var isMassTransitDbCommand = !string.IsNullOrWhiteSpace(commandText) && massTransitOutboxTableNames.Any(commandText.Contains);

        return isMassTransitDbCommand && logEvent is { Level: LogEventLevel.Information };
    }
}

internal sealed class MassTransitOutboxDbOperationTraceFilterProcessor : BaseProcessor<Activity>
{
    public override void OnEnd(Activity activity)
    {
        if (activity.TagObjects.Any(kv => kv.Key == "masstransit.outbox.query"))
            activity.ActivityTraceFlags &= ~ActivityTraceFlags.Recorded;
    }
}

internal sealed class HealthCheckTraceFilterProcessor(IHttpContextAccessor httpContextAccessor) : BaseProcessor<Activity>
{
    public override void OnEnd(Activity activity)
    {
        var excludedHealthCheckPaths = new[]
        {
            "/health/live",
            "/health/ready"
        };

        if (excludedHealthCheckPaths.Any(path =>
                httpContextAccessor.HttpContext?.Request.Path.Value?.Equals(path,
                    StringComparison.OrdinalIgnoreCase) ?? false))
        {
            activity.ActivityTraceFlags &= ~ActivityTraceFlags.Recorded;
        }
    }
}

public class HealthCheckLogsFilter(IHttpContextAccessor httpContextAccessor) : ILogEventFilter
{
    public bool IsEnabled(LogEvent logEvent)
    {
        if (httpContextAccessor.HttpContext is null)
            return true;

        var healthCheckPaths = new[]
        {
            "/health/live",
            "/health/ready"
        };

        var matchesHealthCheckPath = healthCheckPaths.Any(path =>
            httpContextAccessor.HttpContext?.Request.Path.Value?.Equals(path,
                StringComparison.OrdinalIgnoreCase) ?? true);

        return !(matchesHealthCheckPath && logEvent.Level == LogEventLevel.Information);
    }
}

Conclusion

These techniques may not necessarily be the most optimal way to achieve the desired outcome, or quite the OpenTelemetry-native way, so if there are better approaches, please let me know!

OpenTelemetry is indeed enormously powerful & quite customisable in how you can adapt it to your use case. Though it’s worth noting that there are some gotchas with how it integrates into the broader .NET ecosystem which may be unintuitive. This post documents some learnings in my telemetry journey & hopefully these learnings can help cut down on unnecessary telemetry & hopefully save some $$$ in the process.

According to the documentation, this requires that you grant the SQL Server identity additional permissions, which in turn can only be granted by either a Global Administrator or Privileged Role Administrator.

Once the SQL Server identity has these permissions, one can execute the following query to add the application's identity to the database:

-- Create the user with details retrieved from Entra ID
CREATE USER [ManagedIdentityName] FROM EXTERNAL PROVIDER

-- Assign roles to the new user
ALTER ROLE db_datareader ADD MEMBER [ManagedIdentityName]
ALTER ROLE db_datawriter ADD MEMBER [ManagedIdentityName]

Common strategies for enabling this at scale are:

1. Create a single User Assigned Managed Identity with the required permissions & associate it to all SQL Servers

2. Create a dedicated Entra ID group for SQL identities & grant it the Directory Readers role

There exists, however, an alternative approach which doesn't require any of the above pre-configuration.

Given the Client ID of your app's Managed Identity, you can use the following PowerShell to convert the client ID into the SQL Server's User SID format:

$managedIdentitySid = "0x" + [System.BitConverter]::ToString(([guid]$managedIdentityClientId).ToByteArray()).Replace("-", "")

And then add the user to the SQL Database as follows:

-- Create the user using its SID - does not need to talk to Entra ID
CREATE USER [ManagedIdentityName] WITH SID = <managedIdentitySid>, TYPE = E;

-- Assign roles to the new user
ALTER ROLE db_datareader ADD MEMBER [ManagedIdentityName]
ALTER ROLE db_datawriter ADD MEMBER [ManagedIdentityName]

Here is an example of an Azure Pipeline using the above approach:

- task: AzureCLI@2
  displayName: 'Get Managed Identity Client ID'
  inputs:
    azureSubscription: ${{ parameters.serviceConnection }}
    scriptType: 'pscore'
    scriptLocation: 'inlineScript'
    inlineScript: |
      $clientId = az identity show --name $(managedIdentityName) --resource-group $(resourceGroupName) --query clientId -o tsv
      echo "##vso[task.setvariable variable=managedIdentityClientId]$clientId"

- powershell: |
    $clientId = '$(managedIdentityClientId)'
    $sid = "0x" + [System.BitConverter]::ToString(([guid]$clientId).ToByteArray()).Replace("-", "")

    Write-Host "##vso[task.setvariable variable=managedIdentitySid]$sid"
  displayName: Convert Managed Identity Client ID to SID

# The Azure DevOps pipeline Service Connection (Azure AD Service Principal) needs to be a SQL Server Administrator (AAD Group) or Database Owner
# The SqlAzureDacpacDeployment@1 task can only run on windows agents
- task: SqlAzureDacpacDeployment@1
  displayName: 'Add User + Roles for Managed Identity'
  inputs:
    azureSubscription: ${{ parameters.serviceConnection }}
    AuthenticationType: 'servicePrincipal'
    ServerName: '$(sqlServerName).database.windows.net'
    DatabaseName: '$(sqlDatabaseName)'
    TaskNameSelector: 'InlineSqlTask'
    SqlInline: |
      IF NOT EXISTS (SELECT [name] FROM [sys].[database_principals] WHERE [name] = '$(managedIdentityName)')
      CREATE USER [$(managedIdentityName)] WITH SID = $(managedIdentitySid), TYPE = E;
      GO
      ALTER ROLE [db_datareader] ADD MEMBER [$(managedIdentityName)];
      ALTER ROLE [db_datawriter] ADD MEMBER [$(managedIdentityName)];
      GO
    IpDetectionMethod: 'AutoDetect'

I can't take credit for the PowerShell script, I found it by happenstance in this StackOverflow answer by Thomas Vercoutre. Thomas you legend! 👏

Entra ID Managed Identities are great. They enable authentication & authorization to Azure resources without the need to store & manage credentials.

This works well if you, say, have an Azure Container App which connects to an Azure SQL Database. It's possible to use a managed identity to authenticate with the database so that you don't need to hardcode explicitly configure a password in a connection string.

This is all well & good when talking to Azure services - Azure SQL, Cosmos DB, Key Vault, etc. But when an app needs to talk to a custom API protected with Entra ID, the documentation states that one must use an App Registration along with a Client ID & Secret/Certificate to acquire a token for the downstream API. This means that your app will need to store this secret/certificate somewhere & have a process in place to rotate it when it inevitably expires. Not ideal when Managed Identities were created specifically to solve for this scenario.

For ASP.NET Core apps, the Microsoft Identity Web library is the recommended way for interacting with the Microsoft Identity Platform to protect APIs with Entra ID, acquire access tokens & a whole host of other auth-related stuff. The most common approach I've encountered in the wild for acquiring an access token on behalf of an app (as opposed to a user) is to use the Client Credentials flow which relies on supplying a client ID along with a client secret (which is not recommended for production 🤫) or client certificate.

Fast-forward to February 2024, the 2.17.0 release of Microsoft Identity Web included this new feature in the release notes:

• Added support for Managed identity when calling a downstream API on behalf of the app. See Calling APIs with Managed Identity and PR 2650. For details see PR #2645

This was intriguing! Did this mean I could throw my secrets in the rubbish bin? Spoiler: Almost.

To get this to work, a few prerequisites need to be in place:

1. The downstream API which is protected with Entra ID needs at least one App Role defined on the App Registration

2. The calling API should have either a System-Assigned Managed Identity or User-Assigned Managed Identity associated to the compute resource

3. The Managed Identity used by the calling API must be assigned to the App Role on the downstream API's App Registration by following the approach detailed here - There is no UI to do this in the Azure Portal at the time of writing & hence must be done via Azure PowerShell or the Azure CLI

You can confirm that step 3 was successful by navigating to the Microsoft Entra ID blade in the Azure Portal > Enterprise Applications > Switch the 'Application type' filter to Managed Identities > select the app with the same name as your Managed Identity > Permissions - note that when using a System-Assigned Managed Identity, the name of the Managed Identity will be the same as your compute resource name.

So given the following App Registration with its App Roles:

This is what the Managed Identity Enterprise App Permissions should look like:

This is not entirely different from how we used App Registrations to achieve this. The key difference between the App Registration setup & the Managed Identity setup is that, when using Managed Identities, the App Role on the downstream API is assigned to the client app’s Managed Identity rather than to an App Registration. This means that an App Registration need not be created to represent the client app within the Microsoft Identity Platform, it can just use a Managed Identity. However, if the client app is, itself an API which needs to authenticate users or applications, then it will still require an App Registration of its own. A daemon application, for example, would not require an App Registration at all.

What changes from a code perspective? Let's have a look - The following example showcases an ASP.NET Core Web API (protected by Entra ID) which calls a downstream ASP.NET Core Web API (protected by Entra ID) on behalf of the app.

This example shows the configuration & code using the Client Credentials flow to call a downstream API - in this case, the quintessential Weather API. For simplicity, the below code uses the IDownstreamApi helper interface for calling the API, which takes care of acquiring the token, attaching the Authorization header to the request, handling errors & deserialising the response. There are also other options which allow for more flexibility, such as the low-level ITokenAcquisition or ITokenAcquirerFactory interfaces, or the IAuthorizationHeaderProvider interface, depending on your use case or preference.

appsettings.json:

{
  "AzureAd": {
    "Instance": "https://login.microsoftonline.com/",
    "TenantId": "<tenant id guid>",
    "ClientId": "<app registration client id guid>",
    "ClientCredentials": [
      {
        "SourceType": "ClientSecret",
        "ClientSecret": "<client secret>"
      }
    ]
  },
  "DownstreamApi": {
    "BaseUrl": "https://weather-api/",
    "RelativePath": "WeatherForecast",
    "RequestAppToken": true,
    "Scopes": [ "api://<downstream api name or app id>/.default" ]
  }
}

Program.cs:

builder.Services.AddMicrosoftIdentityWebApiAuthentication(builder.Configuration)
    .EnableTokenAcquisitionToCallDownstreamApi()
    .AddDownstreamApi("WeatherApi", builder.Configuration.GetSection("DownstreamApi"))
    .AddInMemoryTokenCaches();

Controller:

[ApiController]
[Route("[controller]")]
public class WeatherForecastController(
    ILogger<WeatherForecastController> logger,
    IDownstreamApi downstreamApi) : ControllerBase
{
    [HttpGet(Name = "GetWeatherForecast")]
    public async Task<IEnumerable<WeatherForecast>> Get()
    {
        try
        {
            var results = await downstreamApi.GetForAppAsync<IEnumerable<WeatherForecast>>("WeatherApi");

            return results ?? [];
        }
        catch (Exception ex)
        {
            logger.LogError(ex, "Something went wrong");
            return [];
        }
    }
}

To enable the Managed Identity flow, update the appsettings.json file to remove the ClientCredentials section from the AzureAd section & add an AcquireTokenOptions section inside the DownstreamApi section as shown:

{
  "AzureAd": {
    "Instance": "https://login.microsoftonline.com/",
    "TenantId": "<tenant id guid>",
    "ClientId": "<app registration client id guid>"
  },
  "DownstreamApi": {
    "BaseUrl": "https://weather-api/",
    "RelativePath": "WeatherForecast",
    "RequestAppToken": true,
    "Scopes": [ "api://<downstream api name or app id>/.default" ],
    "AcquireTokenOptions": {
      "ManagedIdentity": {
        "UserAssignedClientId": "<managed identity client id>"
      }
    }
  }
}

Note that when using a System-Assigned Managed Identity, the UserAssignedClientId property can be omitted, like so:

"AcquireTokenOptions": {
  "ManagedIdentity": {
  }
}

The controller code or Program.cs code does not need to be updated.

The above example holds true for API calling a downstream API, but using a Worker service is also supported, though the configuration differs slightly as the Worker service is a daemon app. There is also a super minimal Console app example in the GitHub repo wiki.

One thing I should call out is that, while this approach is awesome & prevents the need to store credentials & manage their rotation, there are some limitations to using Managed Identities for Authorization. Most notably that Managed Identity tokens are cached by the Azure infrastructure for up to 24 hours with no currently supported way to force a refresh of the token. The docs state that this delay is applicable only when using Entra groups with Managed Identities & recommend using a user-assigned managed identity with permissions applied directly to the identity, instead of adding to or removing managed identities from a Microsoft Entra group that has permissions. I have not tested this extensively, but I will be running some experiments to kick the tyres.

A crucial aspect of using Managed Identities for authorization is that Managed Identities only work in Azure. This might seem obvious, but it's worth noting as this means that this approach will not work for local development scenarios. If you want to call the protected API from your local machine, you will either need to:

1. Create an App Registration with a Client ID & Secret with the sole purpose of local development use - as well as using different config for local dev to use the Client Credentials flow

2. Mock the call to the protected API only for local development/running integration tests locally

And that's why I said you can almost throw your secrets in the rubbish bin. Alas, there are no free lunches. 🙂

I'm sure this approach works for a broad set of use-cases such that we can start slowly chipping away at the number of secrets which need managing. I might be coming across as a secret-basher, but it's not like expired or leaked secrets ever caused anyone any trouble, right? 😉

Observability is important

I agree with that statement. It's crucial to be able to ascertain that a system is functioning correctly, and when implemented well, it eases the troubleshooting process for when something inevitably goes awry.

Unfortunately, oftentimes the above statement gets translated into the following:

Log everything

In a world with free infinite storage, I say go ahead. But that world & our world are not the same.

Cloud providers are acutely aware that observability is important & will reinforce that fact throughout their documentation as a best practice. They will also charge for that observability accordingly. Couple that with the log-everything mantra & that is a recipe for a woefully melancholic invoice at the end of the month.

We experienced this melancholy recently with a simple ASP.NET Core Web API deployed to Azure Container Apps. The application had some supporting resources such as Azure SQL Database, Key Vault, App Configuration, Front Door, and because we strive to be good observability citizens, Application Insights & Log Analytics.

We deployed the API into a development environment, left it running for a couple of days & upon analysing our costs noticed the following:

Upon this realisation, the question that came to mind was:

"Why the **** is Log Analytics' cost so high?"

Keep in mind that this was in a development environment with no real user traffic apart from us doing some initial tests to verify that the app was working. This was while the application was just sitting there. So, we started digging.

We checked the log categories that we were sending to Log Analytics to see if we could spot some offenders:

That's a lot of AppTraces. Digging into what these AppTraces were, we determined that these were generated by another thing that is supposed to be good practice - health checks. A lot of health checks.

The API we deployed included two kinds of health checks: liveness and readiness. These terms are widely used in the Kubernetes world to determine whether an application is running (live) & ready to receive requests (ready) & the Kubernetes API uses these checks to determine if pods should be restarted and if traffic should be sent to them. In our API's case, the liveness check was a /health/live endpoint which returned a simple 200 OK response to indicate that the API was running, whereas the readiness check was a /health/ready endpoint which queried critical dependencies to check that all of them were happy & healthy as well.

Digging into the logging for our API, a call to the /health/live endpoint generated the following logs:

[09:23:32 INF] Request starting HTTP/2 GET https://localhost:5001/health/live - -
[09:23:32 INF] Executing endpoint 'Health checks'
[09:23:32 INF] Executed endpoint 'Health checks'
[09:23:32 INF] HTTP GET /health/live responded 200 in 355.8161 ms
[09:23:32 INF] Request finished HTTP/2 GET https://localhost:5001/health/live - - - 200 - text/plain 417.8957ms

Calling the /health/ready endpoint (which is a more involved check) generated the following logs:

[09:23:45 INF] Request starting HTTP/2 GET https://localhost:5001/health/ready - -
[09:23:45 INF] Executing endpoint 'Health checks'
[09:23:45 INF] Start processing HTTP request GET https://login.microsoftonline.com/<tenantId>/v2.0/.well-known/openid-configuration
[09:23:45 INF] Sending HTTP request GET https://login.microsoftonline.com/<tenantId>/v2.0/.well-known/openid-configuration
[09:23:46 INF] Received HTTP response headers after 1102.9673ms - 200
[09:23:46 INF] End processing HTTP request after 1131.027ms - 200
[09:23:47 INF] Executed endpoint 'Health checks'
[09:23:47 INF] HTTP GET /health/ready responded 200 in 1286.3275 ms
[09:23:47 INF] Request finished HTTP/2 GET https://localhost:5001/health/ready - - - 200 - application/json 1297.1640ms

Keen eyes may notice that we're using Serilog which has been configured to use the Serilog.Sinks.ApplicationInsights package to send logs to Application Insights as TraceTelemetry which explains where all the AppTraces in Log Analytics came from.

That is a non-trivial amount of logging for something that is purely infrastructural. One could argue that generating all these logs for successful calls to health check endpoints is unnecessarily verbose. Of course, I don't expect everyone to share the same opinion. If these logs were collected & analysed for response time trends or something similar, they would be necessary. I only really care when health checks fail and when they do, I would like to be notified that something blew up. It's not like I'm going to send myself PagerDuty alerts every five minutes stating that nothing has blown up yet. So, we came to a collective agreement that we didn't want to store these successful health check logs until we needed them.

Another question we asked was:

"Why is the sheer volume of logs so high?"

The answer to that question has some nuance to it.

Azure Container Apps has an awesome feature called Health Probes. These probes are configured on a container level. Container Apps can also scale out your app to multiple containers based on a Scale Rule.

See where I'm going with this?

Our API was scaled to three containers, each with its own liveness and readiness probes, configured to call their respective endpoints every 10 seconds. That's 14 log entries x 3 containers x 6 = 252 log entries per minute being sent to Log Analytics - just for health checks.

That was not the only contributing factor though. The API is protected by an Azure Front Door resource which has its own Health Probes feature to check if the origin is healthy. In our case, this was configured to probe the backend API's /health/live endpoint every 30 seconds on top of the built-in probes in Container Apps.

And that is why the volume of logs was so high.

With more wisdom gained, we set out on a path of optimisation. The first port of call was to implement the guidance from Andrew Lock's excellent blog post on Excluding health check endpoints from Serilog request logging. This involved setting the log level for log entries generated by the health check endpoints to Verbose by using a custom GetLevel function for the Serilog UseSerilogRequestLogging middleware as shown in Andrew's example repo.

The pertinent code is shown here for reference:

public static Func<HttpContext, double, Exception, LogEventLevel> GetLevel(LogEventLevel traceLevel, params string[] traceEndpointNames)
{
    if (traceEndpointNames is null || traceEndpointNames.Length == 0)
    {
        throw new ArgumentNullException(nameof(traceEndpointNames));
    }

    return (ctx, _, ex) => 
        IsError(ctx, ex) 
        ? LogEventLevel.Error
        : IsTraceEndpoint(ctx, traceEndpointNames)
            ? traceLevel
            : LogEventLevel.Information;
}

static bool IsError(HttpContext ctx, Exception ex) 
    => ex != null || ctx.Response.StatusCode > 499;

static bool IsTraceEndpoint(HttpContext ctx, string[] traceEndpoints)
{
    var endpoint = ctx.GetEndpoint();
    if (endpoint is object)
    {
        for (var i = 0; i < traceEndpoints.Length; i++)
        {
            if (string.Equals(traceEndpoints[i], endpoint.DisplayName, StringComparison.OrdinalIgnoreCase))
            {
                return true;
            }
        }
    }
    return false;
}

This GetLevel method is wired up in Startup.cs/Program.cs as follows:

app.UseSerilogRequestLogging(opts => {
    opts.EnrichDiagnosticContext = LogHelper.EnrichFromRequest;
    opts.GetLevel = LogHelper.GetLevel(LogEventLevel.Verbose, "Health checks");
});

Implementing the above resulted in the logs generated from health check endpoints to be reduced to:

That's right. Zero. Zip. Nada.

It's worthwhile to note that these logs are only filtered out when calls to the health check endpoints are successful. If an error happens to occur, the log level will be set to Error and those logs will still be emitted.

But we can do more!

The amount of AppTraces logs decreased, though there were some more low-hanging fruits in the form of AppRequests and AppDependencies. These log categories are sent by the Application Insights SDK for ASP.NET Core whenever requests reached our API & when a dependency was called, respectively. Therefore, we were generating AppRequests telemetry for each call to the /health/live and /health/ready endpoints, as well as AppDependencies telemetry when the /health/ready health check pinged critical dependencies such as SQL Server, Microsoft Entra ID, Key Vault, etc.

These sets of telemetry also constituted a fair amount of data in Log Analytics:

For these, we could not reach for Serilog as this was outside of its realm. Fortunately, the Application Insights SDK we used in the API project allows for filtering and preprocessing telemetry using ITelemetryProcessor.

With this revelation in mind, we settled on the following:

public class HealthCheckRequestFilter : ITelemetryProcessor
{
    private readonly ITelemetryProcessor _next;
    private readonly IHttpContextAccessor _httpContextAccessor;
    public HealthCheckRequestFilter(ITelemetryProcessor next, IHttpContextAccessor httpContextAccessor)
    {
        _next = next ?? throw new ArgumentNullException(nameof(next));
        _httpContextAccessor = httpContextAccessor ?? throw new ArgumentNullException(nameof(httpContextAccessor));
    }
    public void Process(ITelemetry item)
    {
        var excludedHealthCheckPaths = new[]
        {
            "/health/live",
            "/health/ready"
        };
        if (excludedHealthCheckPaths.Any(path =>
                _httpContextAccessor.HttpContext?.Request.Path.Value?.Equals(path,
                    StringComparison.OrdinalIgnoreCase) ?? false)) return;
        _next. Process(item);
    }
}

The HealthCheckRequestFilter uses IHttpContextAccessor to determine if the incoming request path matches one of our predefined health check endpoints, and if it does, we tell the Application Insights SDK to skip processing the telemetry.

Then the filter is wired up in Startup.cs/Program.cs:

services.AddApplicationInsightsTelemetryProcessor<HealthCheckRequestFilter>();

After deploying this change, graphs trended downwards:

We managed to reduce our daily log ingestion from ±1.1GB to under 50MB. This resulted in cost savings by cutting our Log Analytics cost from a touch over $3/day to 25c/day.

Moral of the story: checking if your app is healthy can come at a cost. Especially if those checks are performed often. But it doesn't have to be - some considered optimisation can bring down your cloud bill without sacrificing best practices.

Through this effort, we gained a better understanding of exactly how our application generates logs, the triggers for these logs, and the various mechanisms to fine-tune exactly which logs get stored.

I hope this post leads to some cash savings on a couple of invoices. 🙂

Overview

There are plenty of ASP.NET Core projects out on the internet. All of them have one thing in common. The Startup.cs file. There's also another thing that a lot of these Startup.cs files have in common - they are HUGE.

The Problem

For those unfamiliar with Startup.cs, this is the class that wires up pretty much everything in ASP.NET Core. It's where the request pipeline, dependency injection container, configuration & others get configured. The official docs can be found here. The issue with being responsible for wiring up pretty much everything is that this class tends to grow quite large as more things need to be configured. This problem is compounded with plenty of documentation & tutorials suggesting that, to wire up [insert feature here], all you need to do is drop [insert code snippet here] into Startup.cs and you're off to the races. And the file just keeps on growing...

This causes a few things:

• The Startup class becomes difficult to navigate because there's a lot happening in it
• The Startup class violates the Single Responsibility Principle (SRP) because it no longer has a single reason to change
• It hurts maintainability because there's a high chance of merge conflicts if multiple developers make changes to the Startup class simultaneously

The Problem in Practice

Let's take the following Startup.cs template as a starting point, taken from the .NET Core Web API template (comments removed for brevity):

public class Startup
{
    public Startup(IConfiguration configuration)
    {
        Configuration = configuration;
    }

    public IConfiguration Configuration { get; }

    public void ConfigureServices(IServiceCollection services)
    {
        services.AddControllers();
    }

    public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
    {
        if (env.IsDevelopment())
        {
            app.UseDeveloperExceptionPage();
        }

        app.UseHttpsRedirection();

        app.UseRouting();

        app.UseAuthorization();

        app.UseEndpoints(endpoints =>
        {
            endpoints.MapControllers();
        });
    }
}

Awesome, clean & simple. For now.

Now let's add our first feature. Any application with reasonable complexity is going to need to store some data somewhere. So, let's add Entity Framework Core & wire it up with SQL Server. As per the docs, we add the following to ConfigureServices:

services.AddDbContext<SchoolContext>(options =>
    options.UseSqlServer(Configuration.GetConnectionString("SchoolContext")));

We also want to be good citizens & document our API so that it can be used by others. So let's use Swashbuckle to add some Open API (Swagger) documentation to our API. So we add the following to ConfigureServices:

services.AddSwaggerGen(c =>
{
    c.SwaggerDoc("v1", new OpenApiInfo { Title = "My Awesome API", Version = "v1" });
});

And the following to Configure, inside the env.IsDevelopment() check:

app.UseSwagger();
app.UseSwaggerUI(c => c.SwaggerEndpoint("/swagger/v1/swagger.json", "My Awesome API v1"));

We've also written a couple of custom classes & interfaces which we need to add to the ServiceCollection for Dependency Injection, so let's add them to ConfigureServices:

services.AddScoped<INotificationService, EmailNotificationService>();
services.AddScoped<INotificationService, SmsNotificationService>();
services.AddScoped<INotificationService, PushNotificationService>();

That's enough features for today. Not too many, but enough to start off with. Now our Startup class looks like this:

public class Startup
{
    public Startup(IConfiguration configuration)
    {
        Configuration = configuration;
    }

    public IConfiguration Configuration { get; }

    public void ConfigureServices(IServiceCollection services)
    {
        services.AddControllers();

        services.AddDbContext<SchoolContext>(options =>
            options.UseSqlServer(Configuration.GetConnectionString("SchoolContext")));

        services.AddSwaggerGen(c =>
        {
            c.SwaggerDoc("v1", new OpenApiInfo { Title = "My Awesome API", Version = "v1" });
        });

        services.AddScoped<INotificationService, EmailNotificationService>();
        services.AddScoped<INotificationService, SmsNotificationService>();
        services.AddScoped<INotificationService, PushNotificationService>();
    }

    public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
    {
        if (env.IsDevelopment())
        {
            app.UseDeveloperExceptionPage();
            app.UseSwagger();
            app.UseSwaggerUI(c => c.SwaggerEndpoint("/swagger/v1/swagger.json", "My Awesome API v1"));
        }

        app.UseHttpsRedirection();

        app.UseRouting();

        app.UseAuthorization();

        app.UseEndpoints(endpoints =>
        {
            endpoints.MapControllers();
        });
    }
}

It's not too hairy quite yet, but we've only added three features. Imagine what this will look like once we add authentication, authorization, validation, exception handling, logging, caching, customizing Swagger, etc. It can get out of hand for a reasonably-sized application.

The Alternative

The following approach aims to address these issues by refactoring our Startup.cs file so that we can all sleep a little better at night, and we all know that sleep is important.

Let's try to separate some of the Startup class' concerns by refactoring some of them to separate classes. We can use some of the same tricks used by the ASP.NET Core team to accomplish this. For example, if we look at the following line:

services.AddControllers();

What that's doing is configuring all the necessary bits & pieces so that we can use Controllers in an ASP.NET Core Web API, including stuff like Authorization, CORS, Data Annotations, ApiExplorer, etc. But it's all just one line of code. Neat.

That is also an extension method that operates on IServiceCollection. So let's try and write some of our own extension methods which work with IServiceCollection, starting with the database:

internal static class DatabaseServiceCollectionExtensions
{
    public static IServiceCollection AddDatabaseConfiguration(this IServiceCollection services,
        IConfiguration configuration)
    {
        services.AddDbContext<SchoolContext>(options =>
            options.UseSqlServer(Configuration.GetConnectionString("SchoolContext")));

        return services;
    }
}

Here we have a static class named DatabaseServiceCollectionExtensions because it contains extension methods which operate on IServiceCollection.

There is one static method, AddDatabaseConfiguration which takes the IServiceCollection & IConfiguration as parameters and returns an IServiceCollection. This method then calls the same code we had in Startup to configure Entity Framework Core, and then returns the updated IServiceCollection.

Then we can use it in our Startup class inside ConfigureServices like so:

services.AddDatabaseConfiguration(Configuration);

The Configuration argument is optional for these extension methods, it's just there for the case where the extension method needs something from config in order to perform it's logic, as is the case with the AddDatabaseConfiguration method.

We can then continue this trend for our own application services:

internal static class ApplicationServicesServiceCollectionExtensions
{
    public static IServiceCollection AddApplicationServicesConfiguration(this IServiceCollection services)
    {
        services.AddScoped<INotificationService, EmailNotificationService>();
        services.AddScoped<INotificationService, SmsNotificationService>();
        services.AddScoped<INotificationService, PushNotificationService>();    

        return services;
    }
}

And use it in Startup:

services.AddApplicationServicesConfiguration();

As for Swagger, we needed to make changes to both ConfigureServices and Configure in order to wire it up. Therefore,we'll need two extension methods, one which operates on IServiceCollection as before, and one which operates on IApplicationBuilder for the logic in the Configure method:

internal static class SwaggerExtensions
{
    public static IServiceCollection AddSwaggerConfiguration(this IServiceCollection services,
        IConfiguration configuration)
    {
        services.AddSwaggerGen(c =>
        {
            c.SwaggerDoc("v1", new OpenApiInfo { Title = "My Awesome API", Version = "v1" });
        });

        return services;
    }

    public static IApplicationBuilder UseSwaggerConfiguration(this IApplicationBuilder app)
    {
        app.UseSwagger();

        return app.UseSwaggerUI(c => c.SwaggerEndpoint("/swagger/v1/swagger.json", "My Awesome API v1"));
    }
}

Then we just use them in Startup in their respective methods. In ConfigureServices:

services.AddSwaggerConfiguration();

And in Configure:

app.UseSwaggerConfiguration();

After all this, our Startup class looks like this:

public class Startup
{
    public Startup(IConfiguration configuration)
    {
        Configuration = configuration;
    }

    public IConfiguration Configuration { get; }

    public void ConfigureServices(IServiceCollection services)
    {
        services.AddControllers();
        services.AddDatabaseConfiguration(Configuration);
        services.AddApplicationServicesConfiguration();
        services.AddSwaggerConfiguration();
    }

    public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
    {
        if (env.IsDevelopment())
        {
            app.UseDeveloperExceptionPage();
            app.UseSwaggerConfiguration();
        }

        app.UseHttpsRedirection();

        app.UseRouting();

        app.UseAuthorization();

        app.UseEndpoints(endpoints =>
        {
            endpoints.MapControllers();
        });
    }
}

That's a bit better. 😁

Conclusion

This approach has a few benefits:

• It keeps the Startup class focused by moving configuration logic out to separate classes
• It prevents the Startup class from growing uncontrollably
• It maintains a better separation of concerns such that, when something changes in the configuration of Swagger, for example, that change is isolated to the class which deals with Swagger configuration
• It improves maintainability because it's easier to find & navigate to the class which deals with a specific feature rather than scrolling through the entirety of the Startup class looking for the specific line which needs to be changed

I hope this has proved useful to someone & that it helps you make your applications' code slightly simpler. 😀