Container telemetry not shown in AppHost dashboard

[paulomorgado]

In preview 5, adding .WithOtlpExporter() to .AddContainer(...) would show OpenTelemetry logs, traces and metrics in the dashboard.

Since preview 6 this doesn't happen.

But if I had a dashboard container (mcr.microsoft.com/dotnet/aspire-dashboard or mcr.microsoft.com/dotnet/nightly/aspire-dashboard:8.0-preview):

var aspireDashboard = builder.AddContainer(
    "aspire-dashboard",
    "mcr.microsoft.com/dotnet/aspire-dashboard",
    "8.0-preview")
    .WithEndpoint(targetPort: 18888, port: 5018, name: "dash", scheme: "http", isProxied: true)
    .WithEndpoint(targetPort: 18889, port: 5019, name: "otel", scheme: "http", isProxied: true)
    .WithEnvironment("DOTNET_DASHBOARD_UNSECURED_ALLOW_ANONYMOUS", "true")
    .WithEnvironment("ASPIRE_ALLOW_UNSECURED_TRANSPORT", "true")
    .WithEnvironment("DASHBOARD__OTLP__AUTHMODE", "ApiKey")
    .WithEnvironment("DASHBOARD__OTLP__PRIMARYAPIKEY", builder.Configuration["AppHost:OtlpApiKey"])
    ;
var service = builder.AddContainer(...)
    .WithEndpoint(targetPort: 8080, port: 5012, name: "http", scheme: "http")
    .WithEnvironment("OTEL_BLRP_SCHEDULE_DELAY", "1000")
    .WithEnvironment("OTEL_BSP_SCHEDULE_DELAY", "1000")
    .WithEnvironment("OTEL_METRIC_EXPORT_INTERVAL", "1000")
    .WithEnvironment("OTEL_TRACES_SAMPLER", "always_on")
    .WithEnvironment("OTEL_EXPORTER_OTLP_PROTOCOL", "grpc")
    .WithEnvironment("OTEL_SERVICE_NAME", "hub")
    .WithEnvironment("OTEL_RESOURCE_ATTRIBUTES", "service.instance.id=hub")
    .WithEnvironment("OTEL_EXPORTER_OTLP_HEADERS", $"x-otlp-api-key={builder.Configuration["AppHost:OtlpApiKey"]}")
    .WithEnvironment(ctx => ctx.EnvironmentVariables["OTEL_EXPORTER_OTLP_ENDPOINT"] = new HostUrl(hubAspireDashboard.GetEndpoint("otel").Url))
...

the dashboard from the aspireDashboard container show telemetry from the service container.

service container is referenced from other resources and references other resources without any issues.

[DamianEdwards]

Which container runtime are you using?

[paulomorgado]

Rancher Desktop: https://rancherdesktop.io/

[davidfowl]

Could it be https?

[paulomorgado]

I think I tried every possibility.

Always works with the external dashboard, but never with the host dashboard.

[davidfowl]

Provide a minimal repro that we can run. That’ll help us diagnosis the problem.

[paulomorgado]

I was able to reproduce with the Aspire starter demo on Rancher Desktop.

1. Create a new starter solution: dotnet new aspire-starter.
2. Change the ApiService to produce a container.
3. Build and publish the container.
4. Change the Aspire host:

var builder = DistributedApplication.CreateBuilder(args);

//var apiService = builder.AddProject<Projects.ApiService>("apiservice")
var apiService = builder.AddContainer("apiservice", "apiservice")
    .WithHttpEndpoint(targetPort: 8080, name: "apiservice")
    .WithOtlpExporter()
    ;

builder.AddProject<Projects.Web>("webfrontend")
    .WithExternalHttpEndpoints()
    .WithReference(apiService.GetEndpoint("apiservice"))
    ;

builder.Build().Run();

6. Run the Aspire host.

You can check that all OTEL environment variables are set for the container, but no telemetry is captured by the Aspire host.

[davidfowl]

I'm guessing it's to do with https.

[paulomorgado]

Were you able to repro?

It used to work.

[DamianEdwards]

Can you turn up logging for the app in the container and look at the logs?

[paulomorgado]

I omitted, for brevity, but I'm setting these environment variables:

    .WithEnvironment("Logging__LogLevel__Default", "Debug")
    .WithEnvironment("Logging__LogLevel__Microsoft.AspNetCore", "Debug")

@davidfowl might be on something here:

I'm guessing it's to do with https.

But it works fine with a containerized dashboard (mcr.microsoft.com/dotnet/aspire-dashboard:8.0-preview). That's what I'm using now:

var aspireDashboard = builder.AddContainer(
    "hub-aspire-dashboard",
    "mcr.microsoft.com/dotnet/aspire-dashboard",
    "8.0-preview")
    .WithEndpoint(targetPort: 18888, name: "dash", scheme: "http", isProxied: true)
    .WithEndpoint(targetPort: 18889, name: "otel", scheme: "http", isProxied: true)
    .WithEnvironment("DOTNET_DASHBOARD_UNSECURED_ALLOW_ANONYMOUS", "true")
    .WithEnvironment("ASPIRE_ALLOW_UNSECURED_TRANSPORT", "true")
    .WithEnvironment("DASHBOARD__OTLP__AUTHMODE", "ApiKey")
    .WithEnvironment("DASHBOARD__OTLP__PRIMARYAPIKEY", builder.Configuration["AppHost:OtlpApiKey"])
    .WithHealthCheck(
        endpointName: "otel",
        path: string.Empty,
        httpClientFactory: () => new HttpClient
        {
            DefaultRequestVersion = HttpVersion.Version20,
            DefaultVersionPolicy = HttpVersionPolicy.RequestVersionOrHigher,
        })
    ;
...
var mongoDB = builder.AddMongoDB("mongodb")
    .WithImageTag("5.0")
    .WithHealthCheck()
    ;
var hubIdp = builder.AddProject<Projects.Project1>(name: "hub-identity", launchProfileName: null)
    .WithEndpoint(scheme: "http")
    .WithEnvironment(ctx => ctx.EnvironmentVariables["OTEL_EXPORTER_OTLP_ENDPOINT"] = new HostUrl(aspireDashboard.GetEndpoint("otel").Url))
    ;
var hubContainer = builder.AddContainer("container", "Image" "Tag")
    .WithEndpoint(targetPort: 8080, name: "http", scheme: "http")
    .WithReference(mongoDB, connectionName: "MongoDB")
    .WithOtlpExporter()
    .WithEnvironment(ctx => ctx.EnvironmentVariables["OTEL_EXPORTER_OTLP_ENDPOINT"] = new 
    ;
...

[afscrome]

OTEL logs don't show up in normal logs - you have to do a bit of extra magic to get them by creating an OTEL_DIAGNOSTICS.json file -
https://github.com/open-telemetry/opentelemetry-dotnet/blob/main/src/OpenTelemetry/README.md#self-diagnostics

[DamianEdwards]

It's likely HTTPS as the HTTPS certificate being used to secure the dashboard endpoints when it's run from the host is the ASP.NET Core Development Certificate which will not be trusted by default from inside a container. This will cause the OTLP exporter to fail. This change was introduced in preview 6 as part of security work. In the example you provided of running the dashboard itself in a container, you're explicitly disabling the HTTPS transport via the ASPIRE_ALLOW_UNSECURED_TRANSPORT environment variable and creating only HTTP endpoints using WithEndpoint.

Having composed containers be automatically configured to trust the ASP.NET Core Development Certificate is something we want to investigate enabling in the future.

[paulomorgado]

It's likely HTTPS as the HTTPS certificate being used to secure the dashboard endpoints when it's run from the host is the ASP.NET Core Development Certificate which will not be trusted by default from inside a container. This will cause the OTLP exporter to fail. This change was introduced in preview 6 as part of security work. In the example you provided of running the dashboard itself in a container, you're explicitly disabling the HTTPS transport via the ASPIRE_ALLOW_UNSECURED_TRANSPORT environment variable and creating only HTTP endpoints using WithEndpoint.

Thanks. I have a demo working with this launch profile:

"http": {
  "commandName": "Project",
  "dotnetRunMessages": true,
  "launchBrowser": true,
  "applicationUrl": "http://localhost:15190",
  "environmentVariables": {
    "ASPNETCORE_ENVIRONMENT": "Development",
    "DOTNET_ENVIRONMENT": "Development",
    "DOTNET_DASHBOARD_UNSECURED_ALLOW_ANONYMOUS": "true",
    "ASPIRE_ALLOW_UNSECURED_TRANSPORT": "true",
    "DOTNET_DASHBOARD_OTLP_ENDPOINT_URL": "http://localhost:19164",
    "DOTNET_RESOURCE_SERVICE_ENDPOINT_URL": "http://localhost:20178"
  }
}

Having composed containers be automatically configured to trust the ASP.NET Core Development Certificate is something we want to investigate enabling in the future.

For images that are being pulled, wouldn't forcing the container to trust a certificate that shouldn't be trusted defeat the whole purpose of the security added to Aspire?

It would probably be better to use HTTP endpoints for OTEL. Or both HTTP and HTTPS.

[DamianEdwards]

For images that are being pulled, wouldn't forcing the container to trust a certificate that shouldn't be trusted defeat the whole purpose of the security added to Aspire?

I don't think so. Configuring a container to trust a self-signed certificate that is already trusted by the host for development purposes (and only at development time) seems no different to what we do today WRT to trusting the ASP.NET Core Development Certificate for the purposes of localhost HTTPS testing. The trust would only happen when running during local development as the certificate is only for localhost anyway.

Using an unencrypted transport for OTEL means processes running as other accounts on the same machine during local development can potentially observe sensitive information being passed from apps to the dashboard's OTLP endpoint so we decided that behavior must be opt-in.