[Monitor OpenTelemetry Exporter] Persist telemetry on shutdown for short-lived applications

[Rajkumar Rangaraj (rajkumar-rangaraj)]

Problem

The exporter is network-first, so short-lived applications lose telemetry. A 2-second dotnet build never reaches a batch interval (5s for traces/logs, 60s for metrics), so its only export happens at shutdown ΓÇö and that export is a blocking POST. MaxRetries is 0 but NetworkTimeout defaults to 100 seconds, so exit either pays a full ingestion round trip or stalls. If the process exits first, the batch is gone; nothing durable was written.

Telemetry that did reach storage was never uploaded either: the drain timer's first tick is at t+120s.

Change

Shutdown/Dispose now write pending telemetry to offline storage and drain it in the background instead of transmitting inline. A blob is deleted only after HTTP 200, so an abrupt exit cannot lose data ΓÇö the next run drains the backlog.

The persist-only scope is opened by the processor, not the exporter, because BaseExportProcessor.OnShutdown exports the remaining batch before it shuts the exporter down. Same for BaseExportingMetricReader.

ForceFlush is unchanged by default ΓÇö callers that flush per invocation expect delivery, not durability. Two AppContext switches: PersistOnForceFlush opts in, DisablePersistOnShutdown reverts. PersistOnForceFlush applies to traces and logs only — a metric reader cannot distinguish a caller's flush from its periodic collection, so metric ForceFlush always transmits.

Two independent budgets. The drain budget may be zero (data is already on disk); the fallback transmission used when storage is unavailable gets its own non-zero budget, since there the request is the durability.

Usage

There is nothing new to call and nothing to configure. Existing code gets the behavior as long as the provider is shut down or disposed, which is what makes the telemetry get written to disk.

Console app / CLI:

using System.Diagnostics;
using Azure.Monitor.OpenTelemetry.Exporter;
using OpenTelemetry;
using OpenTelemetry.Trace;

var activitySource = new ActivitySource("Contoso.Cli");

// Disposing the provider is what triggers the persist-then-upload path.
using (var tracerProvider = Sdk.CreateTracerProviderBuilder()
    .AddSource("Contoso.Cli")
    .AddAzureMonitorTraceExporter(o => o.ConnectionString = connectionString)
    .Build())
{
    using var activity = activitySource.StartActivity("build");
    RunCommand();
}

// Exit is a file write, not an ingestion round trip. Delivery is completed by a
// background drain here or, if the process exits first, by the next run.

Host-based app ΓÇö no change; IHost disposes the providers:

var builder = Host.CreateApplicationBuilder(args);
builder.Services.AddOpenTelemetry().UseAzureMonitorExporter();

await builder.Build().RunAsync();

Opting ForceFlush into the same behavior, or reverting shutdown to the previous blocking transmission ΓÇö set before building the provider:

AppContext.SetSwitch("Azure.Monitor.OpenTelemetry.Exporter.PersistOnForceFlush", true);
AppContext.SetSwitch("Azure.Monitor.OpenTelemetry.Exporter.DisablePersistOnShutdown", true);

Two caveats worth knowing:

• Offline storage must be available. With DisableOfflineStorage = true, or on a read-only filesystem, there is nowhere to persist and shutdown falls back to a transmission ΓÇö now capped at 3 seconds instead of inheriting the 100 second network timeout.
• Ingestion rejects telemetry older than 48 hours, so a run whose machine then sits idle past that is unrecoverable regardless.

Also fixed (pre-existing)

• Leased blobs stranded permanently. TryLease renames to *.lock, which matches neither the provider's "*.blob" enumeration nor its retention sweep. Only the 120s maintenance tick reclaims it ΓÇö which a short-lived process never reaches, so a run that died mid-upload stranded that telemetry forever. Expired leases are now reclaimed on drain.
• Telemetry silently dropped at the storage size cap, with no eviction. Now evicts oldest-first, and only when the directory is genuinely full.
• TransmitFromStorageHandler was never disposed (leaked timer).
• Drain now coalesces blobs into one request and goes oldest-first, instead of one request per blob newest-first.
• The shared transmitter is reference counted, so disposing one exporter no longer stops storage draining for the others sharing its connection string.

Notes

• No public API change ΓÇö api/*.cs is untouched, verified by regenerating with eng/scripts/Export-API.ps1. The overrides are protected on internal processor and reader types; putting them on the exporters themselves would be a surface change.
• Statsbeat deliberately keeps the legacy reader: it sets DisableOfflineStorage = true, so it would land on the fallback path and add exit latency for internal telemetry whose loss is acceptable.
• A coalesced batch rejected outright is retried constituent-by-constituent, so one bad blob can't discard good ones.
• Storage retention stays at the package default of two days; extending it would only hold payloads ingestion is guaranteed to refuse.

Testing

866 tests pass. New coverage in PersistOnShutdownTests (transmitter/drain) and PersistOnShutdownProviderTests (real TracerProvider): shutdown and dispose persist rather than transmit, both switches, lease reclamation incl. the unexpired case, coalescing, poison isolation, 206 partial success in both directions, capacity-gated eviction, shared transmitter lifetime, and a 30-second hanging endpoint no longer stalling exit (~2s).

Not yet covered, tracked for follow-up: exit/startup latency benchmarks, and concurrency tests for parallel MSBuild nodes sharing one storage directory.

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).
11 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Pull request overview

This PR improves reliability for short-lived applications using the Azure Monitor OpenTelemetry Exporter by persisting pending telemetry to offline storage during shutdown/dispose and draining it asynchronously, rather than performing a blocking ingestion request on the exit path. It also strengthens offline-storage draining behavior (lease reclamation, eviction at size cap, batching strategy) and adds targeted test coverage.

Changes:

• Persist pending telemetry on provider shutdown/dispose (and optionally ForceFlush) via a persist-only scope opened at the processor/reader level, then trigger a bounded background drain.
• Improve offline storage drain mechanics: eager drain shortly after startup, reclaim expired leases, coalesce blobs oldest-first, and evict oldest blobs when storage is full.
• Add new unit tests covering shutdown persistence, switch behavior, lease reclamation, coalescing/isolation behavior, and eviction.

Reviewed changes

Copilot reviewed 26 out of 26 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
sdk/monitor/Azure.Monitor.OpenTelemetry.Exporter/tests/Azure.Monitor.OpenTelemetry.Exporter.Tests/PersistOnShutdownTests.cs	New transmitter-level tests for persist-only scope and storage drain behaviors.
sdk/monitor/Azure.Monitor.OpenTelemetry.Exporter/tests/Azure.Monitor.OpenTelemetry.Exporter.Tests/PersistOnShutdownSwitchCollection.cs	New xUnit collection to serialize tests that mutate AppContext switches.
sdk/monitor/Azure.Monitor.OpenTelemetry.Exporter/tests/Azure.Monitor.OpenTelemetry.Exporter.Tests/PersistOnShutdownProviderTests.cs	New end-to-end provider pipeline tests validating shutdown/dispose persistence behavior and switches.
sdk/monitor/Azure.Monitor.OpenTelemetry.Exporter/tests/Azure.Monitor.OpenTelemetry.Exporter.Tests/OfflineStorageTests.cs	Disable eager drain during these tests to avoid storage-content races.
sdk/monitor/Azure.Monitor.OpenTelemetry.Exporter/tests/Azure.Monitor.OpenTelemetry.Exporter.Tests/CommonTestFramework/MockTransmitter.cs	Update test mock to match new ITransmitter surface (persist-only scope + drain).
sdk/monitor/Azure.Monitor.OpenTelemetry.Exporter/src/OpenTelemetryBuilderExtensions.cs	Swap metric reader registration to a custom reader that can persist-on-shutdown.
sdk/monitor/Azure.Monitor.OpenTelemetry.Exporter/src/Internals/TransmitFromStorageHandler.cs	Major rework of drain logic: eager drain, lease reclamation, batching/coalescing, and disposal fixes.
sdk/monitor/Azure.Monitor.OpenTelemetry.Exporter/src/Internals/PersistOnShutdownHelper.cs	New helper to run shutdown/flush in persist-only mode then trigger drain.
sdk/monitor/Azure.Monitor.OpenTelemetry.Exporter/src/Internals/PersistOnShutdownConfig.cs	New configuration + AppContext switches and drain/fallback budgets.
sdk/monitor/Azure.Monitor.OpenTelemetry.Exporter/src/Internals/PersistentStorage/PersistentStorageExtensions.cs	Add save-with-eviction behavior when offline storage hits its size cap.
sdk/monitor/Azure.Monitor.OpenTelemetry.Exporter/src/Internals/LogFilteringProcessor.cs	Route log filtering processor through the new persist-capable batch processor base.
sdk/monitor/Azure.Monitor.OpenTelemetry.Exporter/src/Internals/ITransmitter.cs	Extend transmitter contract to support persist-only scope and bounded drain trigger.
sdk/monitor/Azure.Monitor.OpenTelemetry.Exporter/src/Internals/HttpPipelineHelper.cs	Use save-with-eviction for retry persistence and expose retriable-status helper to internals.
sdk/monitor/Azure.Monitor.OpenTelemetry.Exporter/src/Internals/Diagnostics/AzureMonitorExporterEventSource.cs	Add events for persist-on-shutdown failures and coalesced-batch rejection isolation.
sdk/monitor/Azure.Monitor.OpenTelemetry.Exporter/src/Internals/CustomerSdkStats/DropCodeExtensions.cs	Map new shutdown-persisted drop code to spec bucket.
sdk/monitor/Azure.Monitor.OpenTelemetry.Exporter/src/Internals/CustomerSdkStats/DropCode.cs	Add new drop code for shutdown persistence.
sdk/monitor/Azure.Monitor.OpenTelemetry.Exporter/src/Internals/AzureMonitorTransmitter.cs	Implement persist-only mode, drain triggering/wait budgeting, retention change, and dispose drain handling.
sdk/monitor/Azure.Monitor.OpenTelemetry.Exporter/src/Internals/AzureMonitorPeriodicExportingMetricReader.cs	New metric reader overriding shutdown to persist then drain.
sdk/monitor/Azure.Monitor.OpenTelemetry.Exporter/src/Internals/AzureMonitorBatchLogRecordExportProcessor.cs	New batch log processor overriding shutdown/flush to persist then drain (switch-controlled).
sdk/monitor/Azure.Monitor.OpenTelemetry.Exporter/src/Internals/AzureMonitorBatchActivityExportProcessor.cs	New batch trace processor overriding shutdown/flush to persist then drain (switch-controlled).
sdk/monitor/Azure.Monitor.OpenTelemetry.Exporter/src/ExporterRegistrationHostedService.cs	Register the new persist-capable processors/reader in hosted-service wiring.
sdk/monitor/Azure.Monitor.OpenTelemetry.Exporter/src/AzureMonitorTraceExporter.cs	Ensure exporter shutdown triggers a drain when user supplies a custom processor.
sdk/monitor/Azure.Monitor.OpenTelemetry.Exporter/src/AzureMonitorMetricExporter.cs	Ensure exporter shutdown triggers a drain when user supplies a custom reader.
sdk/monitor/Azure.Monitor.OpenTelemetry.Exporter/src/AzureMonitorLogExporter.cs	Ensure exporter shutdown triggers a drain when user supplies a custom processor.
sdk/monitor/Azure.Monitor.OpenTelemetry.Exporter/src/AzureMonitorExporterExtensions.cs	Update extension wiring to use new processors/reader.
sdk/monitor/Azure.Monitor.OpenTelemetry.Exporter/CHANGELOG.md	Document new shutdown persistence behavior, switches, and offline-storage fixes/retention change.

[Noah Gilson (nagilson)]

Thank you for engaging with us on #60838. I'm grateful for the time you put into these changes. I left a follow-up from our email thread.

[xiang17]

No objection for the code. Comments are doc feedback or not blocking for this PR.