Repository navigation
Report multi-endpoint routing, delivery and storage loss through the event source - #62925
Merged
Harsimar Kaur (harsimar) merged 9 commits intoSep 11, 2026
Conversation
…event source A routed export could not be diagnosed from a trace. An Activity whose routing tags were missing or invalid was dropped by a bare continue, hitting the 64 partition cap returned null, and eviction deleted another endpoint's stored telemetry, all without an event. Each export now carries a sequence number that ties its per-item records to its totals and to the delivery of each endpoint's batch. Informational reports the totals, per-endpoint delivery, eviction naming both endpoints, and a refused partition. Verbose reports each collected Activity with its destination and ids, and each rejected one with the reason it failed validation. Rejection reasons never repeat the endpoint, because credentials in the URI is one of the reasons. TelemetryDebugWriter now names the destination for routed payloads, for live sends and storage replays alike. AB#39637603
…ivities in the demo DescribeOwner was an argument to the event method, so it ran on every eviction even with nothing listening, walking the partitions and allocating. It now sits behind IsEnabled. The demo leaves every tenth Activity unroutable, cycling through missing key, missing endpoint and a non-HTTPS endpoint, and prints the reasons it expects so a run can be reconciled against what the event source reported. Drops the CHANGELOG entry: no public API or behaviour changed.
… events allocating when disabled Events 71 and 72 take parameter shapes with no typed WriteEvent overload, so the call allocated an argument array whether or not anything listened. Both now check IsEnabled before writing. A 206 accepts part of a batch and persists only the rest, but the outcome described the whole group as persisted or dropped. The event now carries what ingestion accepted and distinguishes partial delivery. A send that threw returned without any outcome at all, so an unreachable endpoint - the case this diagnostic exists for - left only collected events and a summary. It now reports what became of the batch. Also corrects a comment that contradicted the code, and adds tests for sequence distinctness and the throwing path.
…tcome A 206 accepts some items, persists the retryable ones and rejects the rest outright. The outcome said 'remainder persisted', which is false when part of the remainder was rejected, and it could also report a group as dropped while naming a non-zero accepted count. It now reports 'partially transmitted' and asserts only the accepted number. An accepted count the batch cannot support, or none at all, is reported as -1 rather than conflated with zero. Disposing the HTTP message can throw after delivery was settled and reported, which produced a second contradictory outcome for the same group. The catch now reports only when nothing was reported yet. Adds tests for the mixed 206 case and for an unusable accepted count.
Logging had grown to roughly 35 lines and four multi-line call sites inside the method that delivers telemetry, which is more risk than a diagnostic should add to that path. Each exit now reports in one line through a single helper. The reported flag is gone: it existed only so a throwing HttpMessage.Dispose could not log a second outcome, which required a mock that throws on disposal to trigger and is not worth a permanent branch in the send path. That case can log two outcomes for one group. Behaviour is unchanged; every exit returns exactly what it returned before.
A 206 settles each item separately. When it accepted nothing the group was reported as persisted, but only the retryable subset is kept and the rest is rejected outright. Any 206 is now reported as partially accepted, asserting only the accepted number. An exception after ingestion answered reported acceptance as zero and status as zero, hiding that a response had been received. Acceptance is now unknown and the observed status is preserved. Adds coverage for a 206 that accepted nothing, and for the two length-limit rejection reasons the theory had omitted.
Rajkumar Rangaraj (rajkumar-rangaraj)
marked this pull request as ready for review
September 11, 2026 19:20
Copilot started reviewing on behalf of
Rajkumar Rangaraj (rajkumar-rangaraj)
September 11, 2026 19:21
View session
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Persisted batches report an inaccurate accepted count, and previous-run eviction events cannot identify the affected endpoint.
Get a fresh assessment by requesting another Copilot review.
Pull request overview
Adds EventSource diagnostics for multi-endpoint routing, delivery, rejection, partition limits, and storage eviction.
Changes:
- Correlates routed exports using sequence numbers.
- Reports routing decisions and delivery outcomes.
- Expands tests and the multi-tenant demo.
File summaries
| File | Description |
|---|---|
MultiTenantStorageTests.cs |
Tests partition and eviction diagnostics. |
MultiTenantRoutingTests.cs |
Tests routing events and correlation. |
MultiTenantIntegrationTests.cs |
Tests endpoint delivery outcomes. |
MultiTenantTraceDemo.cs |
Generates unroutable demo activities. |
Program.cs |
Displays expected rejection totals. |
TraceHelper.cs |
Emits routing records and summaries. |
TenantRouting.cs |
Returns detailed rejection reasons. |
RoutingRejectionReason.cs |
Defines routing failure categories. |
MultiTenantStorage.cs |
Reports partition refusal and eviction. |
EndpointRouteBatch.cs |
Adds export sequence tracking. |
BudgetedBlobProvider.cs |
Supplies the requesting endpoint. |
TelemetryDebugWriter.cs |
Includes routed destinations in debug output. |
AzureMonitorExporterEventSource.cs |
Defines the new diagnostic events. |
AzureMonitorTransmitter.cs |
Reports per-endpoint delivery outcomes. |
ApplicationInsightsRestClient.cs |
Passes destinations to debug output. |
AzureMonitorTraceExporter.cs |
Initializes each export sequence. |
Review details
- Files reviewed: 16/16 changed files
- Comments generated: 2
- Review effort level: Balanced
💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.
…t-diagnostics # Conflicts: # sdk/monitor/Azure.Monitor.OpenTelemetry.Exporter/src/Internals/MultiTenant/TenantRouting.cs # sdk/monitor/Azure.Monitor.OpenTelemetry.Exporter/tests/Azure.Monitor.OpenTelemetry.Exporter.Demo/Traces/MultiTenantTraceDemo.cs
…, and do not present a directory as an endpoint A group persisted because of back-off or persist-only never reaches ingestion, so acceptance is unknown rather than zero. The default is now -1, matching the exception path. SelectOldest reaches partitions left by earlier runs, and nothing in this process maps those directories back to an endpoint. The eviction event now reports the directory name and says so, instead of printing a full path where an endpoint was promised.
…tation AzureMonitorLogExporter never called BeginExport, so every routed log delivery reported sequence 0 while traces through the same transmitter reported unique ones. Caught in review by harsimar; the gap appeared when logs routing merged from main after the trace path was wired. CI failed on a rejection-reason row for '/relative/path'. Linux and macOS parse a leading slash as an absolute file URI, so the reason is IngestionEndpointNotHttps there and IngestionEndpointMalformed on Windows. The row is removed: 'not-a-uri' already covers that reason on every platform, and ActivityWithoutAValidRouteIsDropped still covers the rejection itself.
Harsimar Kaur (harsimar)
approved these changes
Sep 11, 2026
This was referenced Oct 1, 2026
Open
This was referenced Oct 8, 2026
Merged
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Routed (multi-endpoint) export could not be diagnosed from a trace. An Activity with missing or invalid routing tags was dropped by a bare
continue, hitting the 64 partition cap returnednull, and eviction deleted another endpoint's stored telemetry — none of it reported.Adds self-diagnostics on the existing
OpenTelemetry-AzureMonitor-Exporterevent source.Informational — per-export totals (collected, endpoints, rejected); per-endpoint delivery with item count, accepted count and status code; eviction, naming what lost the telemetry (the endpoint, or the storage directory when the partition was left by an earlier run) and the endpoint whose write caused it; a partition refused at the cap.
Verbose — each collected Activity with its destination, instrumentation key, trace id and span id; each rejected Activity with the reason it failed validation.
Every event from one export carries the same sequence number, so per-item records tie to the totals and to the delivery of the batch they became part of.
Routed paths only: no single-endpoint behaviour change, no public API change, no customer SDK stats change. Per-item work sits behind an
IsEnabledcheck. Telemetry bodies are never written to the event source — trace and span id identify an item without carrying customer data, and rejection reasons never echo the endpoint, since credentials in the URI is one of the reasons.TelemetryDebugWriternow names the destination for routed payloads, unchanged in being DEBUG-only and debugger-gated.Validated with 1,098 tests on net8.0, net9.0, net10.0 and net462, and a live run against four Application Insights components across three regions: 400 activities, 360 collected over 3 endpoints, 40 dropped with their reasons, per-component arrivals matching intent, and nothing falling back to the host.
AB#39637603