Fix broken parenting when Ocelot API Gateway is used#8128
Open
Fix broken parenting when Ocelot API Gateway is used#8128
Conversation
bouwkast
added
the
AI Generated
BenchmarksBenchmark execution time: 2026-04-27 13:32:01 Comparing candidate commit eff04e0 in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 27 metrics, 0 unstable metrics, 61 known flaky benchmarks, 26 flaky benchmarks without significant changes.
|
Execution-Time Benchmarks Report ⏱️Execution-time results for samples comparing This PR (8128) and master. ✅ No regressions detected - check the details below Full Metrics ComparisonFakeDbCommand
HttpMessageHandler
Comparison explanationExecution-time benchmarks measure the whole time it takes to execute a program, and are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are highlighted in **red**. The following thresholds were used for comparing the execution times:
Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard. Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph). Duration chartsFakeDbCommand (.NET Framework 4.8)gantt
title Execution time (ms) FakeDbCommand (.NET Framework 4.8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8128) - mean (73ms) : 69, 77
master - mean (73ms) : 70, 76
section Bailout
This PR (8128) - mean (80ms) : 76, 84
master - mean (76ms) : 75, 78
section CallTarget+Inlining+NGEN
This PR (8128) - mean (1,099ms) : 1023, 1176
master - mean (1,080ms) : 1033, 1126
FakeDbCommand (.NET Core 3.1)gantt
title Execution time (ms) FakeDbCommand (.NET Core 3.1)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8128) - mean (117ms) : 110, 124
master - mean (116ms) : 109, 124
section Bailout
This PR (8128) - mean (116ms) : 110, 122
master - mean (114ms) : 112, 117
section CallTarget+Inlining+NGEN
This PR (8128) - mean (779ms) : 748, 809
master - mean (778ms) : 748, 809
FakeDbCommand (.NET 6)gantt
title Execution time (ms) FakeDbCommand (.NET 6)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8128) - mean (104ms) : 99, 110
master - mean (105ms) : 98, 111
section Bailout
This PR (8128) - mean (104ms) : 99, 110
master - mean (102ms) : 98, 107
section CallTarget+Inlining+NGEN
This PR (8128) - mean (941ms) : 896, 985
master - mean (940ms) : 902, 978
FakeDbCommand (.NET 8)gantt
title Execution time (ms) FakeDbCommand (.NET 8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8128) - mean (100ms) : 97, 103
master - mean (102ms) : 96, 107
section Bailout
This PR (8128) - mean (104ms) : 99, 110
master - mean (106ms) : 100, 112
section CallTarget+Inlining+NGEN
This PR (8128) - mean (826ms) : 789, 862
master - mean (822ms) : 788, 856
HttpMessageHandler (.NET Framework 4.8)gantt
title Execution time (ms) HttpMessageHandler (.NET Framework 4.8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8128) - mean (204ms) : 196, 213
master - mean (202ms) : 194, 211
section Bailout
This PR (8128) - mean (208ms) : 200, 216
master - mean (207ms) : 199, 215
section CallTarget+Inlining+NGEN
This PR (8128) - mean (1,209ms) : 1153, 1266
master - mean (1,199ms) : 1148, 1250
HttpMessageHandler (.NET Core 3.1)gantt
title Execution time (ms) HttpMessageHandler (.NET Core 3.1)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8128) - mean (294ms) : 281, 308
master - mean (289ms) : 279, 299
section Bailout
This PR (8128) - mean (296ms) : 282, 309
master - mean (288ms) : 279, 298
section CallTarget+Inlining+NGEN
This PR (8128) - mean (961ms) : 932, 989
master - mean (947ms) : 923, 972
HttpMessageHandler (.NET 6)gantt
title Execution time (ms) HttpMessageHandler (.NET 6)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8128) - mean (290ms) : 277, 302
master - mean (283ms) : 272, 293
section Bailout
This PR (8128) - mean (288ms) : 275, 302
master - mean (281ms) : 271, 291
section CallTarget+Inlining+NGEN
This PR (8128) - mean (1,164ms) : 1133, 1196
master - mean (1,152ms) : 1108, 1196
HttpMessageHandler (.NET 8)gantt
title Execution time (ms) HttpMessageHandler (.NET 8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8128) - mean (287ms) : 272, 302
master - mean (282ms) : 268, 297
section Bailout
This PR (8128) - mean (288ms) : 274, 301
master - mean (284ms) : 270, 299
section CallTarget+Inlining+NGEN
This PR (8128) - mean (1,050ms) : 991, 1108
master - mean (1,046ms) : 975, 1116
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
bouwkast
force-pushed
the
steven/ocelot-instrumentation
branch
from
ae314c0 to
88228c0
Compare
bouwkast
force-pushed
the
steven/ocelot-instrumentation
branch
from
b9bec75 to
1547cf8
Compare
bouwkast
force-pushed
the
steven/ocelot-instrumentation
branch
from
1547cf8 to
7d03568
Compare
bouwkast
changed the title
bouwkast
force-pushed
the
steven/ocelot-instrumentation
branch
2 times, most recently
from
2340187 to
6aaa0a6
Compare
bouwkast
added
type:bug
area:automatic-instrumentation
bouwkast
force-pushed
the
steven/ocelot-instrumentation
branch
from
6aaa0a6 to
1ab66fb
Compare
bouwkast
force-pushed
the
steven/ocelot-instrumentation
branch
from
1ab66fb to
334375f
Compare
Member
So just to confirm, the original issue doesn't repro on earlier versions? |
andrewlock
reviewed
Mar 19, 2026
Member
andrewlock
left a comment
andrewlock
left a comment
There was a problem hiding this comment.
LGTM, just a couple of minor things:
- We generally try to cover the full range of supported TFMs, so we should probably support .NET 6+ as Ocelot does
- Should this be the
HttpMessageHandleror a newOcelotintegration? - Couple of nits (using the more compact aspnetcore setup, using combinatorial data attr etc for future)
| [EditorBrowsable(EditorBrowsableState.Never)] | ||
| public sealed class OcelotMessageInvokerPoolIntegration | ||
| { | ||
| private const string IntegrationName = nameof(Configuration.IntegrationId.HttpMessageHandler); |
Member
There was a problem hiding this comment.
Should this be a new integration, Ocelot, instead of HttpMessageHandler? 🤔 I'm undecided tbh, I get the reasoning.
Collaborator
Author
There was a problem hiding this comment.
Yeah, this initially was Ocelot but Yarp went with HttpMessageHandler so I just forced Claude to follow that
Honestly I'm not sure what is ideal
Collaborator
Author
There was a problem hiding this comment.
I could probably go both ways, but we aren't really "instrumenting" YARP/Ocelot here (YARP uses HttpMessageHandler as well).
I think based on that and that we are essentially attempting to fix re-parenting issues here not calling it Ocelot is the better representation, but I would say that if for some reason someone needs to disable this re-parenting fix I guess they would ideally want the option to do that without having to disable all of the HttpMessageHandler instrumentation.
Maybe a second PR to update both YARP and this to be controlled by either HttpMessageHandler or their integration name itself?
andrewlock
approved these changes
Mar 19, 2026
bouwkast
changed the title
bouwkast
force-pushed
the
steven/ocelot-instrumentation
branch
from
5cd33a5 to
566fef8
Compare
Collaborator
Author
Correct I couldn't get it to reproduce in earlier versions |
bouwkast
force-pushed
the
steven/ocelot-instrumentation
branch
from
ce38e3e to
780e797
Compare
Member
|
@codex review |
|
Codex Review: Didn't find any major issues. 🎉 ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
bouwkast
force-pushed
the
steven/ocelot-instrumentation
branch
from
3a108a5 to
ae9d877
Compare
bouwkast
force-pushed
the
steven/ocelot-instrumentation
branch
from
ae9d877 to
a0df7a6
Compare
NachoEchevarria
approved these changes
Apr 14, 2026
bouwkast
force-pushed
the
steven/ocelot-instrumentation
branch
from
a0df7a6 to
31f994e
Compare
bouwkast
added a commit
that referenced
this pull request
Apr 22, 2026
## Summary of changes This goes back through the recent changes (#8371 and #8340) to `GeneratePackageVersions` and attempts to do some removals, refactorings, and fixes that have come up. Notably this removes the `nuget_cache`, fixes `IncludePackages` (again 😅), and rewrites the cooldown logic to not use `supported_versions.json` and `nuget_cache` and instead reads from the `.g.cs` files in addition to refactoring the verbage used throughout. ## Reason for change While working on (#8128) I started running into several issues with `GeneratePackageVersions` namely that I couldn't get `IncludePackages` to work and then I started fighting issues with the `nuget_cache` as well when attempting to add a new integration. This sets out to do a few things: - Simplify the logic for NuGet cooldown (automated dependency updates must adhere to our current cooldown policy for potential supply chain attacks) - Remove the `nuget_cache` - this was added so that we wouldn't go and query NuGet info for packages not listed in the `IncludePackages` - Change the "verbage" of the cooldown/baseline to be easier to read / follow - Do NOT read / rely upon `supported_versions` (this is what I previously used to determine what the highest version was currently allowed) - this file is planned to be changed/modified/removed by another team in the near future hence why we shouldn't use it. - I didn't account for different timezones in the publish state of the `nuget_cache` before so if someone ran it depending on their locale it would change. ## Implementation details - New `CooldownMode` `enum`: `Normal`, `BypassCooldown`, `Freeze` - Normal drops recent versions that are above the previous max - BypassCooldown accepts all in-range versions (no cooldown) (when `IncludePackages` is used is an example) - Freeze re-emits the previous run's output verbatim - `XUnitFileGenerator.LoadExistingVersions` reads the previous `.g.cs` and returns `Dictionary<IntegrationName, List<(Framework, Versions)>>`. `PackageGroup` loads this once and exposes `TryGetFrozenVersions` - I think we should probably still try and do something different here but this seems a bit more reasonable - Deleted `nuget_version_cache.json` and `NuGetVersionCache.cs` - No longer used/needed - Some readability improvements - `IsWithinCooldown` → `WasPublishedTooRecently` - `baseline` → `previousMaxVersions` - `ApplyCooldown` now uses two named local booleans (`publishedTooRecently`, `atOrBelowPreviousMax`) so the drop condition reads positively: "drop the version if it was published too recently **and** we haven't already shipped against it." ## Test coverage I ran it locally for a handful of cases I think it is better. ## Other details <!-- Fixes #{issue} --> I have ran through several instances with this and it _seems_ to be good, basically default runs, runs with configurable cooldowns, runs with `IncludePackages` and runs with `ExcludePackages`. All appeared as expected to me. <!--⚠️ Note: Where possible, please obtain 2 approvals prior to merging. Unless CODEOWNERS specifies otherwise, for external teams it is typically best to have one review from a team member, and one review from apm-dotnet. Trivial changes do not require 2 reviews. MergeQueue is NOT enabled in this repository. If you have write access to the repo, the PR has 1-2 approvals (see above), and all of the required checks have passed, you can use the Squash and Merge button to merge the PR. If you don't have write access, or you need help, reach out in the #apm-dotnet channel in Slack. --> --------- Co-authored-by: Andrew Lock <andrew.lock@datadoghq.com>
This resolves a reparenting issue when OpenTelemetry.Instrumentation.Http is present and running.
This demonstrates the difference in parenting when we have Ocelot and OpenTelemetry compared to when we have the instrumenation that gets rid of the parenting misconfiguration. Note that in this snapshot there is an "ID_3" referenced as the "ParentId", but that there is no "ID_3" present in the snapshot.
Just doing to this help with reviewers to see what changed / what it should be
Cover Ocelot 23.x which supports net6.0-net8.0, not just net8.0+.
bouwkast
force-pushed
the
steven/ocelot-instrumentation
branch
from
ea34469 to
eff04e0
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary of changes
Add auto-instrumentation for Ocelot API gateway to fix broken distributed traces when Ocelot is used with
DD_TRACE_OTEL_ENABLEDandOpenTelemetry.Instrumentation.Httpis used.Reason for change
When customers use Ocelot v23.0.0+ alongside the
OpenTelemetry.Instrumentation.HttpNuGet package, distributed traces break. Ocelot creates aSocketsHttpHandlerwith a defaultActivityHeadersPropagator(this is normal behavior), which causesDiagnosticsHandlerto be added to the handler chain. When the OpenTelemetry HTTP instrumentation is registered, it subscribes to theDiagnosticSourceevents fromDiagnosticsHandlerand injects its own trace context headers into downstream requests which overwrites thex-datadog-*/ W3Ctraceparentheaders already set by ourHttpClientinstrumentation.Without the OpenTelemetry HTTP instrumentation NuGet installed, the
DiagnosticsHandlerActivity events have no subscriber to inject competing headers, so the issue does not occur.For customers this results in broken parenting for traces/spans.
Implementation details
OcelotMessageInvokerPoolIntegration: InstrumentsOcelot.Requester.MessageInvokerPool.CreateHandlerto setActivityHeadersPropagator = nullon the returnedSocketsHttpHandler, preventingDiagnosticsHandlerfrom being added to the handler chain.[InstrumentMethod]attributes handle the return type change between versions:HttpMessageHandlerSocketsHttpHandlerNote: YARP needs to be updated as well from
CreateNoOutputPropagator()tonullI'll do this in a follow upTest coverage
Samples.Ocelot.DistributedTracingthat has Ocelot configured as a reverse proxy andOpenTelemetry.Instrumentation.Httpinstalled and setup. Note that not havingOpenTelemetry.Instrumentation.Httpcauses it to not re-produce.OcelotDistributedTracingTestswhich follow the same pattern as the YARP testsOther details
Supported Ocelot versions: 23.0.0 to 24.*
Earlier versions do not have the same behavior as
HttpMessageHandlerwas introduced first in 23 and then changed toSocketsHttpHandlerin 24.