Configure GeneratePackageVersions to support a "cooldown" on dependencies

[bouwkast]

Summary of changes

default cooldown period is 2 days

Adds a configurable cooldown period to the GeneratePackageVersions to support remediation efforts for follow up after #incident-51602.

To use supply the optional parameter --PackageVersionCooldownDays X where X is some number of days. The current period is at the moment is going to be 2 days and is the default now when the overall GeneratePackageVersion target is ran. Additionally, this is overridden to 0 if --IncludePackages is supplied (this is commonly used when working on a singular package locally).

After running the tool a "cooldown" report is generated, this file will contain packages that we see have a newer version, but will not incorporate into the test, the fallback version that falls within the cooldown period is provided. This file content will show up in the output of future test package version bump PRs.

Note that this is for automated updates, so if it sees something already updated it will honor it.

Here's an example output ran locally with 14 days set:

## Package Version Cooldown Report

The following versions were published less than **14 days** ago and have been overridden.
These require manual review before inclusion.

| Package | Integration | Overridden Version | Published | Age (days) | Using Instead |
|---------|-------------|--------------------|-----------|------------|---------------|
| AWSSDK.Core | AwsSdk | 4.0.3.22 | 2026-03-25 | 0 | 4.0.3.21 |
| AWSSDK.S3 | AwsS3 | 4.0.19.2 | 2026-03-25 | 0 | 4.0.19.1 |
| StackExchange.Redis | StackExchangeRedis | 2.12.8 | 2026-03-25 | 0 | 2.12.4 |

Reason for change

In #8364 and #incident-51602 all automated dependency updaters to be disabled temporarily, to re-enable we need to supply a 2 day "cooldown" to any version that we update to (in other words the version of the NuGet must be published for at least 2 days before we can update to it).

Implementation details

I made Claude do this 🤖

• NuGetPackageHelper now captures the Published date from IPackageSearchMetadata via a new VersionWithDate record (previously discarded)
• NuGetVersionCache stores the new {Version, Published} format
• PackageVersionGenerator.ApplyCooldown filters selected versions after LatestMajors/LatestMinors/LatestSpecific selection:
  • Versions outside the cooldown window pass through unchanged
  • Versions at or below the baseline (derived from supported_versions.json MaxVersionTestedInclusive) are kept even if
    within cooldown -- no downgrades
  • Versions above the baseline and within cooldown are overridden to the best available fallback
• CooldownReport collects overridden versions and renders a markdown table saved to tracer/build/cooldown_report.md
• The GitHub Actions workflow reads the report and appends it to the auto-bump PR body
• Honeypot IntegrationGroups.cs fixes: MSTest.TestFramework now maps to itself, Hangfire.Core maps to Hangfire.Core (was Hangfire), OpenFeature mapping moved to Datadog.FeatureFlags.OpenFeature

Passing --IncludePackages will override the cooldown to 0

Test coverage

I ran GeneratePackageVersions --PackageVersionCooldownDays 14 locally seems good enough IMO (also ran without, with different days etc)

[WRN] GeneratePackageVersi: 3 package version(s) were excluded due to the 14-day cooldown period
[WRN] GeneratePackageVersi:   AWSSDK.Core 4.0.3.22 overridden (published 2026-03-25, using: 4.0.3.21)
[WRN] GeneratePackageVersi:   AWSSDK.S3 4.0.19.2 overridden (published 2026-03-25, using: 4.0.19.1)
[WRN] GeneratePackageVersi:   StackExchange.Redis 2.12.8 overridden (published 2026-03-25, using: 2.12.4)

Other details

The workflow file (auto_bump_test_package_versions.yml) will be re-enabled with this PR

[pr-commenter]

Benchmarks

Benchmark execution time: 2026-03-30 20:19:33

Comparing candidate commit 580786b in PR branch steven/package-version-release-date-support with baseline commit f51d779 in branch master.

Found 3 performance improvements and 11 performance regressions! Performance is the same for 257 metrics, 17 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

• 🟩 = significantly better candidate vs. baseline
• 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:Benchmarks.Trace.AgentWriterBenchmark.WriteAndFlushEnrichedTraces netcoreapp3.1

• 🟥 execution_time [+90.602ms; +90.846ms] or [+81.970%; +82.192%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleMoreComplexBody net472

• 🟩 throughput [+8516.430op/s; +8897.049op/s] or [+7.003%; +7.316%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleMoreComplexBody net6.0

• 🟥 execution_time [+10.556ms; +15.240ms] or [+5.260%; +7.595%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody netcoreapp3.1

• 🟥 execution_time [+15.202ms; +21.191ms] or [+7.878%; +10.982%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorMoreComplexBody net6.0

• 🟥 execution_time [+9.960ms; +15.876ms] or [+5.062%; +8.069%]

scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeLegacyArgs netcoreapp3.1

• 🟥 execution_time [+20.215ms; +20.747ms] or [+11.189%; +11.484%]

scenario:Benchmarks.Trace.AspNetCoreBenchmark.SendRequest net6.0

• 🟥 execution_time [+4.989ms; +7.119ms] or [+5.813%; +8.295%]

scenario:Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces net6.0

• 🟩 execution_time [-39.159ms; -34.521ms] or [-18.142%; -15.993%]

scenario:Benchmarks.Trace.CharSliceBenchmark.OriginalCharSlice net6.0

• 🟥 execution_time [+121.533µs; +128.307µs] or [+6.353%; +6.707%]
• 🟥 throughput [-32.879op/s; -31.193op/s] or [-6.290%; -5.968%]

scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark net6.0

• 🟥 allocated_mem [+17.286KB; +17.322KB] or [+6.702%; +6.716%]

scenario:Benchmarks.Trace.Log4netBenchmark.EnrichedLog netcoreapp3.1

• 🟥 execution_time [+40.576ms; +43.796ms] or [+25.261%; +27.265%]

scenario:Benchmarks.Trace.SingleSpanAspNetCoreBenchmark.SingleSpanAspNetCore netcoreapp3.1

• 🟥 throughput [-15785431.426op/s; -14418966.789op/s] or [-6.556%; -5.988%]

scenario:Benchmarks.Trace.SpanBenchmark.StartFinishTwoScopes net6.0

• 🟩 execution_time [-20.041ms; -14.730ms] or [-9.264%; -6.809%]

[dd-trace-dotnet-ci-bot]

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing This PR (8371) and master.

⚠️ Potential regressions detected

FakeDbCommand

Metric	Master (Mean ± 95% CI)	Current (Mean ± 95% CI)	Change	Status
.NET Framework 4.8 - Baseline
duration	71.43 ± (71.50 - 71.78) ms	78.53 ± (78.56 - 79.44) ms	+9.9%	❌⬆️
.NET Framework 4.8 - Bailout
duration	76.02 ± (75.97 - 76.33) ms	95.43 ± (95.36 - 96.03) ms	+25.5%	❌⬆️

HttpMessageHandler

Metric	Master (Mean ± 95% CI)	Current (Mean ± 95% CI)	Change	Status
.NET Framework 4.8 - Baseline
duration	193.20 ± (193.07 - 194.03) ms	211.06 ± (210.99 - 212.10) ms	+9.2%	❌⬆️
.NET Framework 4.8 - Bailout
duration	197.04 ± (197.25 - 198.32) ms	216.33 ± (215.91 - 216.99) ms	+9.8%	❌⬆️
.NET Framework 4.8 - CallTarget+Inlining+NGEN
duration	1165.14 ± (1172.42 - 1183.27) ms	1244.33 ± (1242.40 - 1250.32) ms	+6.8%	❌⬆️

Full Metrics Comparison

FakeDbCommand

Metric	Master (Mean ± 95% CI)	Current (Mean ± 95% CI)	Change	Status
.NET Framework 4.8 - Baseline
duration	71.43 ± (71.50 - 71.78) ms	78.53 ± (78.56 - 79.44) ms	+9.9%	❌⬆️
.NET Framework 4.8 - Bailout
duration	76.02 ± (75.97 - 76.33) ms	95.43 ± (95.36 - 96.03) ms	+25.5%	❌⬆️
.NET Framework 4.8 - CallTarget+Inlining+NGEN
duration	1071.89 ± (1072.00 - 1078.73) ms	1089.61 ± (1097.47 - 1111.19) ms	+1.7%	✅⬆️
.NET Core 3.1 - Baseline
process.internal_duration_ms	22.35 ± (22.30 - 22.40) ms	22.41 ± (22.37 - 22.45) ms	+0.3%	✅⬆️
process.time_to_main_ms	83.67 ± (83.50 - 83.84) ms	83.67 ± (83.48 - 83.86) ms	+0.0%	✅⬆️
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	10.90 ± (10.90 - 10.91) MB	10.92 ± (10.92 - 10.93) MB	+0.2%	✅⬆️
runtime.dotnet.threads.count	12 ± (12 - 12)	12 ± (12 - 12)	+0.0%	✅
.NET Core 3.1 - Bailout
process.internal_duration_ms	22.22 ± (22.18 - 22.25) ms	22.37 ± (22.33 - 22.40) ms	+0.7%	✅⬆️
process.time_to_main_ms	85.11 ± (84.95 - 85.28) ms	85.34 ± (85.13 - 85.56) ms	+0.3%	✅⬆️
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	10.95 ± (10.94 - 10.95) MB	10.95 ± (10.94 - 10.95) MB	+0.0%	✅⬆️
runtime.dotnet.threads.count	13 ± (13 - 13)	13 ± (13 - 13)	+0.0%	✅
.NET Core 3.1 - CallTarget+Inlining+NGEN
process.internal_duration_ms	223.16 ± (221.99 - 224.32) ms	223.50 ± (222.24 - 224.76) ms	+0.2%	✅⬆️
process.time_to_main_ms	532.05 ± (530.99 - 533.11) ms	531.97 ± (530.70 - 533.24) ms	-0.0%	✅
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	48.24 ± (48.20 - 48.27) MB	48.31 ± (48.28 - 48.34) MB	+0.2%	✅⬆️
runtime.dotnet.threads.count	28 ± (28 - 28)	28 ± (28 - 28)	+0.3%	✅⬆️
.NET 6 - Baseline
process.internal_duration_ms	20.94 ± (20.91 - 20.97) ms	21.07 ± (21.04 - 21.10) ms	+0.6%	✅⬆️
process.time_to_main_ms	71.92 ± (71.77 - 72.08) ms	72.40 ± (72.22 - 72.58) ms	+0.7%	✅⬆️
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	10.62 ± (10.62 - 10.62) MB	10.65 ± (10.65 - 10.65) MB	+0.3%	✅⬆️
runtime.dotnet.threads.count	10 ± (10 - 10)	10 ± (10 - 10)	+0.0%	✅
.NET 6 - Bailout
process.internal_duration_ms	20.92 ± (20.89 - 20.96) ms	21.04 ± (21.01 - 21.08) ms	+0.6%	✅⬆️
process.time_to_main_ms	73.13 ± (72.97 - 73.29) ms	73.18 ± (73.00 - 73.36) ms	+0.1%	✅⬆️
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	10.66 ± (10.66 - 10.67) MB	10.76 ± (10.75 - 10.76) MB	+0.9%	✅⬆️
runtime.dotnet.threads.count	11 ± (11 - 11)	11 ± (11 - 11)	+0.0%	✅
.NET 6 - CallTarget+Inlining+NGEN
process.internal_duration_ms	387.20 ± (384.70 - 389.69) ms	386.90 ± (384.64 - 389.15) ms	-0.1%	✅
process.time_to_main_ms	530.58 ± (529.61 - 531.55) ms	532.89 ± (531.92 - 533.86) ms	+0.4%	✅⬆️
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	50.23 ± (50.21 - 50.26) MB	50.32 ± (50.29 - 50.34) MB	+0.2%	✅⬆️
runtime.dotnet.threads.count	28 ± (28 - 28)	28 ± (28 - 28)	+0.0%	✅⬆️
.NET 8 - Baseline
process.internal_duration_ms	19.16 ± (19.12 - 19.19) ms	19.29 ± (19.26 - 19.32) ms	+0.7%	✅⬆️
process.time_to_main_ms	71.28 ± (71.13 - 71.43) ms	71.72 ± (71.56 - 71.87) ms	+0.6%	✅⬆️
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	7.69 ± (7.68 - 7.69) MB	7.67 ± (7.67 - 7.68) MB	-0.1%	✅
runtime.dotnet.threads.count	10 ± (10 - 10)	10 ± (10 - 10)	+0.0%	✅
.NET 8 - Bailout
process.internal_duration_ms	19.25 ± (19.21 - 19.28) ms	19.40 ± (19.37 - 19.44) ms	+0.8%	✅⬆️
process.time_to_main_ms	72.88 ± (72.74 - 73.02) ms	72.87 ± (72.70 - 73.04) ms	-0.0%	✅
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	7.75 ± (7.74 - 7.75) MB	7.76 ± (7.75 - 7.77) MB	+0.2%	✅⬆️
runtime.dotnet.threads.count	11 ± (11 - 11)	11 ± (11 - 11)	+0.0%	✅
.NET 8 - CallTarget+Inlining+NGEN
process.internal_duration_ms	308.00 ± (305.65 - 310.34) ms	308.44 ± (305.94 - 310.95) ms	+0.1%	✅⬆️
process.time_to_main_ms	490.98 ± (490.24 - 491.72) ms	491.59 ± (490.80 - 492.38) ms	+0.1%	✅⬆️
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	37.26 ± (37.24 - 37.28) MB	37.25 ± (37.23 - 37.27) MB	-0.0%	✅
runtime.dotnet.threads.count	27 ± (27 - 27)	27 ± (27 - 27)	-0.2%	✅

HttpMessageHandler

Metric	Master (Mean ± 95% CI)	Current (Mean ± 95% CI)	Change	Status
.NET Framework 4.8 - Baseline
duration	193.20 ± (193.07 - 194.03) ms	211.06 ± (210.99 - 212.10) ms	+9.2%	❌⬆️
.NET Framework 4.8 - Bailout
duration	197.04 ± (197.25 - 198.32) ms	216.33 ± (215.91 - 216.99) ms	+9.8%	❌⬆️
.NET Framework 4.8 - CallTarget+Inlining+NGEN
duration	1165.14 ± (1172.42 - 1183.27) ms	1244.33 ± (1242.40 - 1250.32) ms	+6.8%	❌⬆️
.NET Core 3.1 - Baseline
process.internal_duration_ms	187.87 ± (187.50 - 188.24) ms	208.07 ± (207.54 - 208.61) ms	+10.8%	✅⬆️
process.time_to_main_ms	81.24 ± (80.95 - 81.54) ms	90.77 ± (90.47 - 91.06) ms	+11.7%	✅⬆️
runtime.dotnet.exceptions.count	3 ± (3 - 3)	3 ± (3 - 3)	+0.0%	✅
runtime.dotnet.mem.committed	16.11 ± (16.07 - 16.14) MB	15.92 ± (15.90 - 15.94) MB	-1.2%	✅
runtime.dotnet.threads.count	20 ± (19 - 20)	20 ± (20 - 20)	+2.0%	✅⬆️
.NET Core 3.1 - Bailout
process.internal_duration_ms	186.53 ± (186.24 - 186.82) ms	207.57 ± (206.99 - 208.15) ms	+11.3%	✅⬆️
process.time_to_main_ms	81.95 ± (81.79 - 82.12) ms	91.98 ± (91.66 - 92.31) ms	+12.2%	✅⬆️
runtime.dotnet.exceptions.count	3 ± (3 - 3)	3 ± (3 - 3)	+0.0%	✅
runtime.dotnet.mem.committed	16.20 ± (16.17 - 16.24) MB	16.03 ± (16.01 - 16.05) MB	-1.1%	✅
runtime.dotnet.threads.count	21 ± (21 - 21)	21 ± (21 - 21)	+0.7%	✅⬆️
.NET Core 3.1 - CallTarget+Inlining+NGEN
process.internal_duration_ms	395.41 ± (394.15 - 396.68) ms	422.59 ± (421.25 - 423.93) ms	+6.9%	✅⬆️
process.time_to_main_ms	525.23 ± (524.34 - 526.13) ms	570.64 ± (569.21 - 572.08) ms	+8.6%	✅⬆️
runtime.dotnet.exceptions.count	3 ± (3 - 3)	3 ± (3 - 3)	+0.0%	✅
runtime.dotnet.mem.committed	59.05 ± (58.94 - 59.17) MB	59.17 ± (59.11 - 59.23) MB	+0.2%	✅⬆️
runtime.dotnet.threads.count	30 ± (30 - 30)	30 ± (30 - 30)	+0.1%	✅⬆️
.NET 6 - Baseline
process.internal_duration_ms	192.21 ± (191.88 - 192.55) ms	215.25 ± (214.70 - 215.79) ms	+12.0%	✅⬆️
process.time_to_main_ms	70.09 ± (69.91 - 70.26) ms	79.94 ± (79.68 - 80.19) ms	+14.1%	✅⬆️
runtime.dotnet.exceptions.count	4 ± (4 - 4)	4 ± (4 - 4)	+0.0%	✅
runtime.dotnet.mem.committed	16.17 ± (16.03 - 16.31) MB	16.22 ± (16.20 - 16.24) MB	+0.3%	✅⬆️
runtime.dotnet.threads.count	19 ± (19 - 19)	20 ± (19 - 20)	+3.4%	✅⬆️
.NET 6 - Bailout
process.internal_duration_ms	192.26 ± (191.93 - 192.59) ms	212.74 ± (212.21 - 213.28) ms	+10.7%	✅⬆️
process.time_to_main_ms	71.05 ± (70.94 - 71.15) ms	80.16 ± (79.90 - 80.43) ms	+12.8%	✅⬆️
runtime.dotnet.exceptions.count	4 ± (4 - 4)	4 ± (4 - 4)	+0.0%	✅
runtime.dotnet.mem.committed	16.22 ± (16.09 - 16.35) MB	16.28 ± (16.25 - 16.31) MB	+0.4%	✅⬆️
runtime.dotnet.threads.count	19 ± (19 - 20)	21 ± (20 - 21)	+6.5%	✅⬆️
.NET 6 - CallTarget+Inlining+NGEN
process.internal_duration_ms	600.34 ± (597.60 - 603.08) ms	597.76 ± (594.58 - 600.93) ms	-0.4%	✅
process.time_to_main_ms	523.80 ± (522.95 - 524.65) ms	577.12 ± (575.40 - 578.83) ms	+10.2%	✅⬆️
runtime.dotnet.exceptions.count	4 ± (4 - 4)	4 ± (4 - 4)	+0.0%	✅
runtime.dotnet.mem.committed	62.08 ± (62.00 - 62.17) MB	61.59 ± (61.48 - 61.69) MB	-0.8%	✅
runtime.dotnet.threads.count	30 ± (30 - 30)	31 ± (31 - 31)	+1.4%	✅⬆️
.NET 8 - Baseline
process.internal_duration_ms	190.16 ± (189.86 - 190.46) ms	211.59 ± (211.12 - 212.07) ms	+11.3%	✅⬆️
process.time_to_main_ms	69.51 ± (69.33 - 69.68) ms	78.33 ± (78.04 - 78.62) ms	+12.7%	✅⬆️
runtime.dotnet.exceptions.count	4 ± (4 - 4)	4 ± (4 - 4)	+0.0%	✅
runtime.dotnet.mem.committed	11.84 ± (11.81 - 11.88) MB	11.62 ± (11.60 - 11.64) MB	-1.9%	✅
runtime.dotnet.threads.count	18 ± (18 - 18)	19 ± (19 - 19)	+4.4%	✅⬆️
.NET 8 - Bailout
process.internal_duration_ms	190.07 ± (189.67 - 190.47) ms	211.92 ± (211.32 - 212.53) ms	+11.5%	✅⬆️
process.time_to_main_ms	70.62 ± (70.48 - 70.76) ms	79.96 ± (79.73 - 80.18) ms	+13.2%	✅⬆️
runtime.dotnet.exceptions.count	4 ± (4 - 4)	4 ± (4 - 4)	+0.0%	✅
runtime.dotnet.mem.committed	11.88 ± (11.85 - 11.90) MB	11.65 ± (11.63 - 11.67) MB	-1.9%	✅
runtime.dotnet.threads.count	19 ± (19 - 19)	20 ± (20 - 20)	+4.6%	✅⬆️
.NET 8 - CallTarget+Inlining+NGEN
process.internal_duration_ms	519.62 ± (517.02 - 522.21) ms	562.45 ± (554.52 - 570.38) ms	+8.2%	✅⬆️
process.time_to_main_ms	480.70 ± (480.02 - 481.38) ms	532.48 ± (531.32 - 533.64) ms	+10.8%	✅⬆️
runtime.dotnet.exceptions.count	4 ± (4 - 4)	4 ± (4 - 4)	+0.0%	✅
runtime.dotnet.mem.committed	50.96 ± (50.93 - 50.99) MB	51.29 ± (51.20 - 51.39) MB	+0.7%	✅⬆️
runtime.dotnet.threads.count	30 ± (30 - 30)	30 ± (30 - 30)	+0.1%	✅⬆️

Comparison explanation

Execution-time benchmarks measure the whole time it takes to execute a program, and are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are highlighted in **red**. The following thresholds were used for comparing the execution times:

• Welch test with statistical test for significance of 5%
• Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

Duration charts

FakeDbCommand (.NET Framework 4.8)

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8371) - mean (79ms)  : 72, 86
    master - mean (72ms)  : 70, 74

    section Bailout
    This PR (8371) - mean (96ms)  : crit, 91, 100
    master - mean (76ms)  : 74, 78

    section CallTarget+Inlining+NGEN
    This PR (8371) - mean (1,104ms)  : 998, 1211
    master - mean (1,075ms)  : 1027, 1124

Loading

FakeDbCommand (.NET Core 3.1)

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8371) - mean (113ms)  : 109, 117
    master - mean (113ms)  : 110, 116

    section Bailout
    This PR (8371) - mean (114ms)  : 111, 117
    master - mean (114ms)  : 112, 116

    section CallTarget+Inlining+NGEN
    This PR (8371) - mean (792ms)  : 768, 816
    master - mean (792ms)  : 771, 814

Loading

FakeDbCommand (.NET 6)

gantt
    title Execution time (ms) FakeDbCommand (.NET 6)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8371) - mean (100ms)  : 96, 103
    master - mean (99ms)  : 95, 103

    section Bailout
    This PR (8371) - mean (100ms)  : 98, 103
    master - mean (100ms)  : 98, 103

    section CallTarget+Inlining+NGEN
    This PR (8371) - mean (948ms)  : 909, 986
    master - mean (946ms)  : 910, 982

Loading

FakeDbCommand (.NET 8)

gantt
    title Execution time (ms) FakeDbCommand (.NET 8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8371) - mean (99ms)  : 96, 101
    master - mean (98ms)  : 95, 101

    section Bailout
    This PR (8371) - mean (100ms)  : 97, 102
    master - mean (100ms)  : 98, 102

    section CallTarget+Inlining+NGEN
    This PR (8371) - mean (830ms)  : 796, 865
    master - mean (829ms)  : 793, 865

Loading

HttpMessageHandler (.NET Framework 4.8)

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8371) - mean (212ms)  : 205, 218
    master - mean (194ms)  : 189, 198

    section Bailout
    This PR (8371) - mean (216ms)  : crit, 211, 222
    master - mean (198ms)  : 192, 203

    section CallTarget+Inlining+NGEN
    This PR (8371) - mean (1,246ms)  : crit, 1186, 1307
    master - mean (1,178ms)  : 1095, 1261

Loading

HttpMessageHandler (.NET Core 3.1)

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8371) - mean (308ms)  : 299, 317
    master - mean (277ms)  : 272, 283

    section Bailout
    This PR (8371) - mean (309ms)  : crit, 300, 319
    master - mean (277ms)  : 273, 280

    section CallTarget+Inlining+NGEN
    This PR (8371) - mean (1,036ms)  : crit, 1003, 1069
    master - mean (953ms)  : 929, 977

Loading

HttpMessageHandler (.NET 6)

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8371) - mean (305ms)  : 295, 315
    master - mean (271ms)  : 266, 275

    section Bailout
    This PR (8371) - mean (302ms)  : crit, 297, 308
    master - mean (271ms)  : 268, 275

    section CallTarget+Inlining+NGEN
    This PR (8371) - mean (1,209ms)  : 1169, 1250
    master - mean (1,155ms)  : 1111, 1199

Loading

HttpMessageHandler (.NET 8)

gantt
    title Execution time (ms) HttpMessageHandler (.NET 8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8371) - mean (300ms)  : 293, 308
    master - mean (270ms)  : 265, 274

    section Bailout
    This PR (8371) - mean (303ms)  : crit, 295, 311
    master - mean (270ms)  : 265, 275

    section CallTarget+Inlining+NGEN
    This PR (8371) - mean (1,130ms)  : crit, 1019, 1240
    master - mean (1,034ms)  : 988, 1080

Loading

[andrewlock]

This change seems weird 🤔

[bouwkast]

Isn't it "Hangfire.Core" and not "Hangfire" I couldn't get it to honor the cooldown as the NuGet name was wrong

[andrewlock]

OK, so this could be an interesting mis-match.... The assembly we instrument is Hangfire.Core:

dd-trace-dotnet/tracer/src/Datadog.Trace/ClrProfiler/AutoInstrumentation/Hangfire/JobFilterCollectionCtorIntegration.cs

Line 20 in 5ff69b3

AssemblyName = "Hangfire.Core",

but the package we use in the sample is Hangfire:

dd-trace-dotnet/tracer/test/test-applications/integrations/Samples.Hangfire/Samples.Hangfire.csproj

Line 10 in 5ff69b3

<PackageReference Include="Hangfire" Version="$(ApiVersion)" />

And it's the Hangfire package version which we need here, because it's the one we reference in the sample 🤔 And those will likely always match... but they don't have to...

[bouwkast]

I'll opt to follow up on this in another PR I think to try and rectify it

[andrewlock]

That's weird 😅

[andrewlock]

I'm just wondering, would it be easiest to just filter at the source when we load from NuGet? 🤔 and then everything else filters through. I guess you're trying to make it stable, but if we just forced 48 hours always then it would always be stable 🤷‍♂️ Not worried either way, just a thought 😄

[andrewlock]

Wait, what are these even for again? 😅 do we need them in the manual instrumentation ones too?

[bouwkast]

So the issue is that the ones I had to update here were all getting around as the "name"/"key" for them all comes from various locations and there are several disconnects, probably could do with a refactoring somewhere to make sure the names and keys all are consistent amongst all files somehow

If that answers it

[andrewlock]

Honestly I'm completely confused, because I can't even remember what IntegrationGroups is for 😅 But maybe it's for the Hangfire/Hangfire.Core issue mentioned above?

[bouwkast]

I think it is for the supported_versions and the Honeypot generations

I'll probably take a look at it in a follow up PR for the Hangfire/Hangfire.Core naming and see if there is anything we can do to simplify

[lucaspimentel]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7921ed41c7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Scope cooldown baseline by package range

Building baseline as a single max tested version per package lets newer versions in lower compatibility ranges bypass the cooldown check. PackageVersionsGeneratorDefinitions.json contains multiple entries for the same package with different ranges (for example, GraphQL has 4.1.0-6.0.0 and 7.0.0-9.0.0), so a high tested baseline from one range (e.g., 8.x) makes all 5.x versions satisfy version <= baseline in ApplyCooldown, even when they were published within the cooldown window. That defeats the remediation goal for split-range packages; baseline needs to be keyed by the specific entry/range (or assembly+package) instead of package name alone.

Useful? React with 👍 / 👎.

[zacharycmontoya]

I don't understand why the version check is v <= baselineVersion. Can you explain?

[bouwkast]

Sorry it's a bit convoluted at the moment as the data is being pulled from supported_versions.json and PackageVersionsGeneratorDefinitions.json. It also doesn't work 100% perfectly at the moment, but I think that is fine as it will only break for some versions in that that it on CI we'd need to potentially re-run the job that runs once a week if a non-latest major version is released, am planning on fixing it better.

I'll just take the GraphQL example (deleting some bits from the JSON):

supported_versions.json

  {
    "integrationName": "GraphQL",
    "assemblyName": "GraphQL",
    "minAssemblyVersionInclusive": "2.3.0",
    "maxAssemblyVersionInclusive": "8.65535.65535",
    "packages": [
      {
        "name": "GraphQL",
        "maxVersionTestedInclusive": "8.8.4"
      }
    ]
  },

PackageVersionsGeneratorDefinitions.json

 {
    "IntegrationName": "GraphQL",
    "SampleProjectName": "Samples.GraphQL4",
    "NugetPackageSearchName": "GraphQL",
    "MinVersion": "4.1.0",
    "MaxVersionExclusive": "6.0.0",
  },
  {
    "IntegrationName": "GraphQL7",
    "SampleProjectName": "Samples.GraphQL7",
    "NugetPackageSearchName": "GraphQL",
    "MinVersion": "7.0.0",
    "MaxVersionExclusive": "9.0.0",
  },

We first read supported_versions.json which grabs our latest version tested for GraphQL => [8.8.4] (ideally we go and update supported_versions.json to be similar / equal to PackageVersionsGeneratorDefinitions.json)

We then process PackageVersionsGeneratorDefinitions.json and have two entries for GraphQL: 4.1.0 to 6.0.0 and 7.0.0 to 9.0.0

For the first range (4 to 6) we grab the "baseline" and we get that 8.8.4, all versions within 4 to 6 will be less than that so they will always have the cooldown filtering logic applied to their versions, technically a bug but I figure it isn't terribly important at the moment, but something to follow up on. (example is if some 6.* version just came out this will revert it on the CI job if it is within the cooldown period)

For the second range (7.0.0 to 9.0.0) we grab the "baseline" and get 8.8.4, any version on NuGet above 8.8.4 will have the filtering logic applied to it.

Also, the baseline allows for us to update manually to a latest version locally, push it up and have the CI step ignore it. But we can't for that intermediate version at the moment (we could generate the files locally and then update the automated PR)

[zacharycmontoya]

I must admit I'm still a little confused but overall the approach sounds thought-out 😅

[zacharycmontoya]

LGTM, though the baseline and cooldown comparisons are still a little confusing to me 😅