Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ASM] IAST events in span's meta_struct #5803

Merged
merged 9 commits into from
Dec 3, 2024
Merged

[ASM] IAST events in span's meta_struct #5803

merged 9 commits into from
Dec 3, 2024

Conversation

e-n-0
Copy link
Member

@e-n-0 e-n-0 commented Jul 24, 2024 •
edited
Loading

Summary of changes

Move iast data from the _dd.iast.json tag to a messagepack-encoded data serialized into meta struct.
RFC Linked to that change.

  • Added a new telemetry tag _dd.iast.metastruct.size.exceeded to reflect the exceeded size for meta struct when a truncation occured

Reason for change

Better performance.

Implementation details

  • Check for meta struct compatibility with the connected agent
  • Fallback to json serialization if not supported

Test coverage

  • By default, now the mocked trace agent is set to support meta struct (cf [ASM] Appsec events in meta struct #5779), so all iast events will be set in meta struct
  • To keep all existing snapshots unchanged, meta struct iast events are set back as json on the same tag into snapshots
  • Add a new option in the iast test: "forceMetaStruct". When enabled, the tag _dd.iast.json.metastruct.test is set when iast data from metastruct is serialized back to json. This is to be sure that meta struct is used in the test.
  • Updated all unit tests of Vulnerability Batch to also use the meta struct serialization
  • New test for the telemetry tag _dd.iast.metastruct.size.exceeded

Other details

@e-n-0 e-n-0 self-assigned this Jul 24, 2024
@e-n-0 e-n-0 force-pushed the flavien/asm/iast/events_metastruct branch from 2cfe81a to 602631f Compare July 24, 2024 15:36
@datadog-ddstaging
Copy link

datadog-ddstaging bot commented Jul 24, 2024 •
edited
Loading

Datadog Report

Branch report: flavien/asm/iast/events_metastruct
Commit report: e95f002
Test service: dd-trace-dotnet

✅ 0 Failed, 563044 Passed, 3965 Skipped, 34h 23m 39.39s Total Time

@andrewlock
Copy link
Member

andrewlock commented Jul 24, 2024 •
edited
Loading

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5803) - mean (69ms)  : 66, 71
     .   : milestone, 69,
    master - mean (69ms)  : 66, 72
     .   : milestone, 69,

    section CallTarget+Inlining+NGEN
    This PR (5803) - mean (979ms)  : 954, 1004
     .   : milestone, 979,
    master - mean (980ms)  : 954, 1006
     .   : milestone, 980,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5803) - mean (108ms)  : 106, 110
     .   : milestone, 108,
    master - mean (108ms)  : 106, 110
     .   : milestone, 108,

    section CallTarget+Inlining+NGEN
    This PR (5803) - mean (683ms)  : 669, 697
     .   : milestone, 683,
    master - mean (680ms)  : 667, 694
     .   : milestone, 680,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5803) - mean (92ms)  : 89, 94
     .   : milestone, 92,
    master - mean (92ms)  : 90, 94
     .   : milestone, 92,

    section CallTarget+Inlining+NGEN
    This PR (5803) - mean (626ms)  : 610, 641
     .   : milestone, 626,
    master - mean (634ms)  : 618, 650
     .   : milestone, 634,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5803) - mean (190ms)  : 186, 195
     .   : milestone, 190,
    master - mean (190ms)  : 186, 195
     .   : milestone, 190,

    section CallTarget+Inlining+NGEN
    This PR (5803) - mean (1,088ms)  : 1055, 1121
     .   : milestone, 1088,
    master - mean (1,096ms)  : 1054, 1137
     .   : milestone, 1096,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5803) - mean (276ms)  : 272, 280
     .   : milestone, 276,
    master - mean (276ms)  : 272, 280
     .   : milestone, 276,

    section CallTarget+Inlining+NGEN
    This PR (5803) - mean (869ms)  : 834, 903
     .   : milestone, 869,
    master - mean (869ms)  : 842, 895
     .   : milestone, 869,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5803) - mean (266ms)  : 262, 269
     .   : milestone, 266,
    master - mean (266ms)  : 262, 270
     .   : milestone, 266,

    section CallTarget+Inlining+NGEN
    This PR (5803) - mean (845ms)  : 813, 876
     .   : milestone, 845,
    master - mean (855ms)  : 817, 892
     .   : milestone, 855,
Loading

@andrewlock
Copy link
Member

andrewlock commented Jul 24, 2024 •
edited
Loading

Benchmarks Report for appsec 🐌

Benchmarks for #5803 compared to master:

  • 2 benchmarks are slower, with geometric mean 1.124
  • All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.Asm.AppSecBodyBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #5803

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody‑net6.0 1.132 194,764.80 220,518.45
Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleMoreComplexBody‑net6.0 1.115 201,901.15 225,147.04

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody net6.0 195μs 90.6ns 339ns 2.52 0 0 176.97 KB
master AllCycleSimpleBody netcoreapp3.1 311μs 200ns 747ns 2.51 0 0 184.41 KB
master AllCycleSimpleBody net472 263μs 442ns 1.71μs 33.3 2.51 0 210.2 KB
master AllCycleMoreComplexBody net6.0 202μs 159ns 616ns 2.52 0 0 180.47 KB
master AllCycleMoreComplexBody netcoreapp3.1 305μs 170ns 660ns 2.47 0 0 187.82 KB
master AllCycleMoreComplexBody net472 269μs 356ns 1.38μs 33.9 2.47 0 213.72 KB
master ObjectExtractorSimpleBody net6.0 141ns 0.101ns 0.391ns 0.00391 0 0 280 B
master ObjectExtractorSimpleBody netcoreapp3.1 195ns 0.147ns 0.55ns 0.00372 0 0 272 B
master ObjectExtractorSimpleBody net472 211ns 0.148ns 0.555ns 0.0445 0 0 281 B
master ObjectExtractorMoreComplexBody net6.0 3.03μs 1.35ns 5.05ns 0.0536 0 0 3.78 KB
master ObjectExtractorMoreComplexBody netcoreapp3.1 3.97μs 2.53ns 9.79ns 0.0498 0 0 3.69 KB
master ObjectExtractorMoreComplexBody net472 4.51μs 5.49ns 20.6ns 0.601 0.0067 0 3.8 KB
#5803 AllCycleSimpleBody net6.0 220μs 72.4ns 271ns 2.43 0 0 176.83 KB
#5803 AllCycleSimpleBody netcoreapp3.1 322μs 141ns 509ns 2.43 0 0 184.27 KB
#5803 AllCycleSimpleBody net472 286μs 460ns 1.78μs 33.4 2.55 0 210.04 KB
#5803 AllCycleMoreComplexBody net6.0 225μs 45.5ns 164ns 2.47 0 0 180.33 KB
#5803 AllCycleMoreComplexBody netcoreapp3.1 323μs 172ns 644ns 2.46 0 0 187.68 KB
#5803 AllCycleMoreComplexBody net472 292μs 307ns 1.19μs 33.9 2.62 0 213.56 KB
#5803 ObjectExtractorSimpleBody net6.0 154ns 0.162ns 0.584ns 0.00396 0 0 280 B
#5803 ObjectExtractorSimpleBody netcoreapp3.1 212ns 0.12ns 0.447ns 0.00375 0 0 272 B
#5803 ObjectExtractorSimpleBody net472 213ns 0.117ns 0.453ns 0.0446 0 0 281 B
#5803 ObjectExtractorMoreComplexBody net6.0 3.08μs 1.6ns 5.98ns 0.0531 0 0 3.78 KB
#5803 ObjectExtractorMoreComplexBody netcoreapp3.1 4.04μs 1.77ns 6.61ns 0.0486 0 0 3.69 KB
#5803 ObjectExtractorMoreComplexBody net472 4.46μs 3.22ns 12.5ns 0.603 0.00665 0 3.8 KB
Benchmarks.Trace.Asm.AppSecEncoderBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EncodeArgs net6.0 36.3μs 18.6ns 69.7ns 0.454 0 0 32.4 KB
master EncodeArgs netcoreapp3.1 54.1μs 22.2ns 85.8ns 0.431 0 0 32.4 KB
master EncodeArgs net472 67μs 69.6ns 270ns 5.13 0.0662 0 32.5 KB
master EncodeLegacyArgs net6.0 71.1μs 27.2ns 105ns 0 0 0 2.14 KB
master EncodeLegacyArgs netcoreapp3.1 103μs 105ns 406ns 0 0 0 2.15 KB
master EncodeLegacyArgs net472 155μs 60.7ns 235ns 0.311 0 0 2.15 KB
#5803 EncodeArgs net6.0 36.8μs 14.7ns 55ns 0.441 0 0 32.4 KB
#5803 EncodeArgs netcoreapp3.1 54μs 21.5ns 83.2ns 0.431 0 0 32.4 KB
#5803 EncodeArgs net472 68.4μs 27.5ns 106ns 5.16 0.0684 0 32.5 KB
#5803 EncodeLegacyArgs net6.0 70.6μs 43.7ns 151ns 0 0 0 2.14 KB
#5803 EncodeLegacyArgs netcoreapp3.1 107μs 110ns 426ns 0 0 0 2.14 KB
#5803 EncodeLegacyArgs net472 157μs 67.3ns 261ns 0.32 0 0 2.15 KB
Benchmarks.Trace.Asm.AppSecWafBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunWafRealisticBenchmark net6.0 184μs 144ns 518ns 0 0 0 2.44 KB
master RunWafRealisticBenchmark netcoreapp3.1 196μs 167ns 645ns 0 0 0 2.39 KB
master RunWafRealisticBenchmark net472 208μs 50.9ns 197ns 0.314 0 0 2.46 KB
master RunWafRealisticBenchmarkWithAttack net6.0 124μs 160ns 621ns 0 0 0 1.47 KB
master RunWafRealisticBenchmarkWithAttack netcoreapp3.1 131μs 69.4ns 260ns 0 0 0 1.46 KB
master RunWafRealisticBenchmarkWithAttack net472 141μs 67.2ns 260ns 0.212 0 0 1.49 KB
#5803 RunWafRealisticBenchmark net6.0 183μs 107ns 414ns 0 0 0 2.44 KB
#5803 RunWafRealisticBenchmark netcoreapp3.1 193μs 190ns 736ns 0 0 0 2.39 KB
#5803 RunWafRealisticBenchmark net472 211μs 60.2ns 217ns 0.317 0 0 2.46 KB
#5803 RunWafRealisticBenchmarkWithAttack net6.0 123μs 123ns 475ns 0 0 0 1.47 KB
#5803 RunWafRealisticBenchmarkWithAttack netcoreapp3.1 129μs 98ns 380ns 0 0 0 1.46 KB
#5803 RunWafRealisticBenchmarkWithAttack net472 140μs 43.8ns 158ns 0.209 0 0 1.49 KB
Benchmarks.Trace.Iast.StringAspectsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StringConcatBenchmark net6.0 66.1μs 1.33μs 13.1μs 0 0 0 43.44 KB
master StringConcatBenchmark netcoreapp3.1 62.3μs 858ns 8.5μs 0 0 0 42.64 KB
master StringConcatBenchmark net472 38.1μs 121ns 420ns 0 0 0 59.36 KB
master StringConcatAspectBenchmark net6.0 299μs 4.66μs 44.2μs 0 0 0 255.03 KB
master StringConcatAspectBenchmark netcoreapp3.1 337μs 2.21μs 21.2μs 0 0 0 253.61 KB
master StringConcatAspectBenchmark net472 291μs 6.15μs 59.6μs 0 0 0 278.53 KB
#5803 StringConcatBenchmark net6.0 61.7μs 766ns 7.66μs 0 0 0 43.44 KB
#5803 StringConcatBenchmark netcoreapp3.1 61.6μs 767ns 7.52μs 0 0 0 42.64 KB
#5803 StringConcatBenchmark net472 38.2μs 84.4ns 304ns 0 0 0 59.07 KB
#5803 StringConcatAspectBenchmark net6.0 309μs 1.2μs 7.79μs 0 0 0 254.16 KB
#5803 StringConcatAspectBenchmark netcoreapp3.1 344μs 1.57μs 9.14μs 0 0 0 254.1 KB
#5803 StringConcatAspectBenchmark net472 292μs 6.31μs 62.1μs 0 0 0 278.53 KB

@andrewlock
Copy link
Member

andrewlock commented Jul 24, 2024 •
edited
Loading

Benchmarks Report for tracer 🐌

Benchmarks for #5803 compared to master:

  • 2 benchmarks are faster, with geometric mean 1.148
  • 1 benchmarks are slower, with geometric mean 1.163
  • 2 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartStopWithChild net6.0 7.99μs 44.7ns 303ns 0.0159 0.00794 0 5.61 KB
master StartStopWithChild netcoreapp3.1 10.2μs 55.2ns 322ns 0.0197 0.00984 0 5.81 KB
master StartStopWithChild net472 16.1μs 42.9ns 161ns 1.06 0.32 0.104 6.21 KB
#5803 StartStopWithChild net6.0 7.98μs 45.6ns 332ns 0.0158 0.00789 0 5.61 KB
#5803 StartStopWithChild netcoreapp3.1 9.94μs 55.4ns 350ns 0.0148 0.00493 0 5.8 KB
#5803 StartStopWithChild net472 16.3μs 41.4ns 143ns 1.04 0.324 0.113 6.2 KB
Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 502μs 251ns 972ns 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 663μs 376ns 1.41μs 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces net472 855μs 460ns 1.78μs 0.425 0 0 3.3 KB
#5803 WriteAndFlushEnrichedTraces net6.0 489μs 302ns 1.13μs 0 0 0 2.7 KB
#5803 WriteAndFlushEnrichedTraces netcoreapp3.1 668μs 694ns 2.69μs 0 0 0 2.7 KB
#5803 WriteAndFlushEnrichedTraces net472 843μs 788ns 3.05μs 0.419 0 0 3.3 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 150μs 932ns 9.27μs 0.21 0 0 14.47 KB
master SendRequest netcoreapp3.1 170μs 1μs 9.44μs 0.17 0 0 17.27 KB
master SendRequest net472 0.00341ns 0.000616ns 0.00239ns 0 0 0 0 b
#5803 SendRequest net6.0 152μs 886ns 7.82μs 0.15 0 0 14.47 KB
#5803 SendRequest netcoreapp3.1 158μs 861ns 4.79μs 0.151 0 0 17.27 KB
#5803 SendRequest net472 0.00139ns 0.000382ns 0.00143ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #5803

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑netcoreapp3.1 41.73 KB 42.02 KB 295 B 0.71%
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑net6.0 41.42 KB 41.71 KB 287 B 0.69%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 577μs 3.28μs 25μs 0.326 0 0 41.42 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 661μs 2.37μs 8.21μs 0.322 0 0 41.73 KB
master WriteAndFlushEnrichedTraces net472 866μs 3.97μs 14.3μs 8.33 2.5 0.417 53.3 KB
#5803 WriteAndFlushEnrichedTraces net6.0 572μs 2.78μs 14.7μs 0.556 0 0 41.71 KB
#5803 WriteAndFlushEnrichedTraces netcoreapp3.1 679μs 3.43μs 16.8μs 0.345 0 0 42.02 KB
#5803 WriteAndFlushEnrichedTraces net472 870μs 3.22μs 12.5μs 8.39 2.52 0.419 53.34 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 1.31μs 0.946ns 3.67ns 0.0144 0 0 1.02 KB
master ExecuteNonQuery netcoreapp3.1 1.75μs 2.03ns 7.59ns 0.0133 0 0 1.02 KB
master ExecuteNonQuery net472 2.15μs 2.09ns 7.84ns 0.156 0.00108 0 987 B
#5803 ExecuteNonQuery net6.0 1.3μs 1.45ns 5.42ns 0.0142 0 0 1.02 KB
#5803 ExecuteNonQuery netcoreapp3.1 1.77μs 1.15ns 4.45ns 0.0134 0 0 1.02 KB
#5803 ExecuteNonQuery net472 2.12μs 3.65ns 14.1ns 0.156 0.00106 0 987 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.26μs 0.63ns 2.36ns 0.0137 0 0 976 B
master CallElasticsearch netcoreapp3.1 1.51μs 1.48ns 5.72ns 0.0127 0 0 976 B
master CallElasticsearch net472 2.61μs 1.22ns 4.71ns 0.158 0 0 995 B
master CallElasticsearchAsync net6.0 1.29μs 0.568ns 2.2ns 0.0132 0 0 952 B
master CallElasticsearchAsync netcoreapp3.1 1.67μs 0.912ns 3.29ns 0.0134 0 0 1.02 KB
master CallElasticsearchAsync net472 2.67μs 1.05ns 4.08ns 0.166 0 0 1.05 KB
#5803 CallElasticsearch net6.0 1.27μs 0.618ns 2.39ns 0.0134 0 0 976 B
#5803 CallElasticsearch netcoreapp3.1 1.49μs 0.854ns 3.31ns 0.0135 0 0 976 B
#5803 CallElasticsearch net472 2.61μs 1.15ns 4.32ns 0.157 0 0 995 B
#5803 CallElasticsearchAsync net6.0 1.26μs 0.625ns 2.34ns 0.0133 0 0 952 B
#5803 CallElasticsearchAsync netcoreapp3.1 1.68μs 0.709ns 2.65ns 0.0143 0 0 1.02 KB
#5803 CallElasticsearchAsync net472 2.55μs 1.01ns 3.77ns 0.167 0 0 1.05 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.23μs 1.11ns 4.28ns 0.0134 0 0 952 B
master ExecuteAsync netcoreapp3.1 1.63μs 2.54ns 9.84ns 0.013 0 0 952 B
master ExecuteAsync net472 1.81μs 0.599ns 2.24ns 0.145 0 0 915 B
#5803 ExecuteAsync net6.0 1.28μs 0.979ns 3.66ns 0.0135 0 0 952 B
#5803 ExecuteAsync netcoreapp3.1 1.61μs 0.444ns 1.6ns 0.0129 0 0 952 B
#5803 ExecuteAsync net472 1.77μs 0.802ns 3.11ns 0.145 0 0 915 B
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 4.43μs 1.08ns 4.03ns 0.0333 0 0 2.31 KB
master SendAsync netcoreapp3.1 5.35μs 2.34ns 8.74ns 0.0374 0 0 2.85 KB
master SendAsync net472 7.31μs 1.92ns 7.42ns 0.495 0 0 3.12 KB
#5803 SendAsync net6.0 4.35μs 1.06ns 4.11ns 0.0327 0 0 2.31 KB
#5803 SendAsync netcoreapp3.1 5.38μs 2.74ns 10.6ns 0.0375 0 0 2.85 KB
#5803 SendAsync net472 7.26μs 3.15ns 12.2ns 0.493 0 0 3.12 KB
Benchmarks.Trace.ILoggerBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #5803

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.ILoggerBenchmark.EnrichedLog‑net6.0 1.163 1,458.47 1,696.88

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 1.46μs 0.806ns 3.02ns 0.0227 0 0 1.64 KB
master EnrichedLog netcoreapp3.1 2.38μs 1.22ns 4.55ns 0.0225 0 0 1.64 KB
master EnrichedLog net472 2.72μs 1.14ns 4.12ns 0.249 0 0 1.57 KB
#5803 EnrichedLog net6.0 1.7μs 0.823ns 2.97ns 0.0231 0 0 1.64 KB
#5803 EnrichedLog netcoreapp3.1 2.18μs 1.55ns 5.78ns 0.0229 0 0 1.64 KB
#5803 EnrichedLog net472 2.77μs 1.73ns 6.47ns 0.249 0 0 1.57 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 120μs 97.1ns 376ns 0.0597 0 0 4.28 KB
master EnrichedLog netcoreapp3.1 125μs 198ns 767ns 0.0628 0 0 4.28 KB
master EnrichedLog net472 152μs 106ns 410ns 0.681 0.227 0 4.46 KB
#5803 EnrichedLog net6.0 120μs 148ns 574ns 0.0604 0 0 4.28 KB
#5803 EnrichedLog netcoreapp3.1 123μs 106ns 410ns 0 0 0 4.28 KB
#5803 EnrichedLog net472 151μs 149ns 576ns 0.681 0.227 0 4.46 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.96μs 1.28ns 4.97ns 0.0312 0 0 2.2 KB
master EnrichedLog netcoreapp3.1 4.21μs 1.96ns 7.35ns 0.0293 0 0 2.2 KB
master EnrichedLog net472 4.79μs 1.57ns 6.08ns 0.32 0 0 2.02 KB
#5803 EnrichedLog net6.0 3μs 0.887ns 3.43ns 0.03 0 0 2.2 KB
#5803 EnrichedLog netcoreapp3.1 4.3μs 1.53ns 5.74ns 0.0302 0 0 2.2 KB
#5803 EnrichedLog net472 4.87μs 1.97ns 7.1ns 0.319 0 0 2.02 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 1.43μs 3.43ns 13.3ns 0.0161 0 0 1.14 KB
master SendReceive netcoreapp3.1 1.83μs 0.747ns 2.8ns 0.0156 0 0 1.14 KB
master SendReceive net472 2.16μs 1.17ns 4.39ns 0.184 0 0 1.16 KB
#5803 SendReceive net6.0 1.33μs 0.651ns 2.52ns 0.0165 0 0 1.14 KB
#5803 SendReceive netcoreapp3.1 1.87μs 0.789ns 2.95ns 0.0158 0 0 1.14 KB
#5803 SendReceive net472 2μs 1.11ns 3.99ns 0.184 0 0 1.16 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.79μs 0.842ns 3.26ns 0.0224 0 0 1.6 KB
master EnrichedLog netcoreapp3.1 3.77μs 1.7ns 6.15ns 0.0209 0 0 1.65 KB
master EnrichedLog net472 4.39μs 2.12ns 8.21ns 0.323 0 0 2.04 KB
#5803 EnrichedLog net6.0 2.87μs 0.697ns 2.51ns 0.0229 0 0 1.6 KB
#5803 EnrichedLog netcoreapp3.1 4.03μs 2.79ns 10.8ns 0.0224 0 0 1.65 KB
#5803 EnrichedLog net472 4.41μs 1.13ns 4.08ns 0.322 0 0 2.04 KB
Benchmarks.Trace.SpanBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #5803

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net6.0 1.154 623.65 540.27
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑netcoreapp3.1 1.142 744.81 652.10

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 388ns 0.689ns 2.67ns 0.00801 0 0 576 B
master StartFinishSpan netcoreapp3.1 580ns 0.954ns 3.69ns 0.00797 0 0 576 B
master StartFinishSpan net472 574ns 0.736ns 2.85ns 0.0916 0 0 578 B
master StartFinishScope net6.0 624ns 1.31ns 5.07ns 0.00966 0 0 696 B
master StartFinishScope netcoreapp3.1 745ns 1.73ns 6.49ns 0.00931 0 0 696 B
master StartFinishScope net472 881ns 1.65ns 6.41ns 0.104 0 0 658 B
#5803 StartFinishSpan net6.0 402ns 0.479ns 1.85ns 0.00817 0 0 576 B
#5803 StartFinishSpan netcoreapp3.1 559ns 0.919ns 3.56ns 0.00784 0 0 576 B
#5803 StartFinishSpan net472 643ns 1.66ns 6.42ns 0.0918 0 0 578 B
#5803 StartFinishScope net6.0 541ns 0.727ns 2.81ns 0.00969 0 0 696 B
#5803 StartFinishScope netcoreapp3.1 652ns 1.22ns 4.72ns 0.00951 0 0 696 B
#5803 StartFinishScope net472 874ns 1.82ns 7.04ns 0.104 0 0 658 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 700ns 1.87ns 7.23ns 0.00984 0 0 696 B
master RunOnMethodBegin netcoreapp3.1 926ns 1.59ns 6.14ns 0.00912 0 0 696 B
master RunOnMethodBegin net472 1.17μs 2.76ns 10.7ns 0.104 0 0 658 B
#5803 RunOnMethodBegin net6.0 664ns 0.524ns 2.03ns 0.00994 0 0 696 B
#5803 RunOnMethodBegin netcoreapp3.1 904ns 1.9ns 7.36ns 0.00902 0 0 696 B
#5803 RunOnMethodBegin net472 1.14μs 2.11ns 8.19ns 0.105 0 0 658 B

@e-n-0 e-n-0 force-pushed the flavien/asm/iast/events_metastruct branch 3 times, most recently from 1bd423b to 3fd5f33 Compare July 25, 2024 14:21
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 26, 2024 •
edited
Loading

Snapshots difference summary

The following differences have been observed in committed snapshots. It is meant to help the reviewer.
The diff is simplistic, so please check some files anyway while we improve it.

164 occurrences of :

+    },
+    MetaStruct: {
+      iast:

1 occurrences of :

+    {
+      "type": "COMMAND_INJECTION",
+      "evidence": {
+        "value": "MAX SIZE EXCEEDED"
+      },
+      "hash": -430498668,
+      "location": {
+        "spanId": XXX,
+        "path": "Samples.Security.AspNetCore5.Controllers.IastController",
+        "method": "ExecuteCommandInternal"
+      }
+    },
+    {
+      "type": "COMMAND_INJECTION",
+      "evidence": {
+        "value": "MAX SIZE EXCEEDED"
+      },
+      "hash": -430498668,
+      "location": {
+        "spanId": XXX,
+        "path": "Samples.Security.AspNetCore5.Controllers.IastController",
+        "method": "ExecuteCommandInternal"
+      }
+    },
+    {
+      "type": "COMMAND_INJECTION",
+      "evidence": {
+        "value": "MAX SIZE EXCEEDED"
+      },
+      "hash": -430498668,
+      "location": {
+        "spanId": XXX,
+        "path": "Samples.Security.AspNetCore5.Controllers.IastController",
+        "method": "ExecuteCommandInternal"
+      }
+    },
+    {
+      "type": "COMMAND_INJECTION",
+      "evidence": {
+        "value": "MAX SIZE EXCEEDED"
+      },
+      "hash": -430498668,
+      "location": {
+        "spanId": XXX,
+        "path": "Samples.Security.AspNetCore5.Controllers.IastController",
+        "method": "ExecuteCommandInternal"
+      }
+    },
+    {
+      "type": "COMMAND_INJECTION",
+      "evidence": {
+        "value": "MAX SIZE EXCEEDED"
+      },
+      "hash": -430498668,
+      "location": {
+        "spanId": XXX,
+        "path": "Samples.Security.AspNetCore5.Controllers.IastController",
+        "method": "ExecuteCommandInternal"
+      }
+    },

1 occurrences of :

-      _dd.iast.telemetry.executed.propagation: 360.0,
-      _dd.iast.telemetry.executed.sink.command_injection: 30.0,
+      _dd.iast.telemetry.executed.propagation: 420.0,
+      _dd.iast.telemetry.executed.sink.command_injection: 35.0,

26 occurrences of :

+      iast: ,

@e-n-0 e-n-0 force-pushed the flavien/asm/iast/events_metastruct branch 4 times, most recently from 9d0ad2e to 587c2a7 Compare July 26, 2024 15:12
@e-n-0 e-n-0 marked this pull request as ready for review July 26, 2024 15:14
@e-n-0 e-n-0 requested review from a team as code owners July 26, 2024 15:14
@andrewlock
Copy link
Member

andrewlock commented Jul 26, 2024 •
edited
Loading

Throughput/Crank Report ⚡

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5803) (11.260M)   : 0, 11260175
    master (11.361M)   : 0, 11361081
    benchmarks/2.9.0 (11.033M)   : 0, 11032866

    section Automatic
    This PR (5803) (7.382M)   : 0, 7382219
    master (7.386M)   : 0, 7386114
    benchmarks/2.9.0 (7.786M)   : 0, 7785853

    section Trace stats
    master (7.662M)   : 0, 7662233

    section Manual
    master (11.190M)   : 0, 11189983

    section Manual + Automatic
    This PR (5803) (6.816M)   : 0, 6815576
    master (6.815M)   : 0, 6814740

    section DD_TRACE_ENABLED=0
    master (10.320M)   : 0, 10319505
Loading 
gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5803) (9.575M)   : 0, 9574996
    master (9.527M)   : 0, 9527423
    benchmarks/2.9.0 (9.495M)   : 0, 9494821

    section Automatic
    This PR (5803) (6.418M)   : 0, 6417517
    master (6.264M)   : 0, 6263949

    section Trace stats
    master (6.510M)   : 0, 6509886

    section Manual
    master (9.336M)   : 0, 9335765

    section Manual + Automatic
    This PR (5803) (5.940M)   : 0, 5939608
    master (5.881M)   : 0, 5880843

    section DD_TRACE_ENABLED=0
    master (8.790M)   : 0, 8790191
Loading 
gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5803) (9.874M)   : 0, 9874471
    master (10.062M)   : 0, 10061771
    benchmarks/2.9.0 (10.020M)   : 0, 10019592

    section Automatic
    This PR (5803) (6.297M)   : 0, 6296891
    master (6.546M)   : 0, 6545951
    benchmarks/2.9.0 (7.255M)   : 0, 7255257

    section Trace stats
    master (7.305M)   : 0, 7305404

    section Manual
    master (10.146M)   : 0, 10146170

    section Manual + Automatic
    This PR (5803) (6.028M)   : 0, 6027504
    master (6.140M)   : 0, 6140263

    section DD_TRACE_ENABLED=0
    master (9.363M)   : 0, 9362965
Loading

@lucaspimentel lucaspimentel changed the title [ASM] Iast events in meta struct [ASM] IAST events in span's meta_struct Jul 29, 2024
lucaspimentel
lucaspimentel approved these changes Jul 29, 2024
View reviewed changes
Copy link
Member

@lucaspimentel lucaspimentel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Left some non-blocking nit-picky comments about naming conventions and performance. Feel free to ignore.

@e-n-0 e-n-0 force-pushed the flavien/asm/iast/events_metastruct branch 2 times, most recently from 969ab2e to 425ee0e Compare October 3, 2024 18:16
@e-n-0 e-n-0 force-pushed the flavien/asm/iast/events_metastruct branch from 425ee0e to 1af268c Compare October 15, 2024 11:13
@e-n-0 e-n-0 force-pushed the flavien/asm/iast/events_metastruct branch 2 times, most recently from af9890a to 7eff3cf Compare October 15, 2024 14:03
@e-n-0 e-n-0 force-pushed the flavien/asm/iast/events_metastruct branch from 7eff3cf to df06a5f Compare November 16, 2024 14:28
@e-n-0 e-n-0 force-pushed the flavien/asm/iast/events_metastruct branch 4 times, most recently from 9749161 to db497e6 Compare November 26, 2024 23:34
@e-n-0 e-n-0 force-pushed the flavien/asm/iast/events_metastruct branch from e1f00a9 to 9728b26 Compare November 28, 2024 10:17
@e-n-0 e-n-0 marked this pull request as ready for review November 28, 2024 13:52
@e-n-0 e-n-0 merged commit 8e61a75 into master Dec 3, 2024
78 checks passed
@e-n-0 e-n-0 deleted the flavien/asm/iast/events_metastruct branch December 3, 2024 10:08
@github-actions github-actions bot added this to the vNext-v3 milestone Dec 3, 2024
@e-n-0 e-n-0 mentioned this pull request Dec 4, 2024
Merged
veerbia pushed a commit that referenced this pull request Dec 16, 2024
@e-n-0 @veerbia
[ASM] IAST events in span's meta_struct (#5803)
5a7f2d6 
## Summary of changes

Move iast data from the `_dd.iast.json` tag to a messagepack-encoded
data serialized into meta struct.
[RFC Linked to that
change.](https://docs.google.com/document/d/1iWQsOfT6Lg_IFyvQeqry9wVmXOE2Yav0X4MgOTk7mks/)

- Added a new telemetry tag `_dd.iast.metastruct.size.exceeded` to
reflect the exceeded size for meta struct when a truncation occured

## Reason for change

Better performance.

## Implementation details

- Check for meta struct compatibility with the connected agent
- Fallback to json serialization if not supported

## Test coverage

- By default, now the mocked trace agent is set to support meta struct
(cf #5779), so all iast events will be set in meta struct
- To keep all existing snapshots unchanged, meta struct iast events are
set back as json on the same tag into snapshots
- Add a new option in the iast test: "forceMetaStruct". When enabled,
the tag `_dd.iast.json.metastruct.test` is set when iast data from
metastruct is serialized back to json. This is to be sure that meta
struct is used in the test.
- Updated all unit tests of Vulnerability Batch to also use the meta
struct serialization
- New test for the telemetry tag `_dd.iast.metastruct.size.exceeded`

## Other details
<!-- Fixes #{issue} -->

<!-- ⚠️ Note: where possible, please obtain 2 approvals prior to
merging. Unless CODEOWNERS specifies otherwise, for external teams it is
typically best to have one review from a team member, and one review
from apm-dotnet. Trivial changes do not require 2 reviews. -->
@andrewlock andrewlock added type:performance Performance, speed, latency, resource usage (CPU, memory) area:asm-iast labels Dec 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lucaspimentel lucaspimentel lucaspimentel approved these changes

@datadog-datadog-prod-us1 datadog-datadog-prod-us1[bot] datadog-datadog-prod-us1[bot] left review comments

@daniel-romano-DD daniel-romano-DD daniel-romano-DD approved these changes

Assignees

@e-n-0 e-n-0

Labels
area:asm area:asm-iast type:performance Performance, speed, latency, resource usage (CPU, memory)
Projects
None yet
Milestone
3.7.0
Development

Successfully merging this pull request may close these issues.
4 participants
@e-n-0 @andrewlock @lucaspimentel @daniel-romano-DD