Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Do strict JSON comparison when running CI Vis instrumentation tests locally #8327

Merged
merged 2 commits into from
Feb 3, 2025
Merged

Do strict JSON comparison when running CI Vis instrumentation tests locally #8327

merged 2 commits into from
Feb 3, 2025

Conversation

nikita-tkachenko-datadog
Copy link
Contributor

@nikita-tkachenko-datadog nikita-tkachenko-datadog commented Feb 3, 2025 •
edited
Loading

What Does This Do

Updates tests fixtures comparison mechanism in CI Visibility instrumentations/smoke tests:

  • when the tests are run in CI, lenient comparison mode is used (new fields are ignored)
  • when the tests are run locally, strict comparison mode is used (new fields cause tests to fail)

Motivation

Make the tests more reliable, and enforce updating of the test fixtures.

Contributor Checklist

Jira ticket: SDTEST-1483

@nikita-tkachenko-datadog nikita-tkachenko-datadog added comp: testing Testing tag: no release notes Changes to exclude from release notes comp: ci visibility Continuous Integration Visibility labels Feb 3, 2025
Base automatically changed from nikita-tkachenko/quarantined-tests-tagging to master February 3, 2025 15:37
@pr-commenter
Copy link

pr-commenter bot commented Feb 3, 2025

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master nikita-tkachenko/strict-comparison-test-fixtures
git_commit_date 1738578316 1738594510
git_commit_sha 9af5347 9e2a709
release_version 1.47.0-SNAPSHOT~9af5347cda 1.47.0-SNAPSHOT~9e2a709dec
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1738597081 1738597081
ci_job_id 791356815 791356815
ci_pipeline_id 54722306 54722306
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-yk6jynms-project-304-concurrent-0-dwprism1 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux Linux runner-yk6jynms-project-304-concurrent-0-dwprism1 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
module Agent Agent
parent None None
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 52 metrics, 11 unstable metrics.

Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.47.0-SNAPSHOT~9e2a709dec, baseline=1.47.0-SNAPSHOT~9af5347cda

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.044 s) : 0, 1044374
Total [baseline] (10.605 s) : 0, 10605226
Agent [candidate] (1.042 s) : 0, 1042224
Total [candidate] (10.5 s) : 0, 10499573
section appsec
Agent [baseline] (1.184 s) : 0, 1184275
Total [baseline] (10.802 s) : 0, 10801956
Agent [candidate] (1.187 s) : 0, 1186922
Total [candidate] (10.79 s) : 0, 10789950
section iast
Agent [baseline] (1.182 s) : 0, 1181716
Total [baseline] (11.054 s) : 0, 11053655
Agent [candidate] (1.174 s) : 0, 1173916
Total [candidate] (11.007 s) : 0, 11007191
section profiling
Agent [baseline] (1.263 s) : 0, 1263061
Total [baseline] (10.931 s) : 0, 10931011
Agent [candidate] (1.27 s) : 0, 1269717
Total [candidate] (10.893 s) : 0, 10893433
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.044 s -
Agent appsec 1.184 s 139.902 ms (13.4%)
Agent iast 1.182 s 137.343 ms (13.2%)
Agent profiling 1.263 s 218.687 ms (20.9%)
Total tracing 10.605 s -
Total appsec 10.802 s 196.73 ms (1.9%)
Total iast 11.054 s 448.428 ms (4.2%)
Total profiling 10.931 s 325.785 ms (3.1%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.042 s -
Agent appsec 1.187 s 144.698 ms (13.9%)
Agent iast 1.174 s 131.692 ms (12.6%)
Agent profiling 1.27 s 227.493 ms (21.8%)
Total tracing 10.5 s -
Total appsec 10.79 s 290.378 ms (2.8%)
Total iast 11.007 s 507.618 ms (4.8%)
Total profiling 10.893 s 393.861 ms (3.8%) 
gantt
    title petclinic - break down per module: candidate=1.47.0-SNAPSHOT~9e2a709dec, baseline=1.47.0-SNAPSHOT~9af5347cda

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (717.876 ms) : 0, 717876
BytebuddyAgent [candidate] (717.156 ms) : 0, 717156
GlobalTracer [baseline] (243.976 ms) : 0, 243976
GlobalTracer [candidate] (241.617 ms) : 0, 241617
AppSec [baseline] (55.735 ms) : 0, 55735
AppSec [candidate] (55.411 ms) : 0, 55411
Remote Config [baseline] (724.044 µs) : 0, 724
Remote Config [candidate] (723.619 µs) : 0, 724
Telemetry [baseline] (10.9 ms) : 0, 10900
Telemetry [candidate] (12.093 ms) : 0, 12093
section appsec
BytebuddyAgent [baseline] (731.69 ms) : 0, 731690
BytebuddyAgent [candidate] (736.737 ms) : 0, 736737
GlobalTracer [baseline] (241.024 ms) : 0, 241024
GlobalTracer [candidate] (238.418 ms) : 0, 238418
AppSec [baseline] (176.595 ms) : 0, 176595
AppSec [candidate] (176.572 ms) : 0, 176572
Remote Config [baseline] (666.039 µs) : 0, 666
Remote Config [candidate] (652.965 µs) : 0, 653
Telemetry [baseline] (8.315 ms) : 0, 8315
Telemetry [candidate] (8.362 ms) : 0, 8362
IAST [baseline] (21.531 ms) : 0, 21531
IAST [candidate] (21.777 ms) : 0, 21777
section iast
BytebuddyAgent [baseline] (841.584 ms) : 0, 841584
BytebuddyAgent [candidate] (838.024 ms) : 0, 838024
GlobalTracer [baseline] (235.466 ms) : 0, 235466
GlobalTracer [candidate] (231.353 ms) : 0, 231353
AppSec [baseline] (51.445 ms) : 0, 51445
AppSec [candidate] (54.714 ms) : 0, 54714
Remote Config [baseline] (617.846 µs) : 0, 618
Remote Config [candidate] (611.46 µs) : 0, 611
Telemetry [baseline] (8.825 ms) : 0, 8825
Telemetry [candidate] (8.748 ms) : 0, 8748
IAST [baseline] (28.259 ms) : 0, 28259
IAST [candidate] (25.032 ms) : 0, 25032
section profiling
ProfilingAgent [baseline] (96.305 ms) : 0, 96305
ProfilingAgent [candidate] (96.118 ms) : 0, 96118
BytebuddyAgent [baseline] (705.663 ms) : 0, 705663
BytebuddyAgent [candidate] (711.595 ms) : 0, 711595
GlobalTracer [baseline] (353.217 ms) : 0, 353217
GlobalTracer [candidate] (354.746 ms) : 0, 354746
AppSec [baseline] (55.765 ms) : 0, 55765
AppSec [candidate] (54.947 ms) : 0, 54947
Remote Config [baseline] (730.454 µs) : 0, 730
Remote Config [candidate] (713.971 µs) : 0, 714
Telemetry [baseline] (9.053 ms) : 0, 9053
Telemetry [candidate] (9.038 ms) : 0, 9038
Profiling [baseline] (96.33 ms) : 0, 96330
Profiling [candidate] (96.142 ms) : 0, 96142
Loading
Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.47.0-SNAPSHOT~9e2a709dec, baseline=1.47.0-SNAPSHOT~9af5347cda

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.05 s) : 0, 1050385
Total [baseline] (8.697 s) : 0, 8697438
Agent [candidate] (1.04 s) : 0, 1039719
Total [candidate] (8.662 s) : 0, 8661846
section iast
Agent [baseline] (1.173 s) : 0, 1173383
Total [baseline] (9.254 s) : 0, 9253868
Agent [candidate] (1.176 s) : 0, 1176196
Total [candidate] (9.241 s) : 0, 9241118
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.176 s) : 0, 1175809
Total [baseline] (9.216 s) : 0, 9216067
Agent [candidate] (1.179 s) : 0, 1178807
Total [candidate] (9.203 s) : 0, 9203115
section iast_TELEMETRY_OFF
Agent [baseline] (1.176 s) : 0, 1175625
Total [baseline] (9.219 s) : 0, 9219353
Agent [candidate] (1.17 s) : 0, 1169781
Total [candidate] (9.293 s) : 0, 9292934
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.05 s -
Agent iast 1.173 s 122.998 ms (11.7%)
Agent iast_HARDCODED_SECRET_DISABLED 1.176 s 125.424 ms (11.9%)
Agent iast_TELEMETRY_OFF 1.176 s 125.24 ms (11.9%)
Total tracing 8.697 s -
Total iast 9.254 s 556.43 ms (6.4%)
Total iast_HARDCODED_SECRET_DISABLED 9.216 s 518.629 ms (6.0%)
Total iast_TELEMETRY_OFF 9.219 s 521.915 ms (6.0%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.04 s -
Agent iast 1.176 s 136.477 ms (13.1%)
Agent iast_HARDCODED_SECRET_DISABLED 1.179 s 139.088 ms (13.4%)
Agent iast_TELEMETRY_OFF 1.17 s 130.062 ms (12.5%)
Total tracing 8.662 s -
Total iast 9.241 s 579.272 ms (6.7%)
Total iast_HARDCODED_SECRET_DISABLED 9.203 s 541.269 ms (6.2%)
Total iast_TELEMETRY_OFF 9.293 s 631.088 ms (7.3%) 
gantt
    title insecure-bank - break down per module: candidate=1.47.0-SNAPSHOT~9e2a709dec, baseline=1.47.0-SNAPSHOT~9af5347cda

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (722.596 ms) : 0, 722596
BytebuddyAgent [candidate] (714.366 ms) : 0, 714366
GlobalTracer [baseline] (244.699 ms) : 0, 244699
GlobalTracer [candidate] (241.419 ms) : 0, 241419
AppSec [baseline] (55.289 ms) : 0, 55289
AppSec [candidate] (55.753 ms) : 0, 55753
Remote Config [baseline] (727.539 µs) : 0, 728
Remote Config [candidate] (718.361 µs) : 0, 718
Telemetry [baseline] (11.679 ms) : 0, 11679
Telemetry [candidate] (12.27 ms) : 0, 12270
section iast
BytebuddyAgent [baseline] (834.78 ms) : 0, 834780
BytebuddyAgent [candidate] (839.706 ms) : 0, 839706
GlobalTracer [baseline] (234.369 ms) : 0, 234369
GlobalTracer [candidate] (231.918 ms) : 0, 231918
IAST [baseline] (25.73 ms) : 0, 25730
IAST [candidate] (25.029 ms) : 0, 25029
AppSec [baseline] (53.907 ms) : 0, 53907
AppSec [candidate] (54.92 ms) : 0, 54920
Remote Config [baseline] (621.707 µs) : 0, 622
Remote Config [candidate] (624.995 µs) : 0, 625
Telemetry [baseline] (8.684 ms) : 0, 8684
Telemetry [candidate] (8.704 ms) : 0, 8704
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (837.051 ms) : 0, 837051
BytebuddyAgent [candidate] (841.293 ms) : 0, 841293
GlobalTracer [baseline] (233.766 ms) : 0, 233766
GlobalTracer [candidate] (233.057 ms) : 0, 233057
IAST [baseline] (25.924 ms) : 0, 25924
IAST [candidate] (25.807 ms) : 0, 25807
AppSec [baseline] (54.343 ms) : 0, 54343
AppSec [candidate] (53.992 ms) : 0, 53992
Remote Config [baseline] (609.291 µs) : 0, 609
Remote Config [candidate] (623.156 µs) : 0, 623
Telemetry [baseline] (8.738 ms) : 0, 8738
Telemetry [candidate] (8.69 ms) : 0, 8690
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (837.61 ms) : 0, 837610
BytebuddyAgent [candidate] (833.922 ms) : 0, 833922
GlobalTracer [baseline] (234.211 ms) : 0, 234211
GlobalTracer [candidate] (232.116 ms) : 0, 232116
IAST [baseline] (24.599 ms) : 0, 24599
IAST [candidate] (25.326 ms) : 0, 25326
AppSec [baseline] (54.679 ms) : 0, 54679
AppSec [candidate] (53.95 ms) : 0, 53950
Remote Config [baseline] (603.087 µs) : 0, 603
Remote Config [candidate] (612.824 µs) : 0, 613
Telemetry [baseline] (8.581 ms) : 0, 8581
Telemetry [candidate] (8.619 ms) : 0, 8619
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
end_time 2025-02-03T15:04:36 2025-02-03T15:11:39
git_branch master nikita-tkachenko/strict-comparison-test-fixtures
git_commit_date 1738578316 1738594510
git_commit_sha 9af5347 9e2a709
release_version 1.47.0-SNAPSHOT~9af5347cda 1.47.0-SNAPSHOT~9e2a709dec
start_time 2025-02-03T15:04:22 2025-02-03T15:11:26
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1738595856 1738595856
ci_job_id 791356816 791356816
ci_pipeline_id 54722306 54722306
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-hf2spk2o-project-304-concurrent-0-nsqq9bf0 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux Linux runner-hf2spk2o-project-304-concurrent-0-nsqq9bf0 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 10 metrics, 18 unstable metrics.

Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.47.0-SNAPSHOT~9e2a709dec, baseline=1.47.0-SNAPSHOT~9af5347cda
    dateFormat X
    axisFormat %s
section baseline
no_agent (376.376 µs) : 357, 396
.   : milestone, 376,
iast (506.943 µs) : 485, 529
.   : milestone, 507,
iast_FULL (741.528 µs) : 720, 764
.   : milestone, 742,
iast_GLOBAL (553.982 µs) : 533, 575
.   : milestone, 554,
iast_HARDCODED_SECRET_DISABLED (503.565 µs) : 482, 525
.   : milestone, 504,
iast_INACTIVE (455.836 µs) : 435, 477
.   : milestone, 456,
iast_TELEMETRY_OFF (496.368 µs) : 473, 519
.   : milestone, 496,
tracing (460.144 µs) : 435, 485
.   : milestone, 460,
section candidate
no_agent (378.761 µs) : 359, 398
.   : milestone, 379,
iast (510.763 µs) : 489, 533
.   : milestone, 511,
iast_FULL (746.154 µs) : 724, 768
.   : milestone, 746,
iast_GLOBAL (547.714 µs) : 526, 569
.   : milestone, 548,
iast_HARDCODED_SECRET_DISABLED (506.048 µs) : 483, 529
.   : milestone, 506,
iast_INACTIVE (459.718 µs) : 438, 481
.   : milestone, 460,
iast_TELEMETRY_OFF (500.52 µs) : 477, 524
.   : milestone, 501,
tracing (460.884 µs) : 439, 483
.   : milestone, 461,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 376.376 µs [356.562 µs, 396.191 µs] -
iast 506.943 µs [485.154 µs, 528.732 µs] 130.567 µs (34.7%)
iast_FULL 741.528 µs [719.542 µs, 763.513 µs] 365.151 µs (97.0%)
iast_GLOBAL 553.982 µs [532.633 µs, 575.331 µs] 177.606 µs (47.2%)
iast_HARDCODED_SECRET_DISABLED 503.565 µs [482.043 µs, 525.088 µs] 127.189 µs (33.8%)
iast_INACTIVE 455.836 µs [434.506 µs, 477.167 µs] 79.46 µs (21.1%)
iast_TELEMETRY_OFF 496.368 µs [473.243 µs, 519.493 µs] 119.991 µs (31.9%)
tracing 460.144 µs [435.406 µs, 484.881 µs] 83.767 µs (22.3%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 378.761 µs [359.159 µs, 398.363 µs] -
iast 510.763 µs [488.897 µs, 532.63 µs] 132.002 µs (34.9%)
iast_FULL 746.154 µs [724.025 µs, 768.282 µs] 367.393 µs (97.0%)
iast_GLOBAL 547.714 µs [526.092 µs, 569.336 µs] 168.953 µs (44.6%)
iast_HARDCODED_SECRET_DISABLED 506.048 µs [482.928 µs, 529.169 µs] 127.287 µs (33.6%)
iast_INACTIVE 459.718 µs [437.997 µs, 481.439 µs] 80.957 µs (21.4%)
iast_TELEMETRY_OFF 500.52 µs [476.96 µs, 524.081 µs] 121.759 µs (32.1%)
tracing 460.884 µs [438.696 µs, 483.071 µs] 82.123 µs (21.7%)
Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.47.0-SNAPSHOT~9e2a709dec, baseline=1.47.0-SNAPSHOT~9af5347cda
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.354 ms) : 1335, 1373
.   : milestone, 1354,
appsec (1.723 ms) : 1700, 1747
.   : milestone, 1723,
appsec_no_iast (1.749 ms) : 1723, 1775
.   : milestone, 1749,
iast (1.504 ms) : 1480, 1528
.   : milestone, 1504,
profiling (1.535 ms) : 1509, 1560
.   : milestone, 1535,
tracing (1.491 ms) : 1465, 1516
.   : milestone, 1491,
section candidate
no_agent (1.349 ms) : 1330, 1369
.   : milestone, 1349,
appsec (1.746 ms) : 1722, 1770
.   : milestone, 1746,
appsec_no_iast (1.762 ms) : 1738, 1786
.   : milestone, 1762,
iast (1.506 ms) : 1482, 1529
.   : milestone, 1506,
profiling (1.505 ms) : 1483, 1528
.   : milestone, 1505,
tracing (1.48 ms) : 1455, 1505
.   : milestone, 1480,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.354 ms [1.335 ms, 1.373 ms] -
appsec 1.723 ms [1.7 ms, 1.747 ms] 369.331 µs (27.3%)
appsec_no_iast 1.749 ms [1.723 ms, 1.775 ms] 395.216 µs (29.2%)
iast 1.504 ms [1.48 ms, 1.528 ms] 149.499 µs (11.0%)
profiling 1.535 ms [1.509 ms, 1.56 ms] 180.526 µs (13.3%)
tracing 1.491 ms [1.465 ms, 1.516 ms] 136.4 µs (10.1%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.349 ms [1.33 ms, 1.369 ms] -
appsec 1.746 ms [1.722 ms, 1.77 ms] 396.436 µs (29.4%)
appsec_no_iast 1.762 ms [1.738 ms, 1.786 ms] 412.84 µs (30.6%)
iast 1.506 ms [1.482 ms, 1.529 ms] 156.041 µs (11.6%)
profiling 1.505 ms [1.483 ms, 1.528 ms] 155.81 µs (11.5%)
tracing 1.48 ms [1.455 ms, 1.505 ms] 130.38 µs (9.7%)

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master nikita-tkachenko/strict-comparison-test-fixtures
git_commit_date 1738578316 1738594510
git_commit_sha 9af5347 9e2a709
release_version 1.47.0-SNAPSHOT~9af5347cda 1.47.0-SNAPSHOT~9e2a709dec
See matching parameters
Baseline Candidate
application biojava biojava
ci_job_date 1738596577 1738596577
ci_job_id 791356817 791356817
ci_pipeline_id 54722306 54722306
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-rt13twwf-project-304-concurrent-0-ujit8kg7 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux Linux runner-rt13twwf-project-304-concurrent-0-ujit8kg7 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
variant appsec appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for biojava 
gantt
    title biojava - execution time [CI 0.99] : candidate=1.47.0-SNAPSHOT~9e2a709dec, baseline=1.47.0-SNAPSHOT~9af5347cda
    dateFormat X
    axisFormat %s
section baseline
no_agent (14.908 s) : 14908000, 14908000
.   : milestone, 14908000,
appsec (15.054 s) : 15054000, 15054000
.   : milestone, 15054000,
iast (18.431 s) : 18431000, 18431000
.   : milestone, 18431000,
iast_GLOBAL (17.942 s) : 17942000, 17942000
.   : milestone, 17942000,
profiling (15.313 s) : 15313000, 15313000
.   : milestone, 15313000,
tracing (14.958 s) : 14958000, 14958000
.   : milestone, 14958000,
section candidate
no_agent (15.433 s) : 15433000, 15433000
.   : milestone, 15433000,
appsec (14.935 s) : 14935000, 14935000
.   : milestone, 14935000,
iast (18.376 s) : 18376000, 18376000
.   : milestone, 18376000,
iast_GLOBAL (18.204 s) : 18204000, 18204000
.   : milestone, 18204000,
profiling (15.316 s) : 15316000, 15316000
.   : milestone, 15316000,
tracing (14.842 s) : 14842000, 14842000
.   : milestone, 14842000,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 14.908 s [14.908 s, 14.908 s] -
appsec 15.054 s [15.054 s, 15.054 s] 146.0 ms (1.0%)
iast 18.431 s [18.431 s, 18.431 s] 3.523 s (23.6%)
iast_GLOBAL 17.942 s [17.942 s, 17.942 s] 3.034 s (20.4%)
profiling 15.313 s [15.313 s, 15.313 s] 405.0 ms (2.7%)
tracing 14.958 s [14.958 s, 14.958 s] 50.0 ms (0.3%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.433 s [15.433 s, 15.433 s] -
appsec 14.935 s [14.935 s, 14.935 s] -498.0 ms (-3.2%)
iast 18.376 s [18.376 s, 18.376 s] 2.943 s (19.1%)
iast_GLOBAL 18.204 s [18.204 s, 18.204 s] 2.771 s (18.0%)
profiling 15.316 s [15.316 s, 15.316 s] -117.0 ms (-0.8%)
tracing 14.842 s [14.842 s, 14.842 s] -591.0 ms (-3.8%)
Execution time for tomcat 
gantt
    title tomcat - execution time [CI 0.99] : candidate=1.47.0-SNAPSHOT~9e2a709dec, baseline=1.47.0-SNAPSHOT~9af5347cda
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.476 ms) : 1465, 1488
.   : milestone, 1476,
appsec (2.374 ms) : 2331, 2418
.   : milestone, 2374,
iast (2.115 ms) : 2060, 2169
.   : milestone, 2115,
iast_GLOBAL (2.167 ms) : 2111, 2222
.   : milestone, 2167,
profiling (1.981 ms) : 1937, 2024
.   : milestone, 1981,
tracing (1.958 ms) : 1915, 2000
.   : milestone, 1958,
section candidate
no_agent (1.477 ms) : 1465, 1489
.   : milestone, 1477,
appsec (2.364 ms) : 2320, 2407
.   : milestone, 2364,
iast (2.122 ms) : 2067, 2177
.   : milestone, 2122,
iast_GLOBAL (2.157 ms) : 2102, 2213
.   : milestone, 2157,
profiling (1.982 ms) : 1938, 2027
.   : milestone, 1982,
tracing (1.961 ms) : 1919, 2004
.   : milestone, 1961,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.476 ms [1.465 ms, 1.488 ms] -
appsec 2.374 ms [2.331 ms, 2.418 ms] 897.948 µs (60.8%)
iast 2.115 ms [2.06 ms, 2.169 ms] 638.138 µs (43.2%)
iast_GLOBAL 2.167 ms [2.111 ms, 2.222 ms] 690.126 µs (46.7%)
profiling 1.981 ms [1.937 ms, 2.024 ms] 504.068 µs (34.1%)
tracing 1.958 ms [1.915 ms, 2.0 ms] 481.085 µs (32.6%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.477 ms [1.465 ms, 1.489 ms] -
appsec 2.364 ms [2.32 ms, 2.407 ms] 886.579 µs (60.0%)
iast 2.122 ms [2.067 ms, 2.177 ms] 645.189 µs (43.7%)
iast_GLOBAL 2.157 ms [2.102 ms, 2.213 ms] 680.18 µs (46.1%)
profiling 1.982 ms [1.938 ms, 2.027 ms] 505.323 µs (34.2%)
tracing 1.961 ms [1.919 ms, 2.004 ms] 484.306 µs (32.8%)

@nikita-tkachenko-datadog nikita-tkachenko-datadog force-pushed the nikita-tkachenko/strict-comparison-test-fixtures branch from 9e2a709 to 5fc6d58 Compare February 3, 2025 15:54
@nikita-tkachenko-datadog nikita-tkachenko-datadog marked this pull request as ready for review February 3, 2025 16:05
@nikita-tkachenko-datadog nikita-tkachenko-datadog requested a review from a team as a code owner February 3, 2025 16:05
@nikita-tkachenko-datadog nikita-tkachenko-datadog merged commit 5d18c24 into master Feb 3, 2025
195 of 196 checks passed
@nikita-tkachenko-datadog nikita-tkachenko-datadog deleted the nikita-tkachenko/strict-comparison-test-fixtures branch February 3, 2025 16:34
@github-actions github-actions bot added this to the 1.47.0 milestone Feb 3, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@daniel-mohedano daniel-mohedano daniel-mohedano approved these changes

@juan-fernandez juan-fernandez Awaiting requested review from juan-fernandez juan-fernandez is a code owner automatically assigned from DataDog/ci-app-libraries-java

Assignees
No one assigned
Labels
comp: ci visibility Continuous Integration Visibility comp: testing Testing tag: no release notes Changes to exclude from release notes
Projects
None yet
Milestone
1.47.0
Development

Successfully merging this pull request may close these issues.
2 participants
@nikita-tkachenko-datadog @daniel-mohedano