Skip to content

Revert "SSI smoke tests should only run as part of nightly" #8924

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Jun 5, 2025
Merged

Revert "SSI smoke tests should only run as part of nightly" #8924

merged 1 commit into from
Jun 5, 2025

Conversation

@bric3
Copy link
Contributor

@bric3 bric3 commented Jun 4, 2025 •
edited
Loading

Reverts #8914

We cannot proceed to a release without validating the smoke tests.

Related to #8925

@bric3 bric3 requested a review from a team as a code owner June 4, 2025 14:22
@bric3 bric3 requested review from TonyCTHsu and removed request for a team June 4, 2025 14:22
@github-actions
Copy link
Contributor

github-actions bot commented Jun 4, 2025 •
edited
Loading

Hi! 👋 Thanks for your pull request! 🎉

To help us review it, please make sure to:

  • Add at least one type, and one component or instrumentation label to the pull request

If you need help, please check our contributing guidelines.

@bric3 bric3 requested a review from PerfectSlayer June 4, 2025 14:37
@pr-commenter
Copy link

pr-commenter bot commented Jun 4, 2025

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master revert-8914-landerson/ssi-nightly-only
git_commit_date 1749044153 1749046908
git_commit_sha 7787af7 4733dd3
release_version 1.50.0-SNAPSHOT~7787af738f 1.50.0-SNAPSHOT~4733dd3bc1
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1749049314 1749049314
ci_job_id 967715679 967715679
ci_pipeline_id 66943347 66943347
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-xizkxfyi-project-304-concurrent-0-hb2sln65 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-xizkxfyi-project-304-concurrent-0-hb2sln65 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module Agent Agent
parent None None
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 57 metrics, 14 unstable metrics.

Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.50.0-SNAPSHOT~4733dd3bc1, baseline=1.50.0-SNAPSHOT~7787af738f

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.023 s) : 0, 1022876
Total [baseline] (11.17 s) : 0, 11169522
Agent [candidate] (1.042 s) : 0, 1041993
Total [candidate] (11.227 s) : 0, 11226591
section appsec
Agent [baseline] (1.161 s) : 0, 1161003
Total [baseline] (11.255 s) : 0, 11254826
Agent [candidate] (1.165 s) : 0, 1164741
Total [candidate] (11.235 s) : 0, 11235430
section iast
Agent [baseline] (1.149 s) : 0, 1149001
Total [baseline] (11.327 s) : 0, 11326621
Agent [candidate] (1.157 s) : 0, 1157357
Total [candidate] (11.377 s) : 0, 11377322
section profiling
Agent [baseline] (1.271 s) : 0, 1271231
Total [baseline] (11.471 s) : 0, 11470934
Agent [candidate] (1.265 s) : 0, 1264888
Total [candidate] (11.539 s) : 0, 11539232
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.023 s -
Agent appsec 1.161 s 138.126 ms (13.5%)
Agent iast 1.149 s 126.125 ms (12.3%)
Agent profiling 1.271 s 248.355 ms (24.3%)
Total tracing 11.17 s -
Total appsec 11.255 s 85.304 ms (0.8%)
Total iast 11.327 s 157.099 ms (1.4%)
Total profiling 11.471 s 301.412 ms (2.7%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.042 s -
Agent appsec 1.165 s 122.748 ms (11.8%)
Agent iast 1.157 s 115.363 ms (11.1%)
Agent profiling 1.265 s 222.894 ms (21.4%)
Total tracing 11.227 s -
Total appsec 11.235 s 8.838 ms (0.1%)
Total iast 11.377 s 150.731 ms (1.3%)
Total profiling 11.539 s 312.641 ms (2.8%) 
gantt
    title petclinic - break down per module: candidate=1.50.0-SNAPSHOT~4733dd3bc1, baseline=1.50.0-SNAPSHOT~7787af738f

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (682.417 ms) : 0, 682417
BytebuddyAgent [candidate] (695.362 ms) : 0, 695362
GlobalTracer [baseline] (240.459 ms) : 0, 240459
GlobalTracer [candidate] (244.451 ms) : 0, 244451
AppSec [baseline] (56.326 ms) : 0, 56326
AppSec [candidate] (57.994 ms) : 0, 57994
Debugger [baseline] (6.209 ms) : 0, 6209
Debugger [candidate] (6.369 ms) : 0, 6369
Remote Config [baseline] (761.088 µs) : 0, 761
Remote Config [candidate] (757.226 µs) : 0, 757
Telemetry [baseline] (13.011 ms) : 0, 13011
Telemetry [candidate] (13.043 ms) : 0, 13043
section appsec
BytebuddyAgent [baseline] (698.68 ms) : 0, 698680
BytebuddyAgent [candidate] (701.592 ms) : 0, 701592
GlobalTracer [baseline] (237.384 ms) : 0, 237384
GlobalTracer [candidate] (238.112 ms) : 0, 238112
IAST [baseline] (21.756 ms) : 0, 21756
IAST [candidate] (21.875 ms) : 0, 21875
AppSec [baseline] (176.556 ms) : 0, 176556
AppSec [candidate] (176.515 ms) : 0, 176515
Debugger [baseline] (5.977 ms) : 0, 5977
Debugger [candidate] (5.973 ms) : 0, 5973
Remote Config [baseline] (625.052 µs) : 0, 625
Remote Config [candidate] (636.798 µs) : 0, 637
Telemetry [baseline] (7.365 ms) : 0, 7365
Telemetry [candidate] (7.349 ms) : 0, 7349
section iast
BytebuddyAgent [baseline] (800.194 ms) : 0, 800194
BytebuddyAgent [candidate] (806.974 ms) : 0, 806974
GlobalTracer [baseline] (230.635 ms) : 0, 230635
GlobalTracer [candidate] (231.612 ms) : 0, 231612
IAST [baseline] (26.977 ms) : 0, 26977
IAST [candidate] (27.14 ms) : 0, 27140
AppSec [baseline] (53.198 ms) : 0, 53198
AppSec [candidate] (53.219 ms) : 0, 53219
Debugger [baseline] (5.99 ms) : 0, 5990
Debugger [candidate] (6.0 ms) : 0, 6000
Remote Config [baseline] (611.343 µs) : 0, 611
Remote Config [candidate] (611.457 µs) : 0, 611
Telemetry [baseline] (7.93 ms) : 0, 7930
Telemetry [candidate] (8.069 ms) : 0, 8069
section profiling
BytebuddyAgent [baseline] (677.022 ms) : 0, 677022
BytebuddyAgent [candidate] (674.569 ms) : 0, 674569
GlobalTracer [baseline] (361.717 ms) : 0, 361717
GlobalTracer [candidate] (360.113 ms) : 0, 360113
AppSec [baseline] (62.035 ms) : 0, 62035
AppSec [candidate] (61.602 ms) : 0, 61602
Debugger [baseline] (6.086 ms) : 0, 6086
Debugger [candidate] (6.132 ms) : 0, 6132
Remote Config [baseline] (654.226 µs) : 0, 654
Remote Config [candidate] (669.841 µs) : 0, 670
Telemetry [baseline] (8.074 ms) : 0, 8074
Telemetry [candidate] (8.14 ms) : 0, 8140
ProfilingAgent [baseline] (104.606 ms) : 0, 104606
ProfilingAgent [candidate] (102.855 ms) : 0, 102855
Profiling [baseline] (104.632 ms) : 0, 104632
Profiling [candidate] (102.879 ms) : 0, 102879
Loading
Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.50.0-SNAPSHOT~4733dd3bc1, baseline=1.50.0-SNAPSHOT~7787af738f

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.021 s) : 0, 1021250
Total [baseline] (8.523 s) : 0, 8523434
Agent [candidate] (1.022 s) : 0, 1022243
Total [candidate] (8.532 s) : 0, 8532464
section iast
Agent [baseline] (1.149 s) : 0, 1149477
Total [baseline] (9.2 s) : 0, 9200327
Agent [candidate] (1.149 s) : 0, 1148790
Total [candidate] (9.213 s) : 0, 9212907
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.15 s) : 0, 1150492
Total [baseline] (9.143 s) : 0, 9142655
Agent [candidate] (1.148 s) : 0, 1148466
Total [candidate] (9.128 s) : 0, 9127891
section iast_TELEMETRY_OFF
Agent [baseline] (1.144 s) : 0, 1143694
Total [baseline] (9.214 s) : 0, 9214172
Agent [candidate] (1.146 s) : 0, 1145774
Total [candidate] (9.235 s) : 0, 9234835
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.021 s -
Agent iast 1.149 s 128.226 ms (12.6%)
Agent iast_HARDCODED_SECRET_DISABLED 1.15 s 129.242 ms (12.7%)
Agent iast_TELEMETRY_OFF 1.144 s 122.444 ms (12.0%)
Total tracing 8.523 s -
Total iast 9.2 s 676.893 ms (7.9%)
Total iast_HARDCODED_SECRET_DISABLED 9.143 s 619.221 ms (7.3%)
Total iast_TELEMETRY_OFF 9.214 s 690.738 ms (8.1%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.022 s -
Agent iast 1.149 s 126.547 ms (12.4%)
Agent iast_HARDCODED_SECRET_DISABLED 1.148 s 126.222 ms (12.3%)
Agent iast_TELEMETRY_OFF 1.146 s 123.531 ms (12.1%)
Total tracing 8.532 s -
Total iast 9.213 s 680.443 ms (8.0%)
Total iast_HARDCODED_SECRET_DISABLED 9.128 s 595.428 ms (7.0%)
Total iast_TELEMETRY_OFF 9.235 s 702.371 ms (8.2%) 
gantt
    title insecure-bank - break down per module: candidate=1.50.0-SNAPSHOT~4733dd3bc1, baseline=1.50.0-SNAPSHOT~7787af738f

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (681.333 ms) : 0, 681333
BytebuddyAgent [candidate] (682.487 ms) : 0, 682487
GlobalTracer [baseline] (239.494 ms) : 0, 239494
GlobalTracer [candidate] (240.371 ms) : 0, 240371
AppSec [baseline] (56.295 ms) : 0, 56295
AppSec [candidate] (57.562 ms) : 0, 57562
Debugger [baseline] (6.214 ms) : 0, 6214
Debugger [candidate] (6.164 ms) : 0, 6164
Remote Config [baseline] (748.949 µs) : 0, 749
Remote Config [candidate] (728.125 µs) : 0, 728
Telemetry [baseline] (13.616 ms) : 0, 13616
Telemetry [candidate] (11.344 ms) : 0, 11344
section iast
BytebuddyAgent [baseline] (801.246 ms) : 0, 801246
BytebuddyAgent [candidate] (801.444 ms) : 0, 801444
GlobalTracer [baseline] (230.383 ms) : 0, 230383
GlobalTracer [candidate] (230.506 ms) : 0, 230506
IAST [baseline] (26.272 ms) : 0, 26272
IAST [candidate] (27.576 ms) : 0, 27576
AppSec [baseline] (53.584 ms) : 0, 53584
AppSec [candidate] (51.331 ms) : 0, 51331
Debugger [baseline] (6.002 ms) : 0, 6002
Debugger [candidate] (5.943 ms) : 0, 5943
Remote Config [baseline] (588.251 µs) : 0, 588
Remote Config [candidate] (596.254 µs) : 0, 596
Telemetry [baseline] (7.836 ms) : 0, 7836
Telemetry [candidate] (7.933 ms) : 0, 7933
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (802.631 ms) : 0, 802631
BytebuddyAgent [candidate] (800.53 ms) : 0, 800530
GlobalTracer [baseline] (230.622 ms) : 0, 230622
GlobalTracer [candidate] (230.4 ms) : 0, 230400
IAST [baseline] (29.071 ms) : 0, 29071
IAST [candidate] (26.995 ms) : 0, 26995
AppSec [baseline] (50.159 ms) : 0, 50159
AppSec [candidate] (52.58 ms) : 0, 52580
Debugger [baseline] (5.905 ms) : 0, 5905
Debugger [candidate] (5.917 ms) : 0, 5917
Remote Config [baseline] (654.941 µs) : 0, 655
Remote Config [candidate] (604.077 µs) : 0, 604
Telemetry [baseline] (7.924 ms) : 0, 7924
Telemetry [candidate] (7.929 ms) : 0, 7929
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (797.096 ms) : 0, 797096
BytebuddyAgent [candidate] (798.1 ms) : 0, 798100
GlobalTracer [baseline] (230.266 ms) : 0, 230266
GlobalTracer [candidate] (231.022 ms) : 0, 231022
IAST [baseline] (30.013 ms) : 0, 30013
IAST [candidate] (27.088 ms) : 0, 27088
AppSec [baseline] (48.343 ms) : 0, 48343
AppSec [candidate] (51.602 ms) : 0, 51602
Debugger [baseline] (5.976 ms) : 0, 5976
Debugger [candidate] (6.024 ms) : 0, 6024
Remote Config [baseline] (592.502 µs) : 0, 593
Remote Config [candidate] (611.507 µs) : 0, 612
Telemetry [baseline] (7.914 ms) : 0, 7914
Telemetry [candidate] (7.812 ms) : 0, 7812
Loading

Load

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master revert-8914-landerson/ssi-nightly-only
git_commit_date 1749044153 1749046908
git_commit_sha 7787af7 4733dd3
release_version 1.50.0-SNAPSHOT~7787af738f 1.50.0-SNAPSHOT~4733dd3bc1
See matching parameters
Baseline Candidate
application biojava biojava
ci_job_date 1749048815 1749048815
ci_job_id 967715681 967715681
ci_pipeline_id 66943347 66943347
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-xizkxfyi-project-304-concurrent-1-9rjnxr1s 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-xizkxfyi-project-304-concurrent-1-9rjnxr1s 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
variant appsec appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for biojava 
gantt
    title biojava - execution time [CI 0.99] : candidate=1.50.0-SNAPSHOT~4733dd3bc1, baseline=1.50.0-SNAPSHOT~7787af738f
    dateFormat X
    axisFormat %s
section baseline
no_agent (14.96 s) : 14960000, 14960000
.   : milestone, 14960000,
appsec (14.913 s) : 14913000, 14913000
.   : milestone, 14913000,
iast (18.135 s) : 18135000, 18135000
.   : milestone, 18135000,
iast_GLOBAL (18.11 s) : 18110000, 18110000
.   : milestone, 18110000,
profiling (15.335 s) : 15335000, 15335000
.   : milestone, 15335000,
tracing (14.803 s) : 14803000, 14803000
.   : milestone, 14803000,
section candidate
no_agent (15.296 s) : 15296000, 15296000
.   : milestone, 15296000,
appsec (14.639 s) : 14639000, 14639000
.   : milestone, 14639000,
iast (18.87 s) : 18870000, 18870000
.   : milestone, 18870000,
iast_GLOBAL (17.94 s) : 17940000, 17940000
.   : milestone, 17940000,
profiling (15.166 s) : 15166000, 15166000
.   : milestone, 15166000,
tracing (15.111 s) : 15111000, 15111000
.   : milestone, 15111000,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 14.96 s [14.96 s, 14.96 s] -
appsec 14.913 s [14.913 s, 14.913 s] -47.0 ms (-0.3%)
iast 18.135 s [18.135 s, 18.135 s] 3.175 s (21.2%)
iast_GLOBAL 18.11 s [18.11 s, 18.11 s] 3.15 s (21.1%)
profiling 15.335 s [15.335 s, 15.335 s] 375.0 ms (2.5%)
tracing 14.803 s [14.803 s, 14.803 s] -157.0 ms (-1.0%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.296 s [15.296 s, 15.296 s] -
appsec 14.639 s [14.639 s, 14.639 s] -657.0 ms (-4.3%)
iast 18.87 s [18.87 s, 18.87 s] 3.574 s (23.4%)
iast_GLOBAL 17.94 s [17.94 s, 17.94 s] 2.644 s (17.3%)
profiling 15.166 s [15.166 s, 15.166 s] -130.0 ms (-0.8%)
tracing 15.111 s [15.111 s, 15.111 s] -185.0 ms (-1.2%)
Execution time for tomcat 
gantt
    title tomcat - execution time [CI 0.99] : candidate=1.50.0-SNAPSHOT~4733dd3bc1, baseline=1.50.0-SNAPSHOT~7787af738f
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.472 ms) : 1461, 1484
.   : milestone, 1472,
appsec (2.396 ms) : 2347, 2445
.   : milestone, 2396,
iast (2.179 ms) : 2118, 2240
.   : milestone, 2179,
iast_GLOBAL (2.217 ms) : 2155, 2278
.   : milestone, 2217,
profiling (2.027 ms) : 1977, 2076
.   : milestone, 2027,
tracing (1.995 ms) : 1947, 2042
.   : milestone, 1995,
section candidate
no_agent (1.471 ms) : 1460, 1483
.   : milestone, 1471,
appsec (2.394 ms) : 2346, 2443
.   : milestone, 2394,
iast (2.184 ms) : 2122, 2246
.   : milestone, 2184,
iast_GLOBAL (2.221 ms) : 2160, 2283
.   : milestone, 2221,
profiling (2.024 ms) : 1974, 2073
.   : milestone, 2024,
tracing (2.008 ms) : 1960, 2056
.   : milestone, 2008,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.472 ms [1.461 ms, 1.484 ms] -
appsec 2.396 ms [2.347 ms, 2.445 ms] 923.347 µs (62.7%)
iast 2.179 ms [2.118 ms, 2.24 ms] 706.67 µs (48.0%)
iast_GLOBAL 2.217 ms [2.155 ms, 2.278 ms] 744.533 µs (50.6%)
profiling 2.027 ms [1.977 ms, 2.076 ms] 554.111 µs (37.6%)
tracing 1.995 ms [1.947 ms, 2.042 ms] 522.159 µs (35.5%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.471 ms [1.46 ms, 1.483 ms] -
appsec 2.394 ms [2.346 ms, 2.443 ms] 923.179 µs (62.8%)
iast 2.184 ms [2.122 ms, 2.246 ms] 712.696 µs (48.4%)
iast_GLOBAL 2.221 ms [2.16 ms, 2.283 ms] 749.944 µs (51.0%)
profiling 2.024 ms [1.974 ms, 2.073 ms] 552.566 µs (37.6%)
tracing 2.008 ms [1.96 ms, 2.056 ms] 536.95 µs (36.5%)

@PerfectSlayer PerfectSlayer added tag: no release notes Changes to exclude from release notes comp: tooling Build & Tooling labels Jun 4, 2025
@PerfectSlayer PerfectSlayer merged commit 1205c9a into master Jun 5, 2025
508 of 511 checks passed
@PerfectSlayer PerfectSlayer deleted the revert-8914-landerson/ssi-nightly-only branch June 5, 2025 04:45
@github-actions github-actions bot added this to the 1.50.0 milestone Jun 5, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@TonyCTHsu TonyCTHsu Awaiting requested review from TonyCTHsu TonyCTHsu is a code owner automatically assigned from DataDog/apm-release-platform

@PerfectSlayer PerfectSlayer Awaiting requested review from PerfectSlayer

Assignees

No one assigned

Labels

comp: tooling Build & Tooling tag: no release notes Changes to exclude from release notes

Projects

None yet

Milestone

1.50.0

Development

Successfully merging this pull request may close these issues.

3 participants

@bric3 @PerfectSlayer