Remove skip and shouldBeSkipped methods from TestEventsHandler in favor of isSkippable #8286

nikita-tkachenko-datadog · 2025-01-27T12:05:11Z

What Does This Do

Removes shouldBeSkipped and skip methods from TestEventsHandler interface and replaces them with isSkippable.

Motivation

Laying the groundwork for future test quarantining: when test quarantine is implemented, ITR will no longer be the only reason for skipping tests, so some adjustments to skipping logic will have to be made.

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any usefull labels
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the public documentation in case of new configuration flag or behavior

Jira ticket: SDTEST-1483

…or of isSkippable

pr-commenter · 2025-01-27T12:48:19Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	nikita-tkachenko/itr-skip-refactor
git_commit_date	1737969316	1737979436
git_commit_sha	`19da739`	`2d6d97e`
release_version	1.46.0-SNAPSHOT~19da739a7c	1.46.0-SNAPSHOT~2d6d97ea75

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1737982054	1737982054
ci_job_id	781287830	781287830
ci_pipeline_id	54045236	54045236
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-8wlcx2s5-project-304-concurrent-0-volu3dht 6.8.0-1018-aws #19~22.04.1-Ubuntu SMP Wed Oct 9 16:48:22 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-8wlcx2s5-project-304-concurrent-0-volu3dht 6.8.0-1018-aws #19~22.04.1-Ubuntu SMP Wed Oct 9 16:48:22 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 59 metrics, 4 unstable metrics.

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.46.0-SNAPSHOT~2d6d97ea75, baseline=1.46.0-SNAPSHOT~19da739a7c

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.034 s) : 0, 1033968
Total [baseline] (10.486 s) : 0, 10485905
Agent [candidate] (1.041 s) : 0, 1040938
Total [candidate] (10.432 s) : 0, 10431730
section appsec
Agent [baseline] (1.182 s) : 0, 1182178
Total [baseline] (10.763 s) : 0, 10762929
Agent [candidate] (1.185 s) : 0, 1185079
Total [candidate] (10.74 s) : 0, 10739938
section iast
Agent [baseline] (1.172 s) : 0, 1172093
Total [baseline] (11.039 s) : 0, 11038747
Agent [candidate] (1.166 s) : 0, 1166401
Total [candidate] (10.901 s) : 0, 10901346
section profiling
Agent [baseline] (1.252 s) : 0, 1251947
Total [baseline] (10.845 s) : 0, 10844524
Agent [candidate] (1.266 s) : 0, 1265650
Total [candidate] (10.788 s) : 0, 10788393

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.034 s	-
Agent	appsec	1.182 s	148.21 ms (14.3%)
Agent	iast	1.172 s	138.126 ms (13.4%)
Agent	profiling	1.252 s	217.979 ms (21.1%)
Total	tracing	10.486 s	-
Total	appsec	10.763 s	277.024 ms (2.6%)
Total	iast	11.039 s	552.842 ms (5.3%)
Total	profiling	10.845 s	358.619 ms (3.4%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.041 s	-
Agent	appsec	1.185 s	144.141 ms (13.8%)
Agent	iast	1.166 s	125.463 ms (12.1%)
Agent	profiling	1.266 s	224.712 ms (21.6%)
Total	tracing	10.432 s	-
Total	appsec	10.74 s	308.208 ms (3.0%)
Total	iast	10.901 s	469.616 ms (4.5%)
Total	profiling	10.788 s	356.663 ms (3.4%)

gantt
    title petclinic - break down per module: candidate=1.46.0-SNAPSHOT~2d6d97ea75, baseline=1.46.0-SNAPSHOT~19da739a7c

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (713.153 ms) : 0, 713153
BytebuddyAgent [candidate] (718.487 ms) : 0, 718487
GlobalTracer [baseline] (238.911 ms) : 0, 238911
GlobalTracer [candidate] (240.226 ms) : 0, 240226
AppSec [baseline] (55.579 ms) : 0, 55579
AppSec [candidate] (55.626 ms) : 0, 55626
Remote Config [baseline] (712.299 µs) : 0, 712
Remote Config [candidate] (707.44 µs) : 0, 707
Telemetry [baseline] (10.689 ms) : 0, 10689
Telemetry [candidate] (10.847 ms) : 0, 10847
section appsec
BytebuddyAgent [baseline] (735.027 ms) : 0, 735027
BytebuddyAgent [candidate] (736.17 ms) : 0, 736170
GlobalTracer [baseline] (236.628 ms) : 0, 236628
GlobalTracer [candidate] (237.651 ms) : 0, 237651
AppSec [baseline] (176.957 ms) : 0, 176957
AppSec [candidate] (177.771 ms) : 0, 177771
Remote Config [baseline] (665.491 µs) : 0, 665
Remote Config [candidate] (688.449 µs) : 0, 688
Telemetry [baseline] (8.22 ms) : 0, 8220
Telemetry [candidate] (8.216 ms) : 0, 8216
IAST [baseline] (19.351 ms) : 0, 19351
IAST [candidate] (19.094 ms) : 0, 19094
section iast
BytebuddyAgent [baseline] (837.486 ms) : 0, 837486
BytebuddyAgent [candidate] (833.289 ms) : 0, 833289
GlobalTracer [baseline] (230.352 ms) : 0, 230352
GlobalTracer [candidate] (229.487 ms) : 0, 229487
AppSec [baseline] (58.258 ms) : 0, 58258
AppSec [candidate] (58.076 ms) : 0, 58076
Remote Config [baseline] (655.42 µs) : 0, 655
Remote Config [candidate] (651.765 µs) : 0, 652
Telemetry [baseline] (8.717 ms) : 0, 8717
Telemetry [candidate] (8.641 ms) : 0, 8641
IAST [baseline] (21.428 ms) : 0, 21428
IAST [candidate] (21.188 ms) : 0, 21188
section profiling
BytebuddyAgent [baseline] (702.564 ms) : 0, 702564
BytebuddyAgent [candidate] (712.12 ms) : 0, 712120
GlobalTracer [baseline] (347.851 ms) : 0, 347851
GlobalTracer [candidate] (351.411 ms) : 0, 351411
AppSec [baseline] (54.868 ms) : 0, 54868
AppSec [candidate] (54.269 ms) : 0, 54269
Remote Config [baseline] (722.07 µs) : 0, 722
Remote Config [candidate] (734.892 µs) : 0, 735
Telemetry [baseline] (8.855 ms) : 0, 8855
Telemetry [candidate] (8.824 ms) : 0, 8824
ProfilingAgent [baseline] (95.114 ms) : 0, 95114
ProfilingAgent [candidate] (95.946 ms) : 0, 95946
Profiling [baseline] (95.138 ms) : 0, 95138
Profiling [candidate] (95.97 ms) : 0, 95970

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.46.0-SNAPSHOT~2d6d97ea75, baseline=1.46.0-SNAPSHOT~19da739a7c

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.043 s) : 0, 1043480
Total [baseline] (8.653 s) : 0, 8653279
Agent [candidate] (1.035 s) : 0, 1034751
Total [candidate] (8.609 s) : 0, 8609148
section iast
Agent [baseline] (1.166 s) : 0, 1165930
Total [baseline] (9.198 s) : 0, 9197982
Agent [candidate] (1.173 s) : 0, 1172742
Total [candidate] (9.207 s) : 0, 9207302
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.164 s) : 0, 1164409
Total [baseline] (9.183 s) : 0, 9182504
Agent [candidate] (1.164 s) : 0, 1164162
Total [candidate] (9.191 s) : 0, 9190890
section iast_TELEMETRY_OFF
Agent [baseline] (1.161 s) : 0, 1160609
Total [baseline] (9.261 s) : 0, 9260974
Agent [candidate] (1.169 s) : 0, 1169276
Total [candidate] (9.231 s) : 0, 9231391

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.043 s	-
Agent	iast	1.166 s	122.45 ms (11.7%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.164 s	120.929 ms (11.6%)
Agent	iast_TELEMETRY_OFF	1.161 s	117.129 ms (11.2%)
Total	tracing	8.653 s	-
Total	iast	9.198 s	544.703 ms (6.3%)
Total	iast_HARDCODED_SECRET_DISABLED	9.183 s	529.225 ms (6.1%)
Total	iast_TELEMETRY_OFF	9.261 s	607.695 ms (7.0%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.035 s	-
Agent	iast	1.173 s	137.991 ms (13.3%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.164 s	129.41 ms (12.5%)
Agent	iast_TELEMETRY_OFF	1.169 s	134.524 ms (13.0%)
Total	tracing	8.609 s	-
Total	iast	9.207 s	598.154 ms (6.9%)
Total	iast_HARDCODED_SECRET_DISABLED	9.191 s	581.742 ms (6.8%)
Total	iast_TELEMETRY_OFF	9.231 s	622.243 ms (7.2%)

gantt
    title insecure-bank - break down per module: candidate=1.46.0-SNAPSHOT~2d6d97ea75, baseline=1.46.0-SNAPSHOT~19da739a7c

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (720.893 ms) : 0, 720893
BytebuddyAgent [candidate] (713.562 ms) : 0, 713562
GlobalTracer [baseline] (240.317 ms) : 0, 240317
GlobalTracer [candidate] (238.689 ms) : 0, 238689
AppSec [baseline] (55.629 ms) : 0, 55629
AppSec [candidate] (55.457 ms) : 0, 55457
Remote Config [baseline] (724.583 µs) : 0, 725
Remote Config [candidate] (713.837 µs) : 0, 714
Telemetry [baseline] (10.856 ms) : 0, 10856
Telemetry [candidate] (11.378 ms) : 0, 11378
section iast
BytebuddyAgent [baseline] (832.58 ms) : 0, 832580
BytebuddyAgent [candidate] (837.702 ms) : 0, 837702
GlobalTracer [baseline] (229.4 ms) : 0, 229400
GlobalTracer [candidate] (231.076 ms) : 0, 231076
AppSec [baseline] (57.474 ms) : 0, 57474
AppSec [candidate] (58.147 ms) : 0, 58147
IAST [baseline] (22.157 ms) : 0, 22157
IAST [candidate] (21.437 ms) : 0, 21437
Remote Config [baseline] (664.652 µs) : 0, 665
Remote Config [candidate] (671.59 µs) : 0, 672
Telemetry [baseline] (8.66 ms) : 0, 8660
Telemetry [candidate] (8.673 ms) : 0, 8673
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (831.853 ms) : 0, 831853
BytebuddyAgent [candidate] (831.26 ms) : 0, 831260
GlobalTracer [baseline] (229.199 ms) : 0, 229199
GlobalTracer [candidate] (228.93 ms) : 0, 228930
AppSec [baseline] (57.813 ms) : 0, 57813
AppSec [candidate] (58.069 ms) : 0, 58069
IAST [baseline] (21.277 ms) : 0, 21277
IAST [candidate] (21.36 ms) : 0, 21360
Remote Config [baseline] (663.173 µs) : 0, 663
Remote Config [candidate] (673.051 µs) : 0, 673
Telemetry [baseline] (8.639 ms) : 0, 8639
Telemetry [candidate] (8.79 ms) : 0, 8790
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (827.4 ms) : 0, 827400
BytebuddyAgent [candidate] (834.804 ms) : 0, 834804
GlobalTracer [baseline] (229.366 ms) : 0, 229366
GlobalTracer [candidate] (230.651 ms) : 0, 230651
AppSec [baseline] (58.464 ms) : 0, 58464
AppSec [candidate] (58.217 ms) : 0, 58217
IAST [baseline] (20.962 ms) : 0, 20962
IAST [candidate] (21.172 ms) : 0, 21172
Remote Config [baseline] (649.434 µs) : 0, 649
Remote Config [candidate] (661.116 µs) : 0, 661
Telemetry [baseline] (8.735 ms) : 0, 8735
Telemetry [candidate] (8.672 ms) : 0, 8672

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
end_time	2025-01-27T12:15:00	2025-01-27T12:22:01
git_branch	master	nikita-tkachenko/itr-skip-refactor
git_commit_date	1737969316	1737979436
git_commit_sha	`19da739`	`2d6d97e`
release_version	1.46.0-SNAPSHOT~19da739a7c	1.46.0-SNAPSHOT~2d6d97ea75
start_time	2025-01-27T12:14:46	2025-01-27T12:21:47

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1737980876	1737980876
ci_job_id	781287831	781287831
ci_pipeline_id	54045236	54045236
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-fgow3nxt-project-304-concurrent-0-urmxw9n4 6.8.0-1018-aws #19~22.04.1-Ubuntu SMP Wed Oct 9 16:48:22 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-fgow3nxt-project-304-concurrent-0-urmxw9n4 6.8.0-1018-aws #19~22.04.1-Ubuntu SMP Wed Oct 9 16:48:22 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 9 metrics, 19 unstable metrics.

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.46.0-SNAPSHOT~2d6d97ea75, baseline=1.46.0-SNAPSHOT~19da739a7c
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.348 ms) : 1327, 1368
.   : milestone, 1348,
appsec (1.734 ms) : 1710, 1757
.   : milestone, 1734,
appsec_no_iast (1.737 ms) : 1713, 1760
.   : milestone, 1737,
iast (1.507 ms) : 1483, 1532
.   : milestone, 1507,
profiling (1.492 ms) : 1469, 1516
.   : milestone, 1492,
tracing (1.48 ms) : 1457, 1504
.   : milestone, 1480,
section candidate
no_agent (1.338 ms) : 1318, 1358
.   : milestone, 1338,
appsec (1.742 ms) : 1719, 1765
.   : milestone, 1742,
appsec_no_iast (1.74 ms) : 1715, 1764
.   : milestone, 1740,
iast (1.496 ms) : 1472, 1520
.   : milestone, 1496,
profiling (1.495 ms) : 1472, 1518
.   : milestone, 1495,
tracing (1.485 ms) : 1462, 1509
.   : milestone, 1485,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.348 ms [1.327 ms, 1.368 ms]	-
appsec	1.734 ms [1.71 ms, 1.757 ms]	386.383 µs (28.7%)
appsec_no_iast	1.737 ms [1.713 ms, 1.76 ms]	389.318 µs (28.9%)
iast	1.507 ms [1.483 ms, 1.532 ms]	159.985 µs (11.9%)
profiling	1.492 ms [1.469 ms, 1.516 ms]	144.612 µs (10.7%)
tracing	1.48 ms [1.457 ms, 1.504 ms]	132.721 µs (9.8%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.338 ms [1.318 ms, 1.358 ms]	-
appsec	1.742 ms [1.719 ms, 1.765 ms]	403.865 µs (30.2%)
appsec_no_iast	1.74 ms [1.715 ms, 1.764 ms]	401.726 µs (30.0%)
iast	1.496 ms [1.472 ms, 1.52 ms]	158.337 µs (11.8%)
profiling	1.495 ms [1.472 ms, 1.518 ms]	156.797 µs (11.7%)
tracing	1.485 ms [1.462 ms, 1.509 ms]	147.625 µs (11.0%)

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.46.0-SNAPSHOT~2d6d97ea75, baseline=1.46.0-SNAPSHOT~19da739a7c
    dateFormat X
    axisFormat %s
section baseline
no_agent (366.919 µs) : 347, 387
.   : milestone, 367,
iast (511.298 µs) : 487, 535
.   : milestone, 511,
iast_FULL (736.226 µs) : 714, 758
.   : milestone, 736,
iast_GLOBAL (546.185 µs) : 525, 568
.   : milestone, 546,
iast_HARDCODED_SECRET_DISABLED (505.303 µs) : 482, 528
.   : milestone, 505,
iast_INACTIVE (456.811 µs) : 435, 478
.   : milestone, 457,
iast_TELEMETRY_OFF (495.642 µs) : 472, 519
.   : milestone, 496,
tracing (452.248 µs) : 431, 474
.   : milestone, 452,
section candidate
no_agent (377.331 µs) : 358, 397
.   : milestone, 377,
iast (506.889 µs) : 483, 531
.   : milestone, 507,
iast_FULL (742.79 µs) : 720, 766
.   : milestone, 743,
iast_GLOBAL (547.929 µs) : 525, 571
.   : milestone, 548,
iast_HARDCODED_SECRET_DISABLED (501.167 µs) : 478, 524
.   : milestone, 501,
iast_INACTIVE (450.507 µs) : 429, 472
.   : milestone, 451,
iast_TELEMETRY_OFF (488.513 µs) : 465, 512
.   : milestone, 489,
tracing (445.298 µs) : 424, 466
.   : milestone, 445,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	366.919 µs [347.229 µs, 386.609 µs]	-
iast	511.298 µs [487.154 µs, 535.442 µs]	144.379 µs (39.3%)
iast_FULL	736.226 µs [714.197 µs, 758.254 µs]	369.307 µs (100.7%)
iast_GLOBAL	546.185 µs [524.744 µs, 567.627 µs]	179.266 µs (48.9%)
iast_HARDCODED_SECRET_DISABLED	505.303 µs [482.452 µs, 528.154 µs]	138.384 µs (37.7%)
iast_INACTIVE	456.811 µs [435.291 µs, 478.332 µs]	89.893 µs (24.5%)
iast_TELEMETRY_OFF	495.642 µs [471.879 µs, 519.405 µs]	128.723 µs (35.1%)
tracing	452.248 µs [430.871 µs, 473.625 µs]	85.329 µs (23.3%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	377.331 µs [357.649 µs, 397.013 µs]	-
iast	506.889 µs [483.151 µs, 530.627 µs]	129.558 µs (34.3%)
iast_FULL	742.79 µs [719.944 µs, 765.637 µs]	365.459 µs (96.9%)
iast_GLOBAL	547.929 µs [524.539 µs, 571.318 µs]	170.598 µs (45.2%)
iast_HARDCODED_SECRET_DISABLED	501.167 µs [478.065 µs, 524.269 µs]	123.836 µs (32.8%)
iast_INACTIVE	450.507 µs [428.889 µs, 472.124 µs]	73.176 µs (19.4%)
iast_TELEMETRY_OFF	488.513 µs [465.119 µs, 511.906 µs]	111.182 µs (29.5%)
tracing	445.298 µs [424.446 µs, 466.151 µs]	67.967 µs (18.0%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	nikita-tkachenko/itr-skip-refactor
git_commit_date	1737969316	1737979436
git_commit_sha	`19da739`	`2d6d97e`
release_version	1.46.0-SNAPSHOT~19da739a7c	1.46.0-SNAPSHOT~2d6d97ea75

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1737981688	1737981688
ci_job_id	781287832	781287832
ci_pipeline_id	54045236	54045236
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-8wlcx2s5-project-304-concurrent-1-en00rkye 6.8.0-1018-aws #19~22.04.1-Ubuntu SMP Wed Oct 9 16:48:22 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-8wlcx2s5-project-304-concurrent-1-en00rkye 6.8.0-1018-aws #19~22.04.1-Ubuntu SMP Wed Oct 9 16:48:22 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
variant	appsec	appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.46.0-SNAPSHOT~2d6d97ea75, baseline=1.46.0-SNAPSHOT~19da739a7c
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.249 s) : 15249000, 15249000
.   : milestone, 15249000,
appsec (15.054 s) : 15054000, 15054000
.   : milestone, 15054000,
iast (18.859 s) : 18859000, 18859000
.   : milestone, 18859000,
iast_GLOBAL (18.065 s) : 18065000, 18065000
.   : milestone, 18065000,
profiling (15.657 s) : 15657000, 15657000
.   : milestone, 15657000,
tracing (14.782 s) : 14782000, 14782000
.   : milestone, 14782000,
section candidate
no_agent (15.597 s) : 15597000, 15597000
.   : milestone, 15597000,
appsec (15.071 s) : 15071000, 15071000
.   : milestone, 15071000,
iast (18.556 s) : 18556000, 18556000
.   : milestone, 18556000,
iast_GLOBAL (18.182 s) : 18182000, 18182000
.   : milestone, 18182000,
profiling (15.747 s) : 15747000, 15747000
.   : milestone, 15747000,
tracing (15.131 s) : 15131000, 15131000
.   : milestone, 15131000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.249 s [15.249 s, 15.249 s]	-
appsec	15.054 s [15.054 s, 15.054 s]	-195.0 ms (-1.3%)
iast	18.859 s [18.859 s, 18.859 s]	3.61 s (23.7%)
iast_GLOBAL	18.065 s [18.065 s, 18.065 s]	2.816 s (18.5%)
profiling	15.657 s [15.657 s, 15.657 s]	408.0 ms (2.7%)
tracing	14.782 s [14.782 s, 14.782 s]	-467.0 ms (-3.1%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.597 s [15.597 s, 15.597 s]	-
appsec	15.071 s [15.071 s, 15.071 s]	-526.0 ms (-3.4%)
iast	18.556 s [18.556 s, 18.556 s]	2.959 s (19.0%)
iast_GLOBAL	18.182 s [18.182 s, 18.182 s]	2.585 s (16.6%)
profiling	15.747 s [15.747 s, 15.747 s]	150.0 ms (1.0%)
tracing	15.131 s [15.131 s, 15.131 s]	-466.0 ms (-3.0%)

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.46.0-SNAPSHOT~2d6d97ea75, baseline=1.46.0-SNAPSHOT~19da739a7c
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.465 ms) : 1454, 1477
.   : milestone, 1465,
appsec (2.358 ms) : 2315, 2401
.   : milestone, 2358,
iast (2.093 ms) : 2039, 2147
.   : milestone, 2093,
iast_GLOBAL (2.132 ms) : 2077, 2186
.   : milestone, 2132,
profiling (1.948 ms) : 1905, 1991
.   : milestone, 1948,
tracing (1.928 ms) : 1886, 1969
.   : milestone, 1928,
section candidate
no_agent (1.466 ms) : 1454, 1477
.   : milestone, 1466,
appsec (2.343 ms) : 2300, 2386
.   : milestone, 2343,
iast (2.092 ms) : 2038, 2146
.   : milestone, 2092,
iast_GLOBAL (2.14 ms) : 2085, 2195
.   : milestone, 2140,
profiling (1.962 ms) : 1918, 2006
.   : milestone, 1962,
tracing (1.935 ms) : 1893, 1977
.   : milestone, 1935,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.465 ms [1.454 ms, 1.477 ms]	-
appsec	2.358 ms [2.315 ms, 2.401 ms]	892.865 µs (60.9%)
iast	2.093 ms [2.039 ms, 2.147 ms]	627.825 µs (42.8%)
iast_GLOBAL	2.132 ms [2.077 ms, 2.186 ms]	666.779 µs (45.5%)
profiling	1.948 ms [1.905 ms, 1.991 ms]	482.563 µs (32.9%)
tracing	1.928 ms [1.886 ms, 1.969 ms]	462.493 µs (31.6%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.466 ms [1.454 ms, 1.477 ms]	-
appsec	2.343 ms [2.3 ms, 2.386 ms]	877.336 µs (59.9%)
iast	2.092 ms [2.038 ms, 2.146 ms]	626.653 µs (42.8%)
iast_GLOBAL	2.14 ms [2.085 ms, 2.195 ms]	674.223 µs (46.0%)
profiling	1.962 ms [1.918 ms, 2.006 ms]	496.213 µs (33.9%)
tracing	1.935 ms [1.893 ms, 1.977 ms]	469.419 µs (32.0%)

| Package | Type | Package file | Manager | Update | Change | |---|---|---|---|---|---| | [com.google.cloud:google-cloud-datastore](https://github.com/googleapis/java-datastore) | dependencies | misk/gradle/libs.versions.toml | gradle | minor | `2.25.4` -> `2.26.0` | | [com.datadoghq:dd-trace-api](https://github.com/datadog/dd-trace-java) | dependencies | misk/gradle/libs.versions.toml | gradle | minor | `1.45.2` -> `1.46.0` | | [com.datadoghq:dd-trace-ot](https://github.com/datadog/dd-trace-java) | dependencies | misk/gradle/libs.versions.toml | gradle | minor | `1.45.2` -> `1.46.0` | | [software.amazon.awssdk:sdk-core](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.30.9` -> `2.30.10` | | [software.amazon.awssdk:dynamodb-enhanced](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.30.9` -> `2.30.10` | | [software.amazon.awssdk:dynamodb](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.30.9` -> `2.30.10` | | [software.amazon.awssdk:aws-core](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.30.9` -> `2.30.10` | | [software.amazon.awssdk:bom](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.30.9` -> `2.30.10` | | [software.amazon.awssdk:auth](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.30.9` -> `2.30.10` | --- ### Release Notes <details> <summary>googleapis/java-datastore (com.google.cloud:google-cloud-datastore)</summary> ### [`v2.26.0`](https://github.com/googleapis/java-datastore/blob/HEAD/CHANGELOG.md#2260-2025-01-29) ##### Features - Add firestoreInDatastoreMode for datastore emulator ([#1698](googleapis/java-datastore#1698)) ([50f106d](googleapis/java-datastore@50f106d)) ##### Dependencies - Update dependency com.google.cloud:sdk-platform-java-config to v3.42.0 ([#1725](googleapis/java-datastore#1725)) ([1cbaf22](googleapis/java-datastore@1cbaf22)) </details> <details> <summary>datadog/dd-trace-java (com.datadoghq:dd-trace-api)</summary> ### [`v1.46.0`](https://github.com/DataDog/dd-trace-java/releases/tag/v1.46.0): 1.46.0 ##### Breaking Changes > \[!WARNING] > jnr-unixsocket is now an external dependency of dd-trace-ot and must be included when deploying dd-trace-ot. > \[!NOTE] > The API `TracerScope.setAsync(boolean)`, used to manually control asynchronous span propagation, does no more apply to the scope instance but to the active span scope. ##### Components ##### Application Security Management (IAST) - 🐛 Fix String.replace instrumentation for IAST ([#8281](DataDog/dd-trace-java#8281) - [@Mariovido](https://github.com/Mariovido)) - ✨ Apply the standard nomenclature to the stacktrace configs ([#8244](DataDog/dd-trace-java#8244) - [@jandro996](https://github.com/jandro996)) - 🐛 Exclude false positive weak randomness ([#8232](DataDog/dd-trace-java#8232) - [@jandro996](https://github.com/jandro996)) - ✨ Propagation of translateEscapes of String class ([#8186](DataDog/dd-trace-java#8186) - [@sezen-datadog](https://github.com/sezen-datadog)) - ✨ Add security control metrics ([#8175](DataDog/dd-trace-java#8175) - [@jandro996](https://github.com/jandro996)) - ✨ Increase IAST propagation to StringBuffer setLength ([#8128](DataDog/dd-trace-java#8128) - [@Mariovido](https://github.com/Mariovido)) - ✨ Add IAST taint tracking for DB values ([#8072](DataDog/dd-trace-java#8072) - [@Mariovido](https://github.com/Mariovido)) ##### Application Security Management (WAF) - 🐛 Prevents a NPE when there is no subscriber for user events ([#8258](DataDog/dd-trace-java#8258) - [@manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) - ✨ Apply the standard nomenclature to the stacktrace configs ([#8244](DataDog/dd-trace-java#8244) - [@jandro996](https://github.com/jandro996)) - 🐛 Ensure cached subscriptions are cleared on reconfiguration via RC ([#8229](DataDog/dd-trace-java#8229) - [@manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) - ✨ Add support for session tracking in Vertx ([#8167](DataDog/dd-trace-java#8167) - [@manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) - ✨ Create span tag: \_dd.appsec.rasp.timeout ([#8269](DataDog/dd-trace-java#8269) - [@Mariovido](https://github.com/Mariovido)) ##### Build & Tooling - 🐛 Ensure shaded helpers have unique names when injected into class-loaders ([#8192](DataDog/dd-trace-java#8192) - [@mcculls](https://github.com/mcculls)) ##### Configuration at Runtime - 🐛 Remove filtering of `DD_SERVICE` and `DD_ENV` from the tracer ([#8176](DataDog/dd-trace-java#8176) - [@mhlidd](https://github.com/mhlidd)) ##### Continuous Integration Visibility - 🧹 Generalize TestRetryPolicy to TestExecutionPolicy ([#8302](DataDog/dd-trace-java#8302) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - 🧹 Parallelize CI Visibility settings requests ([#8299](DataDog/dd-trace-java#8299) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - 🧹 Generalize test retry logic ([#8289](DataDog/dd-trace-java#8289) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - 🧹 Generalize tests skipping logic ([#8288](DataDog/dd-trace-java#8288) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - 🧹 Remove skip and shouldBeSkipped methods from TestEventsHandler in favor of isSkippable ([#8286](DataDog/dd-trace-java#8286) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - ✨⚡ Optimize Git repository information computation ([#8270](DataDog/dd-trace-java#8270) - [@dougqh](https://github.com/dougqh)) - ✨ Always request known tests from the backend ([#8268](DataDog/dd-trace-java#8268) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - ✨ Fix NPE when trying to get retry analyzer in Test NG ([#8253](DataDog/dd-trace-java#8253) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - 🐛 Set test framework and test framework version tags atomically ([#8252](DataDog/dd-trace-java#8252) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - ✨ Add debug logging to Android Gradle module layout logic ([#8251](DataDog/dd-trace-java#8251) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - 🐛 Fix source and destination folders computation for Android Gradle projects ([#8190](DataDog/dd-trace-java#8190) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - ✨ Add basic Scala Weaver sbt support ([#8189](DataDog/dd-trace-java#8189) - [@daniel-mohedano](https://github.com/daniel-mohedano)) - ✨ Implement impacted tests detection ([#8188](DataDog/dd-trace-java#8188) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) ##### Data Streams Monitoring - ✨ Change hash computation for protobuf to better represent impacting changes + save proto number in schema ([#8201](DataDog/dd-trace-java#8201) - [@vandonr](https://github.com/vandonr)) ##### Database Monitoring - Add peer service tag in dbm sql commenter ([#7913](DataDog/dd-trace-java#7913) - [@jordan-wong](https://github.com/jordan-wong)) ##### Dynamic Instrumentation - ✨ Add support for SymDB to scan directories ([#8306](DataDog/dd-trace-java#8306) - [@jpbempel](https://github.com/jpbempel)) - ✨ Add SymDB report for any jar scanning failures ([#8300](DataDog/dd-trace-java#8300) - [@jpbempel](https://github.com/jpbempel)) - ✨ Use two budgets depending on type ([#8283](DataDog/dd-trace-java#8283) - [@evanchooly](https://github.com/evanchooly)) - ✨ Institute a 10 snapshot per probe per trace budget ([#8277](DataDog/dd-trace-java#8277) - [@evanchooly](https://github.com/evanchooly)) - 🐛 Avoid double snapshots for Exception Replay ([#8273](DataDog/dd-trace-java#8273) - [@jpbempel](https://github.com/jpbempel)) - ✨ Simplify code origins. Separate out snapshot generation. ([#8263](DataDog/dd-trace-java#8263) - [@evanchooly](https://github.com/evanchooly)) - ✨ Add Exception probe custom instrumentation ([#8230](DataDog/dd-trace-java#8230) - [@jpbempel](https://github.com/jpbempel)) - ✨ Enhance log probes to honor debug session tags ([#8215](DataDog/dd-trace-java#8215) - [@evanchooly](https://github.com/evanchooly)) - 🐛 Don't redact env tokens from debugger probe snapshots ([#8211](DataDog/dd-trace-java#8211) - [@watson](https://github.com/watson)) - ✨⚡ Move Trace/SpanId capture at commit time ([#8184](DataDog/dd-trace-java#8184) - [@jpbempel](https://github.com/jpbempel)) - 🐛 Capture values at entry for method probe ([#8169](DataDog/dd-trace-java#8169) - [@jpbempel](https://github.com/jpbempel)) ##### JMX fetch - 🐛 Mute JMXFetch Shutdown in progress error ([#8068](DataDog/dd-trace-java#8068) - [@ygree](https://github.com/ygree)) ##### OpenTracing - ⚠️🧹 Make jnr-unixsocket an explicit dependency of dd-trace-ot ([#8307](DataDog/dd-trace-java#8307) - [@mcculls](https://github.com/mcculls)) ##### Profiling - 🐛 Avoid unsupported API call for creating folders on windows ([#8304](DataDog/dd-trace-java#8304) - [@jbachorik](https://github.com/jbachorik)) - ✨ Tag profiles for serverless ([#8279](DataDog/dd-trace-java#8279) - [@jbachorik](https://github.com/jbachorik)) - ✨ add queue type and length to queue events ([#8242](DataDog/dd-trace-java#8242) - [@richardstartin](https://github.com/richardstartin)) - 🐛 TempLocationManager Fixes and Improvements ([#8191](DataDog/dd-trace-java#8191) - [@jbachorik](https://github.com/jbachorik)) - ✨ Bump ddprof to 1.18.0 ([#8173](DataDog/dd-trace-java#8173) - [@jbachorik](https://github.com/jbachorik)) - ✨ Report profiler initialization and configuration errors to telemetry ([#8171](DataDog/dd-trace-java#8171) - [@jbachorik](https://github.com/jbachorik)) ##### Telemetry - ✨ Add pending traces report in tracer flares ([#8053](DataDog/dd-trace-java#8053) - [@mhlidd](https://github.com/mhlidd)) ##### Testing - ✨ Test http server requests in parallel ([#8222](DataDog/dd-trace-java#8222) - [@amarziali](https://github.com/amarziali)) ##### Trace context propagation - ✨ Add non default propagator registration ([#8310](DataDog/dd-trace-java#8310) - [@PerfectSlayer](https://github.com/PerfectSlayer)) ##### Tracer core - ✨ Probe for existence of IBMSASL or ACCP security providers ([#8276](DataDog/dd-trace-java#8276) - [@mcculls](https://github.com/mcculls)) - ✨⚡ Overhead improvement to agent feedback based sampling ([#8265](DataDog/dd-trace-java#8265) - [@dougqh](https://github.com/dougqh)) - 🧹 Move async propagation API from scope to tracer ([#8231](DataDog/dd-trace-java#8231) - [@PerfectSlayer](https://github.com/PerfectSlayer)) - ✨ Introduce context propagation API ([#8161](DataDog/dd-trace-java#8161) - [@PerfectSlayer](https://github.com/PerfectSlayer)) - ✨🧪 Use env-entry to add tags per webapp deployment ([#8138](DataDog/dd-trace-java#8138) - [@amarziali](https://github.com/amarziali)) - ✨ Introduce context helpers API ([#8134](DataDog/dd-trace-java#8134) - [@PerfectSlayer](https://github.com/PerfectSlayer)) - ✨ Support IPv6 values for `DD_AGENT_HOST` and `DD_TRACE_AGENT_URL` ([#7984](DataDog/dd-trace-java#7984) - [@mhlidd](https://github.com/mhlidd)) ##### Instrumentations ##### Apache HttpComponents - 🐛 Properly finish spans and support latest apache httpclient5 ([#8272](DataDog/dd-trace-java#8272) - [@amarziali](https://github.com/amarziali)) ##### AWS Lambda instrumentation - 🐛 Properly capture lambda payloads for all handler types. ([#8264](DataDog/dd-trace-java#8264) - [@purple4reina](https://github.com/purple4reina)) ##### AWS S3 instrumentation - 💡 Create S3 instrumentation + add span pointers ([#8075](DataDog/dd-trace-java#8075) - [@nhulston](https://github.com/nhulston)) ##### AWS SDK instrumentation - 🐛 Revert "Add avoid double instrumenting lambda non-streaming handlers." ([#8247](DataDog/dd-trace-java#8247) - [@nhulston](https://github.com/nhulston)) ##### Cassandra - ✨ Allow extracting keyspace from statement result ([#8239](DataDog/dd-trace-java#8239) - [@amarziali](https://github.com/amarziali)) ##### Core Java language instrumentation - ✨ Propagation of translateEscapes of String class ([#8186](DataDog/dd-trace-java#8186) - [@sezen-datadog](https://github.com/sezen-datadog)) ##### Eclipse Vert.x instrumentation - 🐛 Fix vertx worker propagation and error handling ([#8237](DataDog/dd-trace-java#8237) - [@amarziali](https://github.com/amarziali)) - ✨ Support vertx 5 ([#8220](DataDog/dd-trace-java#8220) - [@amarziali](https://github.com/amarziali)) - ✨ Add support for session tracking in Vertx ([#8167](DataDog/dd-trace-java#8167) - [@manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) ##### Kafka instrumentation - 🐛 Prevent possible NPE calculating Kafka record header size ([#8292](DataDog/dd-trace-java#8292) - [@ygree](https://github.com/ygree)) ##### Mule instrumentation - 🐛 Fix crash using Mule with JPMS ([#8187](DataDog/dd-trace-java#8187) - [@amarziali](https://github.com/amarziali)) ##### Protocol Buffer instrumentation - ✨ Change hash computation for protobuf to better represent impacting changes + save proto number in schema ([#8201](DataDog/dd-trace-java#8201) - [@vandonr](https://github.com/vandonr)) ##### Spring instrumentation - 🐛 Preserve getQualifier from spring scheduling runnables ([#8293](DataDog/dd-trace-java#8293) - [@amarziali](https://github.com/amarziali)) </details> --- ### Configuration 📅 **Schedule**: Branch creation - "after 6pm every weekday,before 2am every weekday" in timezone Australia/Melbourne, Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 👻 **Immortal**: This PR will be recreated if closed unmerged. Get [config help](https://github.com/renovatebot/renovate/discussions) if that's undesired. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). GitOrigin-RevId: bb09d47e4eed77a003f630273b4d0a84003eb899

Remove skip and shouldBeSkipped methods from TestEventsHandler in fav…

2d6d97e

…or of isSkippable

nikita-tkachenko-datadog added type: refactoring comp: ci visibility Continuous Integration Visibility labels Jan 27, 2025

nikita-tkachenko-datadog marked this pull request as ready for review January 27, 2025 13:16

nikita-tkachenko-datadog requested a review from a team as a code owner January 27, 2025 13:16

nikita-tkachenko-datadog requested a review from juan-fernandez January 27, 2025 13:16

daniel-mohedano approved these changes Jan 27, 2025

View reviewed changes

nikita-tkachenko-datadog merged commit a85ec5f into master Jan 27, 2025
179 of 180 checks passed

nikita-tkachenko-datadog deleted the nikita-tkachenko/itr-skip-refactor branch January 27, 2025 13:38

github-actions bot added this to the 1.46.0 milestone Jan 27, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove skip and shouldBeSkipped methods from TestEventsHandler in favor of isSkippable #8286

Remove skip and shouldBeSkipped methods from TestEventsHandler in favor of isSkippable #8286

nikita-tkachenko-datadog commented Jan 27, 2025 •

edited by jira bot

Loading

pr-commenter bot commented Jan 27, 2025

Remove skip and shouldBeSkipped methods from TestEventsHandler in favor of isSkippable #8286

Remove skip and shouldBeSkipped methods from TestEventsHandler in favor of isSkippable #8286

Conversation

nikita-tkachenko-datadog commented Jan 27, 2025 • edited by jira bot Loading

What Does This Do

Motivation

Contributor Checklist

pr-commenter bot commented Jan 27, 2025

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

nikita-tkachenko-datadog commented Jan 27, 2025 •

edited by jira bot

Loading