Generalize test retry logic #8289

nikita-tkachenko-datadog · 2025-01-27T17:05:15Z

What Does This Do

Does some updates to TestRetryPolicy and the code that uses it:

retry reason is now an enum instead of a String
the contract of suppressFailures() method is changed to return whether failures should be suppressed for the current execution
currentExecutionIsRetry() method is removed

Motivation

Preparing for future flaky tests management changes: quarantined tests will have to be executed without affecting the status of the build.

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any usefull labels
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the public documentation in case of new configuration flag or behavior

Jira ticket: SDTEST-1483

pr-commenter · 2025-01-27T17:49:39Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	nikita-tkachenko/generalize-test-retries
git_commit_date	1738137590	1738143458
git_commit_sha	`f06d59c`	`5f23339`
release_version	1.46.0-SNAPSHOT~f06d59c0dd	1.46.0-SNAPSHOT~5f23339385

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1738145929	1738145929
ci_job_id	784984256	784984256
ci_pipeline_id	54289730	54289730
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-qirit7hx-project-304-concurrent-0-yybczn9i 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-qirit7hx-project-304-concurrent-0-yybczn9i 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 51 metrics, 12 unstable metrics.

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.46.0-SNAPSHOT~5f23339385, baseline=1.46.0-SNAPSHOT~f06d59c0dd

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.045 s) : 0, 1044758
Total [baseline] (10.469 s) : 0, 10468729
Agent [candidate] (1.043 s) : 0, 1042987
Total [candidate] (10.43 s) : 0, 10430332
section appsec
Agent [baseline] (1.184 s) : 0, 1183867
Total [baseline] (10.775 s) : 0, 10774678
Agent [candidate] (1.186 s) : 0, 1186078
Total [candidate] (10.796 s) : 0, 10796442
section iast
Agent [baseline] (1.172 s) : 0, 1172343
Total [baseline] (10.968 s) : 0, 10968202
Agent [candidate] (1.172 s) : 0, 1172489
Total [candidate] (10.925 s) : 0, 10924630
section profiling
Agent [baseline] (1.265 s) : 0, 1265248
Total [baseline] (10.877 s) : 0, 10877376
Agent [candidate] (1.267 s) : 0, 1267410
Total [candidate] (10.954 s) : 0, 10954253

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.045 s	-
Agent	appsec	1.184 s	139.109 ms (13.3%)
Agent	iast	1.172 s	127.584 ms (12.2%)
Agent	profiling	1.265 s	220.49 ms (21.1%)
Total	tracing	10.469 s	-
Total	appsec	10.775 s	305.949 ms (2.9%)
Total	iast	10.968 s	499.473 ms (4.8%)
Total	profiling	10.877 s	408.647 ms (3.9%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.043 s	-
Agent	appsec	1.186 s	143.091 ms (13.7%)
Agent	iast	1.172 s	129.503 ms (12.4%)
Agent	profiling	1.267 s	224.423 ms (21.5%)
Total	tracing	10.43 s	-
Total	appsec	10.796 s	366.111 ms (3.5%)
Total	iast	10.925 s	494.299 ms (4.7%)
Total	profiling	10.954 s	523.921 ms (5.0%)

gantt
    title petclinic - break down per module: candidate=1.46.0-SNAPSHOT~5f23339385, baseline=1.46.0-SNAPSHOT~f06d59c0dd

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (719.31 ms) : 0, 719310
BytebuddyAgent [candidate] (716.64 ms) : 0, 716640
GlobalTracer [baseline] (242.099 ms) : 0, 242099
GlobalTracer [candidate] (240.673 ms) : 0, 240673
AppSec [baseline] (55.851 ms) : 0, 55851
AppSec [candidate] (56.219 ms) : 0, 56219
Remote Config [baseline] (719.823 µs) : 0, 720
Remote Config [candidate] (709.007 µs) : 0, 709
Telemetry [baseline] (11.525 ms) : 0, 11525
Telemetry [candidate] (13.48 ms) : 0, 13480
section appsec
BytebuddyAgent [baseline] (734.64 ms) : 0, 734640
BytebuddyAgent [candidate] (735.249 ms) : 0, 735249
GlobalTracer [baseline] (237.684 ms) : 0, 237684
GlobalTracer [candidate] (238.646 ms) : 0, 238646
AppSec [baseline] (176.593 ms) : 0, 176593
AppSec [candidate] (176.985 ms) : 0, 176985
Remote Config [baseline] (651.915 µs) : 0, 652
Remote Config [candidate] (668.067 µs) : 0, 668
Telemetry [baseline] (8.285 ms) : 0, 8285
Telemetry [candidate] (8.352 ms) : 0, 8352
IAST [baseline] (21.34 ms) : 0, 21340
IAST [candidate] (21.653 ms) : 0, 21653
section iast
BytebuddyAgent [baseline] (836.032 ms) : 0, 836032
BytebuddyAgent [candidate] (836.486 ms) : 0, 836486
GlobalTracer [baseline] (231.685 ms) : 0, 231685
GlobalTracer [candidate] (231.639 ms) : 0, 231639
AppSec [baseline] (54.014 ms) : 0, 54014
AppSec [candidate] (55.543 ms) : 0, 55543
Remote Config [baseline] (618.356 µs) : 0, 618
Remote Config [candidate] (619.137 µs) : 0, 619
Telemetry [baseline] (8.852 ms) : 0, 8852
Telemetry [candidate] (8.773 ms) : 0, 8773
IAST [baseline] (25.798 ms) : 0, 25798
IAST [candidate] (24.056 ms) : 0, 24056
section profiling
BytebuddyAgent [baseline] (709.959 ms) : 0, 709959
BytebuddyAgent [candidate] (710.521 ms) : 0, 710521
GlobalTracer [baseline] (351.181 ms) : 0, 351181
GlobalTracer [candidate] (353.893 ms) : 0, 353893
AppSec [baseline] (55.488 ms) : 0, 55488
AppSec [candidate] (54.397 ms) : 0, 54397
Remote Config [baseline] (712.904 µs) : 0, 713
Remote Config [candidate] (733.68 µs) : 0, 734
Telemetry [baseline] (8.945 ms) : 0, 8945
Telemetry [candidate] (8.942 ms) : 0, 8942
ProfilingAgent [baseline] (96.368 ms) : 0, 96368
ProfilingAgent [candidate] (96.262 ms) : 0, 96262
Profiling [baseline] (96.392 ms) : 0, 96392
Profiling [candidate] (96.287 ms) : 0, 96287

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.46.0-SNAPSHOT~5f23339385, baseline=1.46.0-SNAPSHOT~f06d59c0dd

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.04 s) : 0, 1039995
Total [baseline] (8.67 s) : 0, 8670116
Agent [candidate] (1.039 s) : 0, 1038844
Total [candidate] (8.64 s) : 0, 8639798
section iast
Agent [baseline] (1.167 s) : 0, 1167167
Total [baseline] (9.231 s) : 0, 9230870
Agent [candidate] (1.178 s) : 0, 1177766
Total [candidate] (9.254 s) : 0, 9253990
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.17 s) : 0, 1169657
Total [baseline] (9.199 s) : 0, 9198888
Agent [candidate] (1.17 s) : 0, 1169691
Total [candidate] (9.23 s) : 0, 9230205
section iast_TELEMETRY_OFF
Agent [baseline] (1.164 s) : 0, 1164489
Total [baseline] (9.255 s) : 0, 9255253
Agent [candidate] (1.192 s) : 0, 1192356
Total [candidate] (9.325 s) : 0, 9325096

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.04 s	-
Agent	iast	1.167 s	127.172 ms (12.2%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.17 s	129.662 ms (12.5%)
Agent	iast_TELEMETRY_OFF	1.164 s	124.494 ms (12.0%)
Total	tracing	8.67 s	-
Total	iast	9.231 s	560.754 ms (6.5%)
Total	iast_HARDCODED_SECRET_DISABLED	9.199 s	528.772 ms (6.1%)
Total	iast_TELEMETRY_OFF	9.255 s	585.137 ms (6.7%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.039 s	-
Agent	iast	1.178 s	138.922 ms (13.4%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.17 s	130.847 ms (12.6%)
Agent	iast_TELEMETRY_OFF	1.192 s	153.512 ms (14.8%)
Total	tracing	8.64 s	-
Total	iast	9.254 s	614.192 ms (7.1%)
Total	iast_HARDCODED_SECRET_DISABLED	9.23 s	590.406 ms (6.8%)
Total	iast_TELEMETRY_OFF	9.325 s	685.298 ms (7.9%)

gantt
    title insecure-bank - break down per module: candidate=1.46.0-SNAPSHOT~5f23339385, baseline=1.46.0-SNAPSHOT~f06d59c0dd

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (715.186 ms) : 0, 715186
BytebuddyAgent [candidate] (714.238 ms) : 0, 714238
GlobalTracer [baseline] (240.522 ms) : 0, 240522
GlobalTracer [candidate] (239.475 ms) : 0, 239475
AppSec [baseline] (56.705 ms) : 0, 56705
AppSec [candidate] (56.278 ms) : 0, 56278
Remote Config [baseline] (729.239 µs) : 0, 729
Remote Config [candidate] (713.706 µs) : 0, 714
Telemetry [baseline] (11.641 ms) : 0, 11641
Telemetry [candidate] (12.947 ms) : 0, 12947
section iast
BytebuddyAgent [baseline] (832.916 ms) : 0, 832916
BytebuddyAgent [candidate] (840.561 ms) : 0, 840561
GlobalTracer [baseline] (230.167 ms) : 0, 230167
GlobalTracer [candidate] (231.788 ms) : 0, 231788
AppSec [baseline] (55.497 ms) : 0, 55497
AppSec [candidate] (53.807 ms) : 0, 53807
Remote Config [baseline] (625.864 µs) : 0, 626
Remote Config [candidate] (636.137 µs) : 0, 636
Telemetry [baseline] (8.73 ms) : 0, 8730
Telemetry [candidate] (8.937 ms) : 0, 8937
IAST [baseline] (23.926 ms) : 0, 23926
IAST [candidate] (26.638 ms) : 0, 26638
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (834.322 ms) : 0, 834322
BytebuddyAgent [candidate] (833.888 ms) : 0, 833888
GlobalTracer [baseline] (230.398 ms) : 0, 230398
GlobalTracer [candidate] (230.929 ms) : 0, 230929
AppSec [baseline] (54.149 ms) : 0, 54149
AppSec [candidate] (54.241 ms) : 0, 54241
Remote Config [baseline] (626.213 µs) : 0, 626
Remote Config [candidate] (625.301 µs) : 0, 625
Telemetry [baseline] (8.866 ms) : 0, 8866
Telemetry [candidate] (8.807 ms) : 0, 8807
IAST [baseline] (25.952 ms) : 0, 25952
IAST [candidate] (25.876 ms) : 0, 25876
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (830.025 ms) : 0, 830025
BytebuddyAgent [candidate] (851.234 ms) : 0, 851234
GlobalTracer [baseline] (230.693 ms) : 0, 230693
GlobalTracer [candidate] (235.063 ms) : 0, 235063
AppSec [baseline] (53.335 ms) : 0, 53335
AppSec [candidate] (55.896 ms) : 0, 55896
Remote Config [baseline] (619.719 µs) : 0, 620
Remote Config [candidate] (639.167 µs) : 0, 639
Telemetry [baseline] (8.598 ms) : 0, 8598
Telemetry [candidate] (8.792 ms) : 0, 8792
IAST [baseline] (25.885 ms) : 0, 25885
IAST [candidate] (25.088 ms) : 0, 25088

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
end_time	2025-01-29T09:48:43	2025-01-29T09:55:51
git_branch	master	nikita-tkachenko/generalize-test-retries
git_commit_date	1738137590	1738143458
git_commit_sha	`f06d59c`	`5f23339`
release_version	1.46.0-SNAPSHOT~f06d59c0dd	1.46.0-SNAPSHOT~5f23339385
start_time	2025-01-29T09:48:29	2025-01-29T09:55:37

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1738144911	1738144911
ci_job_id	784984257	784984257
ci_pipeline_id	54289730	54289730
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-k9ef9rs-project-304-concurrent-0-mrqfuojy 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-k9ef9rs-project-304-concurrent-0-mrqfuojy 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 16 unstable metrics.

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.46.0-SNAPSHOT~5f23339385, baseline=1.46.0-SNAPSHOT~f06d59c0dd
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.368 ms) : 1348, 1388
.   : milestone, 1368,
appsec (1.766 ms) : 1743, 1789
.   : milestone, 1766,
appsec_no_iast (1.767 ms) : 1743, 1791
.   : milestone, 1767,
iast (1.518 ms) : 1494, 1542
.   : milestone, 1518,
profiling (1.57 ms) : 1545, 1595
.   : milestone, 1570,
tracing (1.486 ms) : 1461, 1512
.   : milestone, 1486,
section candidate
no_agent (1.371 ms) : 1351, 1391
.   : milestone, 1371,
appsec (1.763 ms) : 1739, 1787
.   : milestone, 1763,
appsec_no_iast (1.763 ms) : 1738, 1787
.   : milestone, 1763,
iast (1.516 ms) : 1492, 1540
.   : milestone, 1516,
profiling (1.515 ms) : 1491, 1539
.   : milestone, 1515,
tracing (1.512 ms) : 1488, 1536
.   : milestone, 1512,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.368 ms [1.348 ms, 1.388 ms]	-
appsec	1.766 ms [1.743 ms, 1.789 ms]	397.799 µs (29.1%)
appsec_no_iast	1.767 ms [1.743 ms, 1.791 ms]	398.867 µs (29.2%)
iast	1.518 ms [1.494 ms, 1.542 ms]	149.472 µs (10.9%)
profiling	1.57 ms [1.545 ms, 1.595 ms]	201.999 µs (14.8%)
tracing	1.486 ms [1.461 ms, 1.512 ms]	118.319 µs (8.6%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.371 ms [1.351 ms, 1.391 ms]	-
appsec	1.763 ms [1.739 ms, 1.787 ms]	391.774 µs (28.6%)
appsec_no_iast	1.763 ms [1.738 ms, 1.787 ms]	391.562 µs (28.6%)
iast	1.516 ms [1.492 ms, 1.54 ms]	144.825 µs (10.6%)
profiling	1.515 ms [1.491 ms, 1.539 ms]	144.256 µs (10.5%)
tracing	1.512 ms [1.488 ms, 1.536 ms]	140.786 µs (10.3%)

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.46.0-SNAPSHOT~5f23339385, baseline=1.46.0-SNAPSHOT~f06d59c0dd
    dateFormat X
    axisFormat %s
section baseline
no_agent (385.87 µs) : 366, 406
.   : milestone, 386,
iast (517.236 µs) : 496, 539
.   : milestone, 517,
iast_FULL (757.241 µs) : 734, 780
.   : milestone, 757,
iast_GLOBAL (574.313 µs) : 550, 599
.   : milestone, 574,
iast_HARDCODED_SECRET_DISABLED (517.986 µs) : 495, 540
.   : milestone, 518,
iast_INACTIVE (470.838 µs) : 449, 492
.   : milestone, 471,
iast_TELEMETRY_OFF (504.475 µs) : 481, 528
.   : milestone, 504,
tracing (463.483 µs) : 442, 485
.   : milestone, 463,
section candidate
no_agent (380.456 µs) : 361, 400
.   : milestone, 380,
iast (516.247 µs) : 493, 539
.   : milestone, 516,
iast_FULL (749.18 µs) : 726, 772
.   : milestone, 749,
iast_GLOBAL (570.577 µs) : 548, 593
.   : milestone, 571,
iast_HARDCODED_SECRET_DISABLED (518.027 µs) : 496, 540
.   : milestone, 518,
iast_INACTIVE (467.477 µs) : 446, 489
.   : milestone, 467,
iast_TELEMETRY_OFF (506.888 µs) : 483, 531
.   : milestone, 507,
tracing (467.503 µs) : 446, 489
.   : milestone, 468,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	385.87 µs [366.059 µs, 405.681 µs]	-
iast	517.236 µs [495.529 µs, 538.943 µs]	131.366 µs (34.0%)
iast_FULL	757.241 µs [734.285 µs, 780.198 µs]	371.371 µs (96.2%)
iast_GLOBAL	574.313 µs [549.52 µs, 599.105 µs]	188.443 µs (48.8%)
iast_HARDCODED_SECRET_DISABLED	517.986 µs [495.473 µs, 540.498 µs]	132.116 µs (34.2%)
iast_INACTIVE	470.838 µs [449.436 µs, 492.24 µs]	84.968 µs (22.0%)
iast_TELEMETRY_OFF	504.475 µs [481.446 µs, 527.504 µs]	118.605 µs (30.7%)
tracing	463.483 µs [442.247 µs, 484.72 µs]	77.613 µs (20.1%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	380.456 µs [360.801 µs, 400.11 µs]	-
iast	516.247 µs [493.045 µs, 539.449 µs]	135.791 µs (35.7%)
iast_FULL	749.18 µs [726.143 µs, 772.216 µs]	368.724 µs (96.9%)
iast_GLOBAL	570.577 µs [547.942 µs, 593.212 µs]	190.121 µs (50.0%)
iast_HARDCODED_SECRET_DISABLED	518.027 µs [495.593 µs, 540.46 µs]	137.571 µs (36.2%)
iast_INACTIVE	467.477 µs [446.233 µs, 488.722 µs]	87.021 µs (22.9%)
iast_TELEMETRY_OFF	506.888 µs [482.698 µs, 531.078 µs]	126.432 µs (33.2%)
tracing	467.503 µs [446.499 µs, 488.507 µs]	87.047 µs (22.9%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	nikita-tkachenko/generalize-test-retries
git_commit_date	1738137590	1738143458
git_commit_sha	`f06d59c`	`5f23339`
release_version	1.46.0-SNAPSHOT~f06d59c0dd	1.46.0-SNAPSHOT~5f23339385

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1738145462	1738145462
ci_job_id	784984258	784984258
ci_pipeline_id	54289730	54289730
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-qirit7hx-project-304-concurrent-1-ft79jxta 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-qirit7hx-project-304-concurrent-1-ft79jxta 6.8.0-1021-aws #23~22.04.1-Ubuntu SMP Tue Dec 10 16:50:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
variant	appsec	appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics.

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.46.0-SNAPSHOT~5f23339385, baseline=1.46.0-SNAPSHOT~f06d59c0dd
    dateFormat X
    axisFormat %s
section baseline
no_agent (14.952 s) : 14952000, 14952000
.   : milestone, 14952000,
appsec (15.063 s) : 15063000, 15063000
.   : milestone, 15063000,
iast (19.081 s) : 19081000, 19081000
.   : milestone, 19081000,
iast_GLOBAL (18.062 s) : 18062000, 18062000
.   : milestone, 18062000,
profiling (15.054 s) : 15054000, 15054000
.   : milestone, 15054000,
tracing (15.191 s) : 15191000, 15191000
.   : milestone, 15191000,
section candidate
no_agent (14.88 s) : 14880000, 14880000
.   : milestone, 14880000,
appsec (14.828 s) : 14828000, 14828000
.   : milestone, 14828000,
iast (19.094 s) : 19094000, 19094000
.   : milestone, 19094000,
iast_GLOBAL (17.926 s) : 17926000, 17926000
.   : milestone, 17926000,
profiling (15.127 s) : 15127000, 15127000
.   : milestone, 15127000,
tracing (15.17 s) : 15170000, 15170000
.   : milestone, 15170000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	14.952 s [14.952 s, 14.952 s]	-
appsec	15.063 s [15.063 s, 15.063 s]	111.0 ms (0.7%)
iast	19.081 s [19.081 s, 19.081 s]	4.129 s (27.6%)
iast_GLOBAL	18.062 s [18.062 s, 18.062 s]	3.11 s (20.8%)
profiling	15.054 s [15.054 s, 15.054 s]	102.0 ms (0.7%)
tracing	15.191 s [15.191 s, 15.191 s]	239.0 ms (1.6%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	14.88 s [14.88 s, 14.88 s]	-
appsec	14.828 s [14.828 s, 14.828 s]	-52.0 ms (-0.3%)
iast	19.094 s [19.094 s, 19.094 s]	4.214 s (28.3%)
iast_GLOBAL	17.926 s [17.926 s, 17.926 s]	3.046 s (20.5%)
profiling	15.127 s [15.127 s, 15.127 s]	247.0 ms (1.7%)
tracing	15.17 s [15.17 s, 15.17 s]	290.0 ms (1.9%)

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.46.0-SNAPSHOT~5f23339385, baseline=1.46.0-SNAPSHOT~f06d59c0dd
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.474 ms) : 1463, 1486
.   : milestone, 1474,
appsec (2.361 ms) : 2318, 2404
.   : milestone, 2361,
iast (2.115 ms) : 2060, 2169
.   : milestone, 2115,
iast_GLOBAL (2.152 ms) : 2097, 2207
.   : milestone, 2152,
profiling (2.46 ms) : 2272, 2648
.   : milestone, 2460,
tracing (1.952 ms) : 1909, 1994
.   : milestone, 1952,
section candidate
no_agent (1.473 ms) : 1461, 1484
.   : milestone, 1473,
appsec (2.363 ms) : 2319, 2407
.   : milestone, 2363,
iast (2.109 ms) : 2054, 2164
.   : milestone, 2109,
iast_GLOBAL (2.15 ms) : 2095, 2205
.   : milestone, 2150,
profiling (1.977 ms) : 1934, 2021
.   : milestone, 1977,
tracing (1.945 ms) : 1903, 1987
.   : milestone, 1945,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.474 ms [1.463 ms, 1.486 ms]	-
appsec	2.361 ms [2.318 ms, 2.404 ms]	886.813 µs (60.2%)
iast	2.115 ms [2.06 ms, 2.169 ms]	640.44 µs (43.4%)
iast_GLOBAL	2.152 ms [2.097 ms, 2.207 ms]	677.593 µs (46.0%)
profiling	2.46 ms [2.272 ms, 2.648 ms]	985.701 µs (66.9%)
tracing	1.952 ms [1.909 ms, 1.994 ms]	477.347 µs (32.4%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.473 ms [1.461 ms, 1.484 ms]	-
appsec	2.363 ms [2.319 ms, 2.407 ms]	890.37 µs (60.5%)
iast	2.109 ms [2.054 ms, 2.164 ms]	636.276 µs (43.2%)
iast_GLOBAL	2.15 ms [2.095 ms, 2.205 ms]	677.295 µs (46.0%)
profiling	1.977 ms [1.934 ms, 2.021 ms]	504.576 µs (34.3%)
tracing	1.945 ms [1.903 ms, 1.987 ms]	471.949 µs (32.0%)

… suppressed for current method

| Package | Type | Package file | Manager | Update | Change | |---|---|---|---|---|---| | [com.google.cloud:google-cloud-datastore](https://github.com/googleapis/java-datastore) | dependencies | misk/gradle/libs.versions.toml | gradle | minor | `2.25.4` -> `2.26.0` | | [com.datadoghq:dd-trace-api](https://github.com/datadog/dd-trace-java) | dependencies | misk/gradle/libs.versions.toml | gradle | minor | `1.45.2` -> `1.46.0` | | [com.datadoghq:dd-trace-ot](https://github.com/datadog/dd-trace-java) | dependencies | misk/gradle/libs.versions.toml | gradle | minor | `1.45.2` -> `1.46.0` | | [software.amazon.awssdk:sdk-core](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.30.9` -> `2.30.10` | | [software.amazon.awssdk:dynamodb-enhanced](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.30.9` -> `2.30.10` | | [software.amazon.awssdk:dynamodb](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.30.9` -> `2.30.10` | | [software.amazon.awssdk:aws-core](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.30.9` -> `2.30.10` | | [software.amazon.awssdk:bom](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.30.9` -> `2.30.10` | | [software.amazon.awssdk:auth](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.30.9` -> `2.30.10` | --- ### Release Notes <details> <summary>googleapis/java-datastore (com.google.cloud:google-cloud-datastore)</summary> ### [`v2.26.0`](https://github.com/googleapis/java-datastore/blob/HEAD/CHANGELOG.md#2260-2025-01-29) ##### Features - Add firestoreInDatastoreMode for datastore emulator ([#1698](googleapis/java-datastore#1698)) ([50f106d](googleapis/java-datastore@50f106d)) ##### Dependencies - Update dependency com.google.cloud:sdk-platform-java-config to v3.42.0 ([#1725](googleapis/java-datastore#1725)) ([1cbaf22](googleapis/java-datastore@1cbaf22)) </details> <details> <summary>datadog/dd-trace-java (com.datadoghq:dd-trace-api)</summary> ### [`v1.46.0`](https://github.com/DataDog/dd-trace-java/releases/tag/v1.46.0): 1.46.0 ##### Breaking Changes > \[!WARNING] > jnr-unixsocket is now an external dependency of dd-trace-ot and must be included when deploying dd-trace-ot. > \[!NOTE] > The API `TracerScope.setAsync(boolean)`, used to manually control asynchronous span propagation, does no more apply to the scope instance but to the active span scope. ##### Components ##### Application Security Management (IAST) - 🐛 Fix String.replace instrumentation for IAST ([#8281](DataDog/dd-trace-java#8281) - [@Mariovido](https://github.com/Mariovido)) - ✨ Apply the standard nomenclature to the stacktrace configs ([#8244](DataDog/dd-trace-java#8244) - [@jandro996](https://github.com/jandro996)) - 🐛 Exclude false positive weak randomness ([#8232](DataDog/dd-trace-java#8232) - [@jandro996](https://github.com/jandro996)) - ✨ Propagation of translateEscapes of String class ([#8186](DataDog/dd-trace-java#8186) - [@sezen-datadog](https://github.com/sezen-datadog)) - ✨ Add security control metrics ([#8175](DataDog/dd-trace-java#8175) - [@jandro996](https://github.com/jandro996)) - ✨ Increase IAST propagation to StringBuffer setLength ([#8128](DataDog/dd-trace-java#8128) - [@Mariovido](https://github.com/Mariovido)) - ✨ Add IAST taint tracking for DB values ([#8072](DataDog/dd-trace-java#8072) - [@Mariovido](https://github.com/Mariovido)) ##### Application Security Management (WAF) - 🐛 Prevents a NPE when there is no subscriber for user events ([#8258](DataDog/dd-trace-java#8258) - [@manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) - ✨ Apply the standard nomenclature to the stacktrace configs ([#8244](DataDog/dd-trace-java#8244) - [@jandro996](https://github.com/jandro996)) - 🐛 Ensure cached subscriptions are cleared on reconfiguration via RC ([#8229](DataDog/dd-trace-java#8229) - [@manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) - ✨ Add support for session tracking in Vertx ([#8167](DataDog/dd-trace-java#8167) - [@manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) - ✨ Create span tag: \_dd.appsec.rasp.timeout ([#8269](DataDog/dd-trace-java#8269) - [@Mariovido](https://github.com/Mariovido)) ##### Build & Tooling - 🐛 Ensure shaded helpers have unique names when injected into class-loaders ([#8192](DataDog/dd-trace-java#8192) - [@mcculls](https://github.com/mcculls)) ##### Configuration at Runtime - 🐛 Remove filtering of `DD_SERVICE` and `DD_ENV` from the tracer ([#8176](DataDog/dd-trace-java#8176) - [@mhlidd](https://github.com/mhlidd)) ##### Continuous Integration Visibility - 🧹 Generalize TestRetryPolicy to TestExecutionPolicy ([#8302](DataDog/dd-trace-java#8302) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - 🧹 Parallelize CI Visibility settings requests ([#8299](DataDog/dd-trace-java#8299) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - 🧹 Generalize test retry logic ([#8289](DataDog/dd-trace-java#8289) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - 🧹 Generalize tests skipping logic ([#8288](DataDog/dd-trace-java#8288) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - 🧹 Remove skip and shouldBeSkipped methods from TestEventsHandler in favor of isSkippable ([#8286](DataDog/dd-trace-java#8286) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - ✨⚡ Optimize Git repository information computation ([#8270](DataDog/dd-trace-java#8270) - [@dougqh](https://github.com/dougqh)) - ✨ Always request known tests from the backend ([#8268](DataDog/dd-trace-java#8268) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - ✨ Fix NPE when trying to get retry analyzer in Test NG ([#8253](DataDog/dd-trace-java#8253) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - 🐛 Set test framework and test framework version tags atomically ([#8252](DataDog/dd-trace-java#8252) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - ✨ Add debug logging to Android Gradle module layout logic ([#8251](DataDog/dd-trace-java#8251) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - 🐛 Fix source and destination folders computation for Android Gradle projects ([#8190](DataDog/dd-trace-java#8190) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - ✨ Add basic Scala Weaver sbt support ([#8189](DataDog/dd-trace-java#8189) - [@daniel-mohedano](https://github.com/daniel-mohedano)) - ✨ Implement impacted tests detection ([#8188](DataDog/dd-trace-java#8188) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) ##### Data Streams Monitoring - ✨ Change hash computation for protobuf to better represent impacting changes + save proto number in schema ([#8201](DataDog/dd-trace-java#8201) - [@vandonr](https://github.com/vandonr)) ##### Database Monitoring - Add peer service tag in dbm sql commenter ([#7913](DataDog/dd-trace-java#7913) - [@jordan-wong](https://github.com/jordan-wong)) ##### Dynamic Instrumentation - ✨ Add support for SymDB to scan directories ([#8306](DataDog/dd-trace-java#8306) - [@jpbempel](https://github.com/jpbempel)) - ✨ Add SymDB report for any jar scanning failures ([#8300](DataDog/dd-trace-java#8300) - [@jpbempel](https://github.com/jpbempel)) - ✨ Use two budgets depending on type ([#8283](DataDog/dd-trace-java#8283) - [@evanchooly](https://github.com/evanchooly)) - ✨ Institute a 10 snapshot per probe per trace budget ([#8277](DataDog/dd-trace-java#8277) - [@evanchooly](https://github.com/evanchooly)) - 🐛 Avoid double snapshots for Exception Replay ([#8273](DataDog/dd-trace-java#8273) - [@jpbempel](https://github.com/jpbempel)) - ✨ Simplify code origins. Separate out snapshot generation. ([#8263](DataDog/dd-trace-java#8263) - [@evanchooly](https://github.com/evanchooly)) - ✨ Add Exception probe custom instrumentation ([#8230](DataDog/dd-trace-java#8230) - [@jpbempel](https://github.com/jpbempel)) - ✨ Enhance log probes to honor debug session tags ([#8215](DataDog/dd-trace-java#8215) - [@evanchooly](https://github.com/evanchooly)) - 🐛 Don't redact env tokens from debugger probe snapshots ([#8211](DataDog/dd-trace-java#8211) - [@watson](https://github.com/watson)) - ✨⚡ Move Trace/SpanId capture at commit time ([#8184](DataDog/dd-trace-java#8184) - [@jpbempel](https://github.com/jpbempel)) - 🐛 Capture values at entry for method probe ([#8169](DataDog/dd-trace-java#8169) - [@jpbempel](https://github.com/jpbempel)) ##### JMX fetch - 🐛 Mute JMXFetch Shutdown in progress error ([#8068](DataDog/dd-trace-java#8068) - [@ygree](https://github.com/ygree)) ##### OpenTracing - ⚠️🧹 Make jnr-unixsocket an explicit dependency of dd-trace-ot ([#8307](DataDog/dd-trace-java#8307) - [@mcculls](https://github.com/mcculls)) ##### Profiling - 🐛 Avoid unsupported API call for creating folders on windows ([#8304](DataDog/dd-trace-java#8304) - [@jbachorik](https://github.com/jbachorik)) - ✨ Tag profiles for serverless ([#8279](DataDog/dd-trace-java#8279) - [@jbachorik](https://github.com/jbachorik)) - ✨ add queue type and length to queue events ([#8242](DataDog/dd-trace-java#8242) - [@richardstartin](https://github.com/richardstartin)) - 🐛 TempLocationManager Fixes and Improvements ([#8191](DataDog/dd-trace-java#8191) - [@jbachorik](https://github.com/jbachorik)) - ✨ Bump ddprof to 1.18.0 ([#8173](DataDog/dd-trace-java#8173) - [@jbachorik](https://github.com/jbachorik)) - ✨ Report profiler initialization and configuration errors to telemetry ([#8171](DataDog/dd-trace-java#8171) - [@jbachorik](https://github.com/jbachorik)) ##### Telemetry - ✨ Add pending traces report in tracer flares ([#8053](DataDog/dd-trace-java#8053) - [@mhlidd](https://github.com/mhlidd)) ##### Testing - ✨ Test http server requests in parallel ([#8222](DataDog/dd-trace-java#8222) - [@amarziali](https://github.com/amarziali)) ##### Trace context propagation - ✨ Add non default propagator registration ([#8310](DataDog/dd-trace-java#8310) - [@PerfectSlayer](https://github.com/PerfectSlayer)) ##### Tracer core - ✨ Probe for existence of IBMSASL or ACCP security providers ([#8276](DataDog/dd-trace-java#8276) - [@mcculls](https://github.com/mcculls)) - ✨⚡ Overhead improvement to agent feedback based sampling ([#8265](DataDog/dd-trace-java#8265) - [@dougqh](https://github.com/dougqh)) - 🧹 Move async propagation API from scope to tracer ([#8231](DataDog/dd-trace-java#8231) - [@PerfectSlayer](https://github.com/PerfectSlayer)) - ✨ Introduce context propagation API ([#8161](DataDog/dd-trace-java#8161) - [@PerfectSlayer](https://github.com/PerfectSlayer)) - ✨🧪 Use env-entry to add tags per webapp deployment ([#8138](DataDog/dd-trace-java#8138) - [@amarziali](https://github.com/amarziali)) - ✨ Introduce context helpers API ([#8134](DataDog/dd-trace-java#8134) - [@PerfectSlayer](https://github.com/PerfectSlayer)) - ✨ Support IPv6 values for `DD_AGENT_HOST` and `DD_TRACE_AGENT_URL` ([#7984](DataDog/dd-trace-java#7984) - [@mhlidd](https://github.com/mhlidd)) ##### Instrumentations ##### Apache HttpComponents - 🐛 Properly finish spans and support latest apache httpclient5 ([#8272](DataDog/dd-trace-java#8272) - [@amarziali](https://github.com/amarziali)) ##### AWS Lambda instrumentation - 🐛 Properly capture lambda payloads for all handler types. ([#8264](DataDog/dd-trace-java#8264) - [@purple4reina](https://github.com/purple4reina)) ##### AWS S3 instrumentation - 💡 Create S3 instrumentation + add span pointers ([#8075](DataDog/dd-trace-java#8075) - [@nhulston](https://github.com/nhulston)) ##### AWS SDK instrumentation - 🐛 Revert "Add avoid double instrumenting lambda non-streaming handlers." ([#8247](DataDog/dd-trace-java#8247) - [@nhulston](https://github.com/nhulston)) ##### Cassandra - ✨ Allow extracting keyspace from statement result ([#8239](DataDog/dd-trace-java#8239) - [@amarziali](https://github.com/amarziali)) ##### Core Java language instrumentation - ✨ Propagation of translateEscapes of String class ([#8186](DataDog/dd-trace-java#8186) - [@sezen-datadog](https://github.com/sezen-datadog)) ##### Eclipse Vert.x instrumentation - 🐛 Fix vertx worker propagation and error handling ([#8237](DataDog/dd-trace-java#8237) - [@amarziali](https://github.com/amarziali)) - ✨ Support vertx 5 ([#8220](DataDog/dd-trace-java#8220) - [@amarziali](https://github.com/amarziali)) - ✨ Add support for session tracking in Vertx ([#8167](DataDog/dd-trace-java#8167) - [@manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) ##### Kafka instrumentation - 🐛 Prevent possible NPE calculating Kafka record header size ([#8292](DataDog/dd-trace-java#8292) - [@ygree](https://github.com/ygree)) ##### Mule instrumentation - 🐛 Fix crash using Mule with JPMS ([#8187](DataDog/dd-trace-java#8187) - [@amarziali](https://github.com/amarziali)) ##### Protocol Buffer instrumentation - ✨ Change hash computation for protobuf to better represent impacting changes + save proto number in schema ([#8201](DataDog/dd-trace-java#8201) - [@vandonr](https://github.com/vandonr)) ##### Spring instrumentation - 🐛 Preserve getQualifier from spring scheduling runnables ([#8293](DataDog/dd-trace-java#8293) - [@amarziali](https://github.com/amarziali)) </details> --- ### Configuration 📅 **Schedule**: Branch creation - "after 6pm every weekday,before 2am every weekday" in timezone Australia/Melbourne, Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 👻 **Immortal**: This PR will be recreated if closed unmerged. Get [config help](https://github.com/renovatebot/renovate/discussions) if that's undesired. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). GitOrigin-RevId: bb09d47e4eed77a003f630273b4d0a84003eb899

nikita-tkachenko-datadog added type: refactoring comp: ci visibility Continuous Integration Visibility labels Jan 27, 2025

Base automatically changed from nikita-tkachenko/generalize-test-skipping to master January 28, 2025 16:47

Generalize test retry logic

3e58f24

nikita-tkachenko-datadog force-pushed the nikita-tkachenko/generalize-test-retries branch from 4d8593e to 3e58f24 Compare January 28, 2025 16:52

nikita-tkachenko-datadog and others added 2 commits January 28, 2025 17:58

Update suppressFailures() method to return whether failures should be…

3138572

… suppressed for current method

Merge branch 'master' into nikita-tkachenko/generalize-test-retries

5f23339

nikita-tkachenko-datadog marked this pull request as ready for review January 29, 2025 12:57

nikita-tkachenko-datadog requested a review from a team as a code owner January 29, 2025 12:57

nikita-tkachenko-datadog requested review from anmarchenko and daniel-mohedano January 29, 2025 12:57

daniel-mohedano approved these changes Jan 29, 2025

View reviewed changes

nikita-tkachenko-datadog merged commit bcddb29 into master Jan 29, 2025
162 of 194 checks passed

nikita-tkachenko-datadog deleted the nikita-tkachenko/generalize-test-retries branch January 29, 2025 13:23

github-actions bot added this to the 1.46.0 milestone Jan 29, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Generalize test retry logic #8289

Generalize test retry logic #8289

nikita-tkachenko-datadog commented Jan 27, 2025 •

edited

Loading

pr-commenter bot commented Jan 27, 2025 •

edited

Loading

Generalize test retry logic #8289

Generalize test retry logic #8289

Conversation

nikita-tkachenko-datadog commented Jan 27, 2025 • edited Loading

What Does This Do

Motivation

Contributor Checklist

pr-commenter bot commented Jan 27, 2025 • edited Loading

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

nikita-tkachenko-datadog commented Jan 27, 2025 •

edited

Loading

pr-commenter bot commented Jan 27, 2025 •

edited

Loading