Always request known tests from the backend #8268

nikita-tkachenko-datadog · 2025-01-22T13:14:06Z

What Does This Do

Updates CI Visibility to always request the list of known tests from the backend.

Motivation

The list of known tests retrieved from the backend is used for marking new test cases with test.is_new tag, and for some additional features (like Early Flakiness Detection or tests reordering).

Previously the list of known tests was only requested from the backend if those additional features were enabled.

Now it is requested regardless of whether the features are active or not, so test.is_new tagging is done always.

Additional Notes

https://docs.google.com/document/d/1mPMM7Uye2jHUMx5ADTMSRt_Rff98klRLj5OpTruszzQ/edit?tab=t.0

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any usefull labels
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the public documentation in case of new configuration flag or behavior

Jira ticket: SDTEST-1275

pr-commenter · 2025-01-22T13:56:08Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	nikita-tkachenko/known-tests
git_commit_date	1737561779	1737561866
git_commit_sha	`ad44687`	`79081f8`
release_version	1.46.0-SNAPSHOT~ad44687e99	1.46.0-SNAPSHOT~79081f8f82

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1737564407	1737564407
ci_job_id	776153667	776153667
ci_pipeline_id	53702083	53702083
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module	Agent	Agent
parent	None	None
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 59 metrics, 4 unstable metrics.

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.46.0-SNAPSHOT~79081f8f82, baseline=1.46.0-SNAPSHOT~ad44687e99

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.056 s) : 0, 1055750
Total [baseline] (10.529 s) : 0, 10528646
Agent [candidate] (1.055 s) : 0, 1055217
Total [candidate] (10.489 s) : 0, 10488823
section appsec
Agent [baseline] (1.191 s) : 0, 1190567
Total [baseline] (10.714 s) : 0, 10713989
Agent [candidate] (1.19 s) : 0, 1190463
Total [candidate] (10.691 s) : 0, 10691428
section iast
Agent [baseline] (1.19 s) : 0, 1190300
Total [baseline] (11.014 s) : 0, 11014402
Agent [candidate] (1.186 s) : 0, 1185605
Total [candidate] (10.966 s) : 0, 10966443
section profiling
Agent [baseline] (1.261 s) : 0, 1261347
Total [baseline] (10.754 s) : 0, 10754452
Agent [candidate] (1.253 s) : 0, 1252924
Total [candidate] (10.826 s) : 0, 10826457

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.056 s	-
Agent	appsec	1.191 s	134.817 ms (12.8%)
Agent	iast	1.19 s	134.55 ms (12.7%)
Agent	profiling	1.261 s	205.597 ms (19.5%)
Total	tracing	10.529 s	-
Total	appsec	10.714 s	185.343 ms (1.8%)
Total	iast	11.014 s	485.756 ms (4.6%)
Total	profiling	10.754 s	225.806 ms (2.1%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.055 s	-
Agent	appsec	1.19 s	135.246 ms (12.8%)
Agent	iast	1.186 s	130.387 ms (12.4%)
Agent	profiling	1.253 s	197.706 ms (18.7%)
Total	tracing	10.489 s	-
Total	appsec	10.691 s	202.605 ms (1.9%)
Total	iast	10.966 s	477.62 ms (4.6%)
Total	profiling	10.826 s	337.634 ms (3.2%)

gantt
    title petclinic - break down per module: candidate=1.46.0-SNAPSHOT~79081f8f82, baseline=1.46.0-SNAPSHOT~ad44687e99

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (713.723 ms) : 0, 713723
BytebuddyAgent [candidate] (713.563 ms) : 0, 713563
GlobalTracer [baseline] (255.345 ms) : 0, 255345
GlobalTracer [candidate] (255.376 ms) : 0, 255376
AppSec [baseline] (55.326 ms) : 0, 55326
AppSec [candidate] (56.137 ms) : 0, 56137
Remote Config [baseline] (742.623 µs) : 0, 743
Remote Config [candidate] (747.378 µs) : 0, 747
Telemetry [baseline] (15.592 ms) : 0, 15592
Telemetry [candidate] (14.358 ms) : 0, 14358
section appsec
BytebuddyAgent [baseline] (732.709 ms) : 0, 732709
BytebuddyAgent [candidate] (733.289 ms) : 0, 733289
GlobalTracer [baseline] (253.276 ms) : 0, 253276
GlobalTracer [candidate] (252.862 ms) : 0, 252862
AppSec [baseline] (171.279 ms) : 0, 171279
AppSec [candidate] (170.81 ms) : 0, 170810
IAST [baseline] (19.091 ms) : 0, 19091
IAST [candidate] (19.309 ms) : 0, 19309
Remote Config [baseline] (674.545 µs) : 0, 675
Remote Config [candidate] (683.294 µs) : 0, 683
Telemetry [baseline] (8.113 ms) : 0, 8113
Telemetry [candidate] (8.16 ms) : 0, 8160
section iast
BytebuddyAgent [baseline] (836.955 ms) : 0, 836955
BytebuddyAgent [candidate] (833.761 ms) : 0, 833761
GlobalTracer [baseline] (247.86 ms) : 0, 247860
GlobalTracer [candidate] (247.351 ms) : 0, 247351
AppSec [baseline] (58.973 ms) : 0, 58973
AppSec [candidate] (58.139 ms) : 0, 58139
IAST [baseline] (21.787 ms) : 0, 21787
IAST [candidate] (21.788 ms) : 0, 21788
Remote Config [baseline] (685.004 µs) : 0, 685
Remote Config [candidate] (672.252 µs) : 0, 672
Telemetry [baseline] (8.898 ms) : 0, 8898
Telemetry [candidate] (8.831 ms) : 0, 8831
section profiling
BytebuddyAgent [baseline] (707.997 ms) : 0, 707997
BytebuddyAgent [candidate] (703.325 ms) : 0, 703325
GlobalTracer [baseline] (349.958 ms) : 0, 349958
GlobalTracer [candidate] (349.509 ms) : 0, 349509
AppSec [baseline] (55.656 ms) : 0, 55656
AppSec [candidate] (53.732 ms) : 0, 53732
Remote Config [baseline] (734.698 µs) : 0, 735
Remote Config [candidate] (728.197 µs) : 0, 728
Telemetry [baseline] (8.84 ms) : 0, 8840
Telemetry [candidate] (8.712 ms) : 0, 8712
ProfilingAgent [baseline] (95.951 ms) : 0, 95951
ProfilingAgent [candidate] (95.036 ms) : 0, 95036
Profiling [baseline] (95.976 ms) : 0, 95976
Profiling [candidate] (95.06 ms) : 0, 95060

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.46.0-SNAPSHOT~79081f8f82, baseline=1.46.0-SNAPSHOT~ad44687e99

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.06 s) : 0, 1060480
Total [baseline] (8.654 s) : 0, 8654086
Agent [candidate] (1.065 s) : 0, 1064656
Total [candidate] (8.698 s) : 0, 8697995
section iast
Agent [baseline] (1.183 s) : 0, 1183482
Total [baseline] (9.227 s) : 0, 9227322
Agent [candidate] (1.188 s) : 0, 1188072
Total [candidate] (9.241 s) : 0, 9241311
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.191 s) : 0, 1190706
Total [baseline] (9.175 s) : 0, 9174788
Agent [candidate] (1.185 s) : 0, 1185011
Total [candidate] (9.177 s) : 0, 9177010
section iast_TELEMETRY_OFF
Agent [baseline] (1.188 s) : 0, 1188451
Total [baseline] (9.217 s) : 0, 9216592
Agent [candidate] (1.178 s) : 0, 1178037
Total [candidate] (9.186 s) : 0, 9185711

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.06 s	-
Agent	iast	1.183 s	123.002 ms (11.6%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.191 s	130.227 ms (12.3%)
Agent	iast_TELEMETRY_OFF	1.188 s	127.971 ms (12.1%)
Total	tracing	8.654 s	-
Total	iast	9.227 s	573.235 ms (6.6%)
Total	iast_HARDCODED_SECRET_DISABLED	9.175 s	520.702 ms (6.0%)
Total	iast_TELEMETRY_OFF	9.217 s	562.506 ms (6.5%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.065 s	-
Agent	iast	1.188 s	123.416 ms (11.6%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.185 s	120.355 ms (11.3%)
Agent	iast_TELEMETRY_OFF	1.178 s	113.381 ms (10.6%)
Total	tracing	8.698 s	-
Total	iast	9.241 s	543.316 ms (6.2%)
Total	iast_HARDCODED_SECRET_DISABLED	9.177 s	479.014 ms (5.5%)
Total	iast_TELEMETRY_OFF	9.186 s	487.716 ms (5.6%)

gantt
    title insecure-bank - break down per module: candidate=1.46.0-SNAPSHOT~79081f8f82, baseline=1.46.0-SNAPSHOT~ad44687e99

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (718.701 ms) : 0, 718701
BytebuddyAgent [candidate] (718.878 ms) : 0, 718878
GlobalTracer [baseline] (256.716 ms) : 0, 256716
GlobalTracer [candidate] (258.804 ms) : 0, 258804
AppSec [baseline] (56.245 ms) : 0, 56245
AppSec [candidate] (57.181 ms) : 0, 57181
Remote Config [baseline] (743.87 µs) : 0, 744
Remote Config [candidate] (771.374 µs) : 0, 771
Telemetry [baseline] (13.073 ms) : 0, 13073
Telemetry [candidate] (13.901 ms) : 0, 13901
section iast
BytebuddyAgent [baseline] (831.054 ms) : 0, 831054
BytebuddyAgent [candidate] (836.701 ms) : 0, 836701
GlobalTracer [baseline] (248.211 ms) : 0, 248211
GlobalTracer [candidate] (247.33 ms) : 0, 247330
AppSec [baseline] (58.16 ms) : 0, 58160
AppSec [candidate] (57.987 ms) : 0, 57987
IAST [baseline] (21.712 ms) : 0, 21712
IAST [candidate] (21.475 ms) : 0, 21475
Remote Config [baseline] (656.249 µs) : 0, 656
Remote Config [candidate] (666.285 µs) : 0, 666
Telemetry [baseline] (8.72 ms) : 0, 8720
Telemetry [candidate] (8.751 ms) : 0, 8751
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (837.886 ms) : 0, 837886
BytebuddyAgent [candidate] (833.061 ms) : 0, 833061
GlobalTracer [baseline] (248.229 ms) : 0, 248229
GlobalTracer [candidate] (247.087 ms) : 0, 247087
AppSec [baseline] (58.347 ms) : 0, 58347
AppSec [candidate] (58.337 ms) : 0, 58337
IAST [baseline] (21.674 ms) : 0, 21674
IAST [candidate] (21.747 ms) : 0, 21747
Remote Config [baseline] (676.91 µs) : 0, 677
Remote Config [candidate] (669.083 µs) : 0, 669
Telemetry [baseline] (8.823 ms) : 0, 8823
Telemetry [candidate] (8.946 ms) : 0, 8946
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (836.108 ms) : 0, 836108
BytebuddyAgent [candidate] (828.048 ms) : 0, 828048
GlobalTracer [baseline] (248.052 ms) : 0, 248052
GlobalTracer [candidate] (246.25 ms) : 0, 246250
AppSec [baseline] (58.427 ms) : 0, 58427
AppSec [candidate] (57.449 ms) : 0, 57449
IAST [baseline] (21.256 ms) : 0, 21256
IAST [candidate] (21.832 ms) : 0, 21832
Remote Config [baseline] (691.447 µs) : 0, 691
Remote Config [candidate] (658.237 µs) : 0, 658
Telemetry [baseline] (8.814 ms) : 0, 8814
Telemetry [candidate] (8.716 ms) : 0, 8716

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
end_time	2025-01-22T16:16:33	2025-01-22T16:23:36
git_branch	master	nikita-tkachenko/known-tests
git_commit_date	1737561779	1737561866
git_commit_sha	`ad44687`	`79081f8`
release_version	1.46.0-SNAPSHOT~ad44687e99	1.46.0-SNAPSHOT~79081f8f82
start_time	2025-01-22T16:16:19	2025-01-22T16:23:22

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1737563374	1737563374
ci_job_id	776153668	776153668
ci_pipeline_id	53702083	53702083
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 17 unstable metrics.

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.46.0-SNAPSHOT~79081f8f82, baseline=1.46.0-SNAPSHOT~ad44687e99
    dateFormat X
    axisFormat %s
section baseline
no_agent (379.761 µs) : 360, 399
.   : milestone, 380,
iast (509.298 µs) : 487, 531
.   : milestone, 509,
iast_FULL (747.485 µs) : 726, 769
.   : milestone, 747,
iast_GLOBAL (548.125 µs) : 527, 570
.   : milestone, 548,
iast_HARDCODED_SECRET_DISABLED (505.367 µs) : 484, 527
.   : milestone, 505,
iast_INACTIVE (455.905 µs) : 435, 477
.   : milestone, 456,
iast_TELEMETRY_OFF (488.139 µs) : 467, 510
.   : milestone, 488,
tracing (451.551 µs) : 430, 473
.   : milestone, 452,
section candidate
no_agent (375.675 µs) : 355, 397
.   : milestone, 376,
iast (515.621 µs) : 493, 538
.   : milestone, 516,
iast_FULL (739.267 µs) : 717, 761
.   : milestone, 739,
iast_GLOBAL (552.429 µs) : 530, 575
.   : milestone, 552,
iast_HARDCODED_SECRET_DISABLED (505.142 µs) : 484, 527
.   : milestone, 505,
iast_INACTIVE (459.174 µs) : 438, 481
.   : milestone, 459,
iast_TELEMETRY_OFF (500.869 µs) : 478, 523
.   : milestone, 501,
tracing (450.845 µs) : 429, 472
.   : milestone, 451,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	379.761 µs [360.198 µs, 399.324 µs]	-
iast	509.298 µs [487.358 µs, 531.237 µs]	129.536 µs (34.1%)
iast_FULL	747.485 µs [725.667 µs, 769.303 µs]	367.724 µs (96.8%)
iast_GLOBAL	548.125 µs [526.571 µs, 569.679 µs]	168.364 µs (44.3%)
iast_HARDCODED_SECRET_DISABLED	505.367 µs [483.733 µs, 527.002 µs]	125.606 µs (33.1%)
iast_INACTIVE	455.905 µs [434.84 µs, 476.969 µs]	76.144 µs (20.1%)
iast_TELEMETRY_OFF	488.139 µs [466.588 µs, 509.691 µs]	108.378 µs (28.5%)
tracing	451.551 µs [430.401 µs, 472.701 µs]	71.79 µs (18.9%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	375.675 µs [354.68 µs, 396.67 µs]	-
iast	515.621 µs [492.883 µs, 538.36 µs]	139.946 µs (37.3%)
iast_FULL	739.267 µs [717.336 µs, 761.198 µs]	363.592 µs (96.8%)
iast_GLOBAL	552.429 µs [530.038 µs, 574.819 µs]	176.754 µs (47.0%)
iast_HARDCODED_SECRET_DISABLED	505.142 µs [483.71 µs, 526.574 µs]	129.467 µs (34.5%)
iast_INACTIVE	459.174 µs [437.644 µs, 480.705 µs]	83.499 µs (22.2%)
iast_TELEMETRY_OFF	500.869 µs [478.281 µs, 523.458 µs]	125.194 µs (33.3%)
tracing	450.845 µs [429.252 µs, 472.439 µs]	75.17 µs (20.0%)

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.46.0-SNAPSHOT~79081f8f82, baseline=1.46.0-SNAPSHOT~ad44687e99
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.353 ms) : 1334, 1373
.   : milestone, 1353,
appsec (1.747 ms) : 1723, 1771
.   : milestone, 1747,
appsec_no_iast (1.753 ms) : 1728, 1779
.   : milestone, 1753,
iast (1.525 ms) : 1502, 1549
.   : milestone, 1525,
profiling (1.51 ms) : 1486, 1534
.   : milestone, 1510,
tracing (1.471 ms) : 1445, 1496
.   : milestone, 1471,
section candidate
no_agent (1.353 ms) : 1333, 1373
.   : milestone, 1353,
appsec (1.73 ms) : 1706, 1753
.   : milestone, 1730,
appsec_no_iast (1.78 ms) : 1755, 1804
.   : milestone, 1780,
iast (1.504 ms) : 1480, 1529
.   : milestone, 1504,
profiling (1.555 ms) : 1530, 1580
.   : milestone, 1555,
tracing (1.492 ms) : 1468, 1516
.   : milestone, 1492,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.353 ms [1.334 ms, 1.373 ms]	-
appsec	1.747 ms [1.723 ms, 1.771 ms]	393.478 µs (29.1%)
appsec_no_iast	1.753 ms [1.728 ms, 1.779 ms]	399.853 µs (29.5%)
iast	1.525 ms [1.502 ms, 1.549 ms]	171.825 µs (12.7%)
profiling	1.51 ms [1.486 ms, 1.534 ms]	156.508 µs (11.6%)
tracing	1.471 ms [1.445 ms, 1.496 ms]	117.372 µs (8.7%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.353 ms [1.333 ms, 1.373 ms]	-
appsec	1.73 ms [1.706 ms, 1.753 ms]	376.91 µs (27.9%)
appsec_no_iast	1.78 ms [1.755 ms, 1.804 ms]	426.976 µs (31.6%)
iast	1.504 ms [1.48 ms, 1.529 ms]	151.638 µs (11.2%)
profiling	1.555 ms [1.53 ms, 1.58 ms]	202.022 µs (14.9%)
tracing	1.492 ms [1.468 ms, 1.516 ms]	139.65 µs (10.3%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	nikita-tkachenko/known-tests
git_commit_date	1737561779	1737561866
git_commit_sha	`ad44687`	`79081f8`
release_version	1.46.0-SNAPSHOT~ad44687e99	1.46.0-SNAPSHOT~79081f8f82

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1737563981	1737563981
ci_job_id	776153669	776153669
ci_pipeline_id	53702083	53702083
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant	appsec	appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.46.0-SNAPSHOT~79081f8f82, baseline=1.46.0-SNAPSHOT~ad44687e99
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.298 s) : 15298000, 15298000
.   : milestone, 15298000,
appsec (15.105 s) : 15105000, 15105000
.   : milestone, 15105000,
iast (18.838 s) : 18838000, 18838000
.   : milestone, 18838000,
iast_GLOBAL (18.017 s) : 18017000, 18017000
.   : milestone, 18017000,
profiling (14.979 s) : 14979000, 14979000
.   : milestone, 14979000,
tracing (14.849 s) : 14849000, 14849000
.   : milestone, 14849000,
section candidate
no_agent (15.619 s) : 15619000, 15619000
.   : milestone, 15619000,
appsec (15.001 s) : 15001000, 15001000
.   : milestone, 15001000,
iast (18.561 s) : 18561000, 18561000
.   : milestone, 18561000,
iast_GLOBAL (17.876 s) : 17876000, 17876000
.   : milestone, 17876000,
profiling (14.951 s) : 14951000, 14951000
.   : milestone, 14951000,
tracing (15.174 s) : 15174000, 15174000
.   : milestone, 15174000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.298 s [15.298 s, 15.298 s]	-
appsec	15.105 s [15.105 s, 15.105 s]	-193.0 ms (-1.3%)
iast	18.838 s [18.838 s, 18.838 s]	3.54 s (23.1%)
iast_GLOBAL	18.017 s [18.017 s, 18.017 s]	2.719 s (17.8%)
profiling	14.979 s [14.979 s, 14.979 s]	-319.0 ms (-2.1%)
tracing	14.849 s [14.849 s, 14.849 s]	-449.0 ms (-2.9%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.619 s [15.619 s, 15.619 s]	-
appsec	15.001 s [15.001 s, 15.001 s]	-618.0 ms (-4.0%)
iast	18.561 s [18.561 s, 18.561 s]	2.942 s (18.8%)
iast_GLOBAL	17.876 s [17.876 s, 17.876 s]	2.257 s (14.5%)
profiling	14.951 s [14.951 s, 14.951 s]	-668.0 ms (-4.3%)
tracing	15.174 s [15.174 s, 15.174 s]	-445.0 ms (-2.8%)

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.46.0-SNAPSHOT~79081f8f82, baseline=1.46.0-SNAPSHOT~ad44687e99
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.471 ms) : 1459, 1482
.   : milestone, 1471,
appsec (2.37 ms) : 2327, 2413
.   : milestone, 2370,
iast (2.103 ms) : 2049, 2157
.   : milestone, 2103,
iast_GLOBAL (2.15 ms) : 2095, 2205
.   : milestone, 2150,
profiling (1.965 ms) : 1922, 2008
.   : milestone, 1965,
tracing (1.946 ms) : 1904, 1988
.   : milestone, 1946,
section candidate
no_agent (1.477 ms) : 1466, 1489
.   : milestone, 1477,
appsec (2.365 ms) : 2322, 2409
.   : milestone, 2365,
iast (2.113 ms) : 2059, 2168
.   : milestone, 2113,
iast_GLOBAL (2.162 ms) : 2107, 2217
.   : milestone, 2162,
profiling (1.979 ms) : 1935, 2024
.   : milestone, 1979,
tracing (1.948 ms) : 1906, 1990
.   : milestone, 1948,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.471 ms [1.459 ms, 1.482 ms]	-
appsec	2.37 ms [2.327 ms, 2.413 ms]	899.134 µs (61.1%)
iast	2.103 ms [2.049 ms, 2.157 ms]	632.382 µs (43.0%)
iast_GLOBAL	2.15 ms [2.095 ms, 2.205 ms]	679.413 µs (46.2%)
profiling	1.965 ms [1.922 ms, 2.008 ms]	494.301 µs (33.6%)
tracing	1.946 ms [1.904 ms, 1.988 ms]	475.345 µs (32.3%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.477 ms [1.466 ms, 1.489 ms]	-
appsec	2.365 ms [2.322 ms, 2.409 ms]	888.006 µs (60.1%)
iast	2.113 ms [2.059 ms, 2.168 ms]	635.992 µs (43.0%)
iast_GLOBAL	2.162 ms [2.107 ms, 2.217 ms]	684.131 µs (46.3%)
profiling	1.979 ms [1.935 ms, 2.024 ms]	501.871 µs (34.0%)
tracing	1.948 ms [1.906 ms, 1.99 ms]	470.681 µs (31.9%)

| Package | Type | Package file | Manager | Update | Change | |---|---|---|---|---|---| | [com.google.cloud:google-cloud-datastore](https://github.com/googleapis/java-datastore) | dependencies | misk/gradle/libs.versions.toml | gradle | minor | `2.25.4` -> `2.26.0` | | [com.datadoghq:dd-trace-api](https://github.com/datadog/dd-trace-java) | dependencies | misk/gradle/libs.versions.toml | gradle | minor | `1.45.2` -> `1.46.0` | | [com.datadoghq:dd-trace-ot](https://github.com/datadog/dd-trace-java) | dependencies | misk/gradle/libs.versions.toml | gradle | minor | `1.45.2` -> `1.46.0` | | [software.amazon.awssdk:sdk-core](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.30.9` -> `2.30.10` | | [software.amazon.awssdk:dynamodb-enhanced](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.30.9` -> `2.30.10` | | [software.amazon.awssdk:dynamodb](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.30.9` -> `2.30.10` | | [software.amazon.awssdk:aws-core](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.30.9` -> `2.30.10` | | [software.amazon.awssdk:bom](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.30.9` -> `2.30.10` | | [software.amazon.awssdk:auth](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.30.9` -> `2.30.10` | --- ### Release Notes <details> <summary>googleapis/java-datastore (com.google.cloud:google-cloud-datastore)</summary> ### [`v2.26.0`](https://github.com/googleapis/java-datastore/blob/HEAD/CHANGELOG.md#2260-2025-01-29) ##### Features - Add firestoreInDatastoreMode for datastore emulator ([#1698](googleapis/java-datastore#1698)) ([50f106d](googleapis/java-datastore@50f106d)) ##### Dependencies - Update dependency com.google.cloud:sdk-platform-java-config to v3.42.0 ([#1725](googleapis/java-datastore#1725)) ([1cbaf22](googleapis/java-datastore@1cbaf22)) </details> <details> <summary>datadog/dd-trace-java (com.datadoghq:dd-trace-api)</summary> ### [`v1.46.0`](https://github.com/DataDog/dd-trace-java/releases/tag/v1.46.0): 1.46.0 ##### Breaking Changes > \[!WARNING] > jnr-unixsocket is now an external dependency of dd-trace-ot and must be included when deploying dd-trace-ot. > \[!NOTE] > The API `TracerScope.setAsync(boolean)`, used to manually control asynchronous span propagation, does no more apply to the scope instance but to the active span scope. ##### Components ##### Application Security Management (IAST) - 🐛 Fix String.replace instrumentation for IAST ([#8281](DataDog/dd-trace-java#8281) - [@Mariovido](https://github.com/Mariovido)) - ✨ Apply the standard nomenclature to the stacktrace configs ([#8244](DataDog/dd-trace-java#8244) - [@jandro996](https://github.com/jandro996)) - 🐛 Exclude false positive weak randomness ([#8232](DataDog/dd-trace-java#8232) - [@jandro996](https://github.com/jandro996)) - ✨ Propagation of translateEscapes of String class ([#8186](DataDog/dd-trace-java#8186) - [@sezen-datadog](https://github.com/sezen-datadog)) - ✨ Add security control metrics ([#8175](DataDog/dd-trace-java#8175) - [@jandro996](https://github.com/jandro996)) - ✨ Increase IAST propagation to StringBuffer setLength ([#8128](DataDog/dd-trace-java#8128) - [@Mariovido](https://github.com/Mariovido)) - ✨ Add IAST taint tracking for DB values ([#8072](DataDog/dd-trace-java#8072) - [@Mariovido](https://github.com/Mariovido)) ##### Application Security Management (WAF) - 🐛 Prevents a NPE when there is no subscriber for user events ([#8258](DataDog/dd-trace-java#8258) - [@manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) - ✨ Apply the standard nomenclature to the stacktrace configs ([#8244](DataDog/dd-trace-java#8244) - [@jandro996](https://github.com/jandro996)) - 🐛 Ensure cached subscriptions are cleared on reconfiguration via RC ([#8229](DataDog/dd-trace-java#8229) - [@manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) - ✨ Add support for session tracking in Vertx ([#8167](DataDog/dd-trace-java#8167) - [@manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) - ✨ Create span tag: \_dd.appsec.rasp.timeout ([#8269](DataDog/dd-trace-java#8269) - [@Mariovido](https://github.com/Mariovido)) ##### Build & Tooling - 🐛 Ensure shaded helpers have unique names when injected into class-loaders ([#8192](DataDog/dd-trace-java#8192) - [@mcculls](https://github.com/mcculls)) ##### Configuration at Runtime - 🐛 Remove filtering of `DD_SERVICE` and `DD_ENV` from the tracer ([#8176](DataDog/dd-trace-java#8176) - [@mhlidd](https://github.com/mhlidd)) ##### Continuous Integration Visibility - 🧹 Generalize TestRetryPolicy to TestExecutionPolicy ([#8302](DataDog/dd-trace-java#8302) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - 🧹 Parallelize CI Visibility settings requests ([#8299](DataDog/dd-trace-java#8299) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - 🧹 Generalize test retry logic ([#8289](DataDog/dd-trace-java#8289) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - 🧹 Generalize tests skipping logic ([#8288](DataDog/dd-trace-java#8288) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - 🧹 Remove skip and shouldBeSkipped methods from TestEventsHandler in favor of isSkippable ([#8286](DataDog/dd-trace-java#8286) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - ✨⚡ Optimize Git repository information computation ([#8270](DataDog/dd-trace-java#8270) - [@dougqh](https://github.com/dougqh)) - ✨ Always request known tests from the backend ([#8268](DataDog/dd-trace-java#8268) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - ✨ Fix NPE when trying to get retry analyzer in Test NG ([#8253](DataDog/dd-trace-java#8253) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - 🐛 Set test framework and test framework version tags atomically ([#8252](DataDog/dd-trace-java#8252) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - ✨ Add debug logging to Android Gradle module layout logic ([#8251](DataDog/dd-trace-java#8251) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - 🐛 Fix source and destination folders computation for Android Gradle projects ([#8190](DataDog/dd-trace-java#8190) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - ✨ Add basic Scala Weaver sbt support ([#8189](DataDog/dd-trace-java#8189) - [@daniel-mohedano](https://github.com/daniel-mohedano)) - ✨ Implement impacted tests detection ([#8188](DataDog/dd-trace-java#8188) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) ##### Data Streams Monitoring - ✨ Change hash computation for protobuf to better represent impacting changes + save proto number in schema ([#8201](DataDog/dd-trace-java#8201) - [@vandonr](https://github.com/vandonr)) ##### Database Monitoring - Add peer service tag in dbm sql commenter ([#7913](DataDog/dd-trace-java#7913) - [@jordan-wong](https://github.com/jordan-wong)) ##### Dynamic Instrumentation - ✨ Add support for SymDB to scan directories ([#8306](DataDog/dd-trace-java#8306) - [@jpbempel](https://github.com/jpbempel)) - ✨ Add SymDB report for any jar scanning failures ([#8300](DataDog/dd-trace-java#8300) - [@jpbempel](https://github.com/jpbempel)) - ✨ Use two budgets depending on type ([#8283](DataDog/dd-trace-java#8283) - [@evanchooly](https://github.com/evanchooly)) - ✨ Institute a 10 snapshot per probe per trace budget ([#8277](DataDog/dd-trace-java#8277) - [@evanchooly](https://github.com/evanchooly)) - 🐛 Avoid double snapshots for Exception Replay ([#8273](DataDog/dd-trace-java#8273) - [@jpbempel](https://github.com/jpbempel)) - ✨ Simplify code origins. Separate out snapshot generation. ([#8263](DataDog/dd-trace-java#8263) - [@evanchooly](https://github.com/evanchooly)) - ✨ Add Exception probe custom instrumentation ([#8230](DataDog/dd-trace-java#8230) - [@jpbempel](https://github.com/jpbempel)) - ✨ Enhance log probes to honor debug session tags ([#8215](DataDog/dd-trace-java#8215) - [@evanchooly](https://github.com/evanchooly)) - 🐛 Don't redact env tokens from debugger probe snapshots ([#8211](DataDog/dd-trace-java#8211) - [@watson](https://github.com/watson)) - ✨⚡ Move Trace/SpanId capture at commit time ([#8184](DataDog/dd-trace-java#8184) - [@jpbempel](https://github.com/jpbempel)) - 🐛 Capture values at entry for method probe ([#8169](DataDog/dd-trace-java#8169) - [@jpbempel](https://github.com/jpbempel)) ##### JMX fetch - 🐛 Mute JMXFetch Shutdown in progress error ([#8068](DataDog/dd-trace-java#8068) - [@ygree](https://github.com/ygree)) ##### OpenTracing - ⚠️🧹 Make jnr-unixsocket an explicit dependency of dd-trace-ot ([#8307](DataDog/dd-trace-java#8307) - [@mcculls](https://github.com/mcculls)) ##### Profiling - 🐛 Avoid unsupported API call for creating folders on windows ([#8304](DataDog/dd-trace-java#8304) - [@jbachorik](https://github.com/jbachorik)) - ✨ Tag profiles for serverless ([#8279](DataDog/dd-trace-java#8279) - [@jbachorik](https://github.com/jbachorik)) - ✨ add queue type and length to queue events ([#8242](DataDog/dd-trace-java#8242) - [@richardstartin](https://github.com/richardstartin)) - 🐛 TempLocationManager Fixes and Improvements ([#8191](DataDog/dd-trace-java#8191) - [@jbachorik](https://github.com/jbachorik)) - ✨ Bump ddprof to 1.18.0 ([#8173](DataDog/dd-trace-java#8173) - [@jbachorik](https://github.com/jbachorik)) - ✨ Report profiler initialization and configuration errors to telemetry ([#8171](DataDog/dd-trace-java#8171) - [@jbachorik](https://github.com/jbachorik)) ##### Telemetry - ✨ Add pending traces report in tracer flares ([#8053](DataDog/dd-trace-java#8053) - [@mhlidd](https://github.com/mhlidd)) ##### Testing - ✨ Test http server requests in parallel ([#8222](DataDog/dd-trace-java#8222) - [@amarziali](https://github.com/amarziali)) ##### Trace context propagation - ✨ Add non default propagator registration ([#8310](DataDog/dd-trace-java#8310) - [@PerfectSlayer](https://github.com/PerfectSlayer)) ##### Tracer core - ✨ Probe for existence of IBMSASL or ACCP security providers ([#8276](DataDog/dd-trace-java#8276) - [@mcculls](https://github.com/mcculls)) - ✨⚡ Overhead improvement to agent feedback based sampling ([#8265](DataDog/dd-trace-java#8265) - [@dougqh](https://github.com/dougqh)) - 🧹 Move async propagation API from scope to tracer ([#8231](DataDog/dd-trace-java#8231) - [@PerfectSlayer](https://github.com/PerfectSlayer)) - ✨ Introduce context propagation API ([#8161](DataDog/dd-trace-java#8161) - [@PerfectSlayer](https://github.com/PerfectSlayer)) - ✨🧪 Use env-entry to add tags per webapp deployment ([#8138](DataDog/dd-trace-java#8138) - [@amarziali](https://github.com/amarziali)) - ✨ Introduce context helpers API ([#8134](DataDog/dd-trace-java#8134) - [@PerfectSlayer](https://github.com/PerfectSlayer)) - ✨ Support IPv6 values for `DD_AGENT_HOST` and `DD_TRACE_AGENT_URL` ([#7984](DataDog/dd-trace-java#7984) - [@mhlidd](https://github.com/mhlidd)) ##### Instrumentations ##### Apache HttpComponents - 🐛 Properly finish spans and support latest apache httpclient5 ([#8272](DataDog/dd-trace-java#8272) - [@amarziali](https://github.com/amarziali)) ##### AWS Lambda instrumentation - 🐛 Properly capture lambda payloads for all handler types. ([#8264](DataDog/dd-trace-java#8264) - [@purple4reina](https://github.com/purple4reina)) ##### AWS S3 instrumentation - 💡 Create S3 instrumentation + add span pointers ([#8075](DataDog/dd-trace-java#8075) - [@nhulston](https://github.com/nhulston)) ##### AWS SDK instrumentation - 🐛 Revert "Add avoid double instrumenting lambda non-streaming handlers." ([#8247](DataDog/dd-trace-java#8247) - [@nhulston](https://github.com/nhulston)) ##### Cassandra - ✨ Allow extracting keyspace from statement result ([#8239](DataDog/dd-trace-java#8239) - [@amarziali](https://github.com/amarziali)) ##### Core Java language instrumentation - ✨ Propagation of translateEscapes of String class ([#8186](DataDog/dd-trace-java#8186) - [@sezen-datadog](https://github.com/sezen-datadog)) ##### Eclipse Vert.x instrumentation - 🐛 Fix vertx worker propagation and error handling ([#8237](DataDog/dd-trace-java#8237) - [@amarziali](https://github.com/amarziali)) - ✨ Support vertx 5 ([#8220](DataDog/dd-trace-java#8220) - [@amarziali](https://github.com/amarziali)) - ✨ Add support for session tracking in Vertx ([#8167](DataDog/dd-trace-java#8167) - [@manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) ##### Kafka instrumentation - 🐛 Prevent possible NPE calculating Kafka record header size ([#8292](DataDog/dd-trace-java#8292) - [@ygree](https://github.com/ygree)) ##### Mule instrumentation - 🐛 Fix crash using Mule with JPMS ([#8187](DataDog/dd-trace-java#8187) - [@amarziali](https://github.com/amarziali)) ##### Protocol Buffer instrumentation - ✨ Change hash computation for protobuf to better represent impacting changes + save proto number in schema ([#8201](DataDog/dd-trace-java#8201) - [@vandonr](https://github.com/vandonr)) ##### Spring instrumentation - 🐛 Preserve getQualifier from spring scheduling runnables ([#8293](DataDog/dd-trace-java#8293) - [@amarziali](https://github.com/amarziali)) </details> --- ### Configuration 📅 **Schedule**: Branch creation - "after 6pm every weekday,before 2am every weekday" in timezone Australia/Melbourne, Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 👻 **Immortal**: This PR will be recreated if closed unmerged. Get [config help](https://github.com/renovatebot/renovate/discussions) if that's undesired. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). GitOrigin-RevId: bb09d47e4eed77a003f630273b4d0a84003eb899

nikita-tkachenko-datadog added type: enhancement comp: ci visibility Continuous Integration Visibility labels Jan 22, 2025

Base automatically changed from nikita-tkachenko/impacted-tests to master January 22, 2025 16:03

nikita-tkachenko-datadog added 3 commits January 22, 2025 17:04

Always request known tests from the backend

a3dd9fa

Add test.retry_reason to test fixtures

61ffadd

Rename EFD metrics constants to known tests

79081f8

nikita-tkachenko-datadog force-pushed the nikita-tkachenko/known-tests branch from dcd8878 to 79081f8 Compare January 22, 2025 16:04

nikita-tkachenko-datadog changed the title ~~Nikita tkachenko/known tests~~ Always request known tests from the backend Jan 22, 2025

nikita-tkachenko-datadog marked this pull request as ready for review January 22, 2025 16:46

nikita-tkachenko-datadog requested review from a team as code owners January 22, 2025 16:46

nikita-tkachenko-datadog requested review from tonyredondo, dougqh and daniel-mohedano January 22, 2025 16:46

daniel-mohedano approved these changes Jan 23, 2025

View reviewed changes

nikita-tkachenko-datadog merged commit 1efc71f into master Jan 23, 2025
175 checks passed

nikita-tkachenko-datadog deleted the nikita-tkachenko/known-tests branch January 23, 2025 10:25

github-actions bot added this to the 1.46.0 milestone Jan 23, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Always request known tests from the backend #8268

Always request known tests from the backend #8268

nikita-tkachenko-datadog commented Jan 22, 2025 •

edited by jira bot

Loading

pr-commenter bot commented Jan 22, 2025 •

edited

Loading

Always request known tests from the backend #8268

Always request known tests from the backend #8268

Conversation

nikita-tkachenko-datadog commented Jan 22, 2025 • edited by jira bot Loading

What Does This Do

Motivation

Additional Notes

Contributor Checklist

pr-commenter bot commented Jan 22, 2025 • edited Loading

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

nikita-tkachenko-datadog commented Jan 22, 2025 •

edited by jira bot

Loading

pr-commenter bot commented Jan 22, 2025 •

edited

Loading