Add the possibility to output the logs of the Java tracer in JSON #8083

cecile75 · 2024-12-12T16:55:11Z

What Does This Do

Add the possibility to output the logs of the Java tracer in JSON.
The JSON keys have been chosen to be compatible out of the box with the Logs UI of Datadog.
You will be able to identify all the logs coming from the tracer by filtering them using @origin:dd.trace

Note:

If you don’t configure a specific datadog.slf4j.simpleLogger.dateTimeFormat,

when JSON output is enabled, the dateTime format will be yyyy-MM-dd'T'HH:mm:ss.SSSZ to be compatible with the logs UI .
when JSON output is disabled, the default dateTime is still [dd.trace 'yyyy-MM-dd HH:mm:ss:SSS Z']

Usage

System property: datadog.slf4j.simpleLogger.jsonEnabled

Default: false

Description: This configuration key set to true will output the logs of the Java tracer in JSON.

Motivation

#1109

Additional Notes

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any usefull labels
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the public documentation in case of new configuration flag or behavior

Jira ticket: [PROJ-IDENT]

github-actions · 2024-12-12T16:55:24Z

Hi! 👋 Looks like you updated a Git Submodule.
If this was not intentional please make sure to:

Update the submodule to the latest commit on the master branch using the git submodule update command,
Check you properly set up your environment for contributing.

github-actions · 2024-12-12T16:55:52Z

Hi! 👋 Looks like you updated a Git Submodule.
If this was not intentional please make sure to:

Update the submodule to the latest commit on the master branch using the git submodule update command,
Check you properly set up your environment for contributing.

pr-commenter · 2024-12-12T17:01:14Z

Benchmarks

Startup

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
end_time	2025-01-08T17:19:54	2025-01-08T17:26:52
git_branch	master	cecile/json-logs
git_commit_date	1736352030	1736356142
git_commit_sha	`73f194e`	`e85525e`
release_version	1.45.0-SNAPSHOT~73f194e3ea	1.45.0-SNAPSHOT~e85525e12c
start_time	2025-01-08T17:19:41	2025-01-08T17:26:38

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1736357564	1736357564
ci_job_id	758972138	758972138
ci_pipeline_id	52466259	52466259
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 17 unstable metrics.

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.45.0-SNAPSHOT~e85525e12c, baseline=1.45.0-SNAPSHOT~73f194e3ea
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.353 ms) : 1333, 1372
.   : milestone, 1353,
appsec (1.743 ms) : 1720, 1767
.   : milestone, 1743,
appsec_no_iast (1.737 ms) : 1713, 1760
.   : milestone, 1737,
iast (1.505 ms) : 1482, 1529
.   : milestone, 1505,
profiling (1.548 ms) : 1523, 1572
.   : milestone, 1548,
tracing (1.465 ms) : 1440, 1490
.   : milestone, 1465,
section candidate
no_agent (1.357 ms) : 1337, 1377
.   : milestone, 1357,
appsec (1.758 ms) : 1734, 1781
.   : milestone, 1758,
appsec_no_iast (1.764 ms) : 1740, 1789
.   : milestone, 1764,
iast (1.518 ms) : 1495, 1540
.   : milestone, 1518,
profiling (1.504 ms) : 1481, 1528
.   : milestone, 1504,
tracing (1.481 ms) : 1457, 1506
.   : milestone, 1481,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.353 ms [1.333 ms, 1.372 ms]	-
appsec	1.743 ms [1.72 ms, 1.767 ms]	390.758 µs (28.9%)
appsec_no_iast	1.737 ms [1.713 ms, 1.76 ms]	383.978 µs (28.4%)
iast	1.505 ms [1.482 ms, 1.529 ms]	152.329 µs (11.3%)
profiling	1.548 ms [1.523 ms, 1.572 ms]	194.841 µs (14.4%)
tracing	1.465 ms [1.44 ms, 1.49 ms]	111.847 µs (8.3%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.357 ms [1.337 ms, 1.377 ms]	-
appsec	1.758 ms [1.734 ms, 1.781 ms]	400.72 µs (29.5%)
appsec_no_iast	1.764 ms [1.74 ms, 1.789 ms]	407.304 µs (30.0%)
iast	1.518 ms [1.495 ms, 1.54 ms]	160.749 µs (11.8%)
profiling	1.504 ms [1.481 ms, 1.528 ms]	147.139 µs (10.8%)
tracing	1.481 ms [1.457 ms, 1.506 ms]	124.316 µs (9.2%)

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.45.0-SNAPSHOT~e85525e12c, baseline=1.45.0-SNAPSHOT~73f194e3ea
    dateFormat X
    axisFormat %s
section baseline
no_agent (375.781 µs) : 354, 397
.   : milestone, 376,
iast (489.228 µs) : 468, 511
.   : milestone, 489,
iast_FULL (649.325 µs) : 628, 671
.   : milestone, 649,
iast_GLOBAL (523.021 µs) : 501, 545
.   : milestone, 523,
iast_HARDCODED_SECRET_DISABLED (488.877 µs) : 468, 510
.   : milestone, 489,
iast_INACTIVE (447.248 µs) : 426, 468
.   : milestone, 447,
iast_TELEMETRY_OFF (482.844 µs) : 461, 505
.   : milestone, 483,
tracing (443.509 µs) : 423, 464
.   : milestone, 444,
section candidate
no_agent (372.637 µs) : 352, 394
.   : milestone, 373,
iast (491.834 µs) : 471, 513
.   : milestone, 492,
iast_FULL (652.121 µs) : 630, 674
.   : milestone, 652,
iast_GLOBAL (517.036 µs) : 495, 539
.   : milestone, 517,
iast_HARDCODED_SECRET_DISABLED (491.619 µs) : 470, 513
.   : milestone, 492,
iast_INACTIVE (454.537 µs) : 433, 476
.   : milestone, 455,
iast_TELEMETRY_OFF (476.574 µs) : 455, 498
.   : milestone, 477,
tracing (449.971 µs) : 429, 471
.   : milestone, 450,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	375.781 µs [354.306 µs, 397.257 µs]	-
iast	489.228 µs [467.914 µs, 510.542 µs]	113.447 µs (30.2%)
iast_FULL	649.325 µs [627.592 µs, 671.058 µs]	273.544 µs (72.8%)
iast_GLOBAL	523.021 µs [501.428 µs, 544.614 µs]	147.24 µs (39.2%)
iast_HARDCODED_SECRET_DISABLED	488.877 µs [467.734 µs, 510.02 µs]	113.096 µs (30.1%)
iast_INACTIVE	447.248 µs [426.36 µs, 468.136 µs]	71.467 µs (19.0%)
iast_TELEMETRY_OFF	482.844 µs [460.747 µs, 504.942 µs]	107.063 µs (28.5%)
tracing	443.509 µs [422.99 µs, 464.029 µs]	67.728 µs (18.0%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	372.637 µs [351.724 µs, 393.549 µs]	-
iast	491.834 µs [470.55 µs, 513.117 µs]	119.197 µs (32.0%)
iast_FULL	652.121 µs [630.232 µs, 674.01 µs]	279.484 µs (75.0%)
iast_GLOBAL	517.036 µs [494.716 µs, 539.356 µs]	144.399 µs (38.8%)
iast_HARDCODED_SECRET_DISABLED	491.619 µs [470.037 µs, 513.201 µs]	118.982 µs (31.9%)
iast_INACTIVE	454.537 µs [433.216 µs, 475.858 µs]	81.9 µs (22.0%)
iast_TELEMETRY_OFF	476.574 µs [455.349 µs, 497.8 µs]	103.937 µs (27.9%)
tracing	449.971 µs [428.844 µs, 471.097 µs]	77.334 µs (20.8%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	cecile/json-logs
git_commit_date	1736352030	1736356142
git_commit_sha	`73f194e`	`e85525e`
release_version	1.45.0-SNAPSHOT~73f194e3ea	1.45.0-SNAPSHOT~e85525e12c

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1736358009	1736358009
ci_job_id	758972141	758972141
ci_pipeline_id	52466259	52466259
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant	appsec	appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.45.0-SNAPSHOT~e85525e12c, baseline=1.45.0-SNAPSHOT~73f194e3ea
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.472 ms) : 1461, 1484
.   : milestone, 1472,
appsec (2.366 ms) : 2322, 2409
.   : milestone, 2366,
iast (2.11 ms) : 2056, 2165
.   : milestone, 2110,
iast_GLOBAL (2.156 ms) : 2101, 2211
.   : milestone, 2156,
profiling (1.983 ms) : 1938, 2028
.   : milestone, 1983,
tracing (1.956 ms) : 1913, 1998
.   : milestone, 1956,
section candidate
no_agent (1.476 ms) : 1465, 1488
.   : milestone, 1476,
appsec (2.368 ms) : 2326, 2411
.   : milestone, 2368,
iast (2.091 ms) : 2038, 2145
.   : milestone, 2091,
iast_GLOBAL (2.134 ms) : 2081, 2187
.   : milestone, 2134,
profiling (1.969 ms) : 1926, 2012
.   : milestone, 1969,
tracing (1.936 ms) : 1895, 1977
.   : milestone, 1936,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.472 ms [1.461 ms, 1.484 ms]	-
appsec	2.366 ms [2.322 ms, 2.409 ms]	893.588 µs (60.7%)
iast	2.11 ms [2.056 ms, 2.165 ms]	638.208 µs (43.3%)
iast_GLOBAL	2.156 ms [2.101 ms, 2.211 ms]	684.061 µs (46.5%)
profiling	1.983 ms [1.938 ms, 2.028 ms]	510.945 µs (34.7%)
tracing	1.956 ms [1.913 ms, 1.998 ms]	483.245 µs (32.8%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.476 ms [1.465 ms, 1.488 ms]	-
appsec	2.368 ms [2.326 ms, 2.411 ms]	891.991 µs (60.4%)
iast	2.091 ms [2.038 ms, 2.145 ms]	615.064 µs (41.7%)
iast_GLOBAL	2.134 ms [2.081 ms, 2.187 ms]	657.637 µs (44.5%)
profiling	1.969 ms [1.926 ms, 2.012 ms]	492.85 µs (33.4%)
tracing	1.936 ms [1.895 ms, 1.977 ms]	459.502 µs (31.1%)

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.45.0-SNAPSHOT~e85525e12c, baseline=1.45.0-SNAPSHOT~73f194e3ea
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.282 s) : 15282000, 15282000
.   : milestone, 15282000,
appsec (15.119 s) : 15119000, 15119000
.   : milestone, 15119000,
iast (19.067 s) : 19067000, 19067000
.   : milestone, 19067000,
iast_GLOBAL (17.986 s) : 17986000, 17986000
.   : milestone, 17986000,
profiling (15.79 s) : 15790000, 15790000
.   : milestone, 15790000,
tracing (14.876 s) : 14876000, 14876000
.   : milestone, 14876000,
section candidate
no_agent (15.636 s) : 15636000, 15636000
.   : milestone, 15636000,
appsec (15.152 s) : 15152000, 15152000
.   : milestone, 15152000,
iast (18.771 s) : 18771000, 18771000
.   : milestone, 18771000,
iast_GLOBAL (17.583 s) : 17583000, 17583000
.   : milestone, 17583000,
profiling (14.982 s) : 14982000, 14982000
.   : milestone, 14982000,
tracing (15.255 s) : 15255000, 15255000
.   : milestone, 15255000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.282 s [15.282 s, 15.282 s]	-
appsec	15.119 s [15.119 s, 15.119 s]	-163.0 ms (-1.1%)
iast	19.067 s [19.067 s, 19.067 s]	3.785 s (24.8%)
iast_GLOBAL	17.986 s [17.986 s, 17.986 s]	2.704 s (17.7%)
profiling	15.79 s [15.79 s, 15.79 s]	508.0 ms (3.3%)
tracing	14.876 s [14.876 s, 14.876 s]	-406.0 ms (-2.7%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.636 s [15.636 s, 15.636 s]	-
appsec	15.152 s [15.152 s, 15.152 s]	-484.0 ms (-3.1%)
iast	18.771 s [18.771 s, 18.771 s]	3.135 s (20.0%)
iast_GLOBAL	17.583 s [17.583 s, 17.583 s]	1.947 s (12.5%)
profiling	14.982 s [14.982 s, 14.982 s]	-654.0 ms (-4.2%)
tracing	15.255 s [15.255 s, 15.255 s]	-381.0 ms (-2.4%)

bm1549 and others added 11 commits September 13, 2024 09:01

POC - log in JSON format

07b28a0

initial commit

10f5803

init tests + formatting

4ff3266

Adapt JSON keys to DD LOG UI

3882881

add some tests

4e90249

remove empty lines

df21ed3

merge with master

de59f03

use new JSON writer component

4da342e

remove unnecessary toString()

e46d6c2

delete useless file

5593f27

clean

4f4a1b0

Merge branch 'master' into cecile/json-logs

8b60eee

Cecile Terpin and others added 5 commits December 26, 2024 17:29

Merge branch 'master' into cecile/json-logs

5d05184

rename property + remove env var check

e4c4b13

remove unused import

1179ee5

Merge branch 'master' into cecile/json-logs

4cdb964

api is enough

e85525e

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add the possibility to output the logs of the Java tracer in JSON #8083

Add the possibility to output the logs of the Java tracer in JSON #8083

cecile75 commented Dec 12, 2024 •

edited

Loading

github-actions bot commented Dec 12, 2024

github-actions bot commented Dec 12, 2024

pr-commenter bot commented Dec 12, 2024 •

edited

Loading

Add the possibility to output the logs of the Java tracer in JSON #8083

Are you sure you want to change the base?

Add the possibility to output the logs of the Java tracer in JSON #8083

Conversation

cecile75 commented Dec 12, 2024 • edited Loading

What Does This Do

Note:

Usage

Motivation

Additional Notes

Contributor Checklist

github-actions bot commented Dec 12, 2024

github-actions bot commented Dec 12, 2024

pr-commenter bot commented Dec 12, 2024 • edited Loading

Benchmarks

Startup

Load

Parameters

Summary

Dacapo

Parameters

Summary

cecile75 commented Dec 12, 2024 •

edited

Loading

pr-commenter bot commented Dec 12, 2024 •

edited

Loading