V2.42.0 proposal #3431

szegedi · 2023-07-20T15:42:26Z

Features

profiling: Associate tracing span IDs with samples in the wall profiler ([profiling] Associate tracing span IDs with samples in the wall profiler #3371)
iast: Taint request URI (Taint request URI #3302)
iast: Detect missing header vulnerabilities (Detect missing header vulnerabilities #3269)
graphql: implement naming schema for graphql plugin (implement naming schema for graphql plugin #3279)

Enhancements

openai: be defensive about response object shape (openai: be defensive about response object shape #3408)
tracing: Ensure OTel span name is copied to resource.name (Ensure OTel span name is copied to resource.name #3412)
config: Fix service representation config (Fix service representation config #3419)
ci: Add BITBUCKET_GIT_HTTP_ORIGIN env variable (Add BITBUCKET_GIT_HTTP_ORIGIN env variable #3401)

Bug fixes

serverless: Handle case where Lambda handler is using callbacks instead of promises (fix: Handle case where Lambda handler is using callbacks instead of p… #3404)
ci-visibility: Do not modify how jest reports timeouts ([ci-visibility] Do not modify how jest reports timeouts #3399)
profiling: Allow https datadog agent url in continuous profiling (Continuous Profiler doesn't work with a datadog agent serving on HTTPS #3190)
tracing: Fix distributed tracing issue caused by single span ingestion changes (Fix distributed tracing issue caused by single span ingestion changes #3413)
tracing: Make datadog distributed tracing header take priority over tracecontext (Make datadog distributed tracing header take priority over tracecontext #3414)
tracing: Exclude empty telemetry metrics (Exclude empty telemetry metrics #3421)
openai: make services entirely optional when initializing (openai: make services entirely optional when initializing #3420)

Minor stuff (probably not notable for release notes)

profiling: Fix erroneous resetting of profiler state to undefined (Fix erroneous resetting of profiler state to undefined #3394)
Update yarn.lock with current pprof-nodejs version ([profiling] Update yarn.lock with current pprof-nodejs version #3398)
delete old build scripts (delete old build scripts #3396)
Add eslint rule for object-curly-newline (Add eslint rule for object-curly-newline #3406)
Fix a phrase in README (Fix a phrase in README #3405)
Fix same dependency detection (Fix same dependency detection #3386)
Ignore esbuild output from integration tests in esbuild (Ignore esbuild output from integration tests in esbuild #3410)
Fix flaky serverless integration test (Fix flaky serverless integration test #3418)
Fix native-iast-taint-tracking version number (Fix native-iast-taint-tracking version number #3433)
openai: remove String#replaceAll for Node.js <=14 (openai: remove String#replaceAll for Node.js <=14 #3432)
Make integration test for jest compatible with node 12 ([ci-visibility] Make integration test for jest compatible with node 12 #3440)

…ler (#3371) * Initial code for code hotspots and endpoint aggregation in wall profiler Hide functionality behind DD_PROFILING_EXPERIMENTAL_CODEHOTSPOTS_ENABLED and DD_PROFILING_EXPERIMENTAL_ENDPOINT_COLLECTION_ENABLED flags. Profiler tracks changes of active span by subscribing to async hooks before channel and to AsyncResourceStorage enter channel that notifies when current store is changed with enterWith/run. Profiler remembers previous span and previous started spans, upon change of active span, it checks if a sample has been taken by native profiler addon, then if that's the case, it updates the context of the sample with span id and root span id computed from previous span / started spans. It also add the tags from the last web span to the context to allow computation of endpoint later on. Minor changes to other parts of the code to make it all work: === * Report error if start fails * Avoid clearing span context tags upon export Profiler may need the tags to determine span resource name for endpoint aggregation: profiler keeps a reference on web span that was active when each profiling sample is taken. Then during profile export, span tags are used to determine endpoint. Endpoint cannot be determined right away because sometimes tags necessary to determine endpoint are only set just before span is finished. * Add request tags just after span creation For web spans, add request tags just after span creation instead of when span finishes. This is needed for profiler code hotspots implementation to determine which span are web spans and should be used to find current endpoint.

* Taint request URI * Add check for safe tainted origins on unvalidated redirect analyzer * Change assertion construction for unvalidated redirect analyzer test * Add metric for uri sourcing * Fix PR comments

* Detect X-Content-Type-Options missing header * HSTS Header missing analyzer and refactor of xcontenttype-header-missing-aanalyzer * Move function to class method * use startswith instead of index===0 * Do not send evidence if value is undefined * Fix comment in PR and add test * Changes to support telemetry * Rename method name * Rename analyzer object key

Forgot to update yarn.lock when bumping pprof-nodejs version in a previous PR.

* Fix same dependency detection * Do not send same dependency:version twice * Clear map on stop * Update savedDependencies to savedDependenciesToSend to improve understanding

…romises (#3404)

Co-authored-by: Attila Szegedi <[email protected]>

…#3413)

…xt (#3414)

* address and fix edge cases * move to better tracer option name for service name flattening

* implement naming schema for graphql plugin * implement schema v1 and add tests --------- Co-authored-by: Thomas Hunter II <[email protected]> Co-authored-by: Jordi Bertran de Balanda <[email protected]>

github-actions · 2023-07-20T15:43:24Z

Overall package size

Self size: 4.92 MB
Deduped: 68.24 MB
No deduping: 68.33 MB

Dependency sizes

name	version	self size	total size
@datadog/native-iast-taint-tracking	1.5.0	14.86 MB	14.86 MB
@datadog/native-appsec	3.2.0	13.38 MB	13.39 MB
@datadog/pprof	3.1.0	10.66 MB	11.5 MB
@datadog/native-metrics	1.6.0	7.88 MB	7.89 MB
protobufjs	7.2.4	2.74 MB	6.62 MB
@types/node	18.11.19	3.58 MB	3.58 MB
@datadog/native-iast-rewriter	2.0.1	2.09 MB	2.1 MB
@opentelemetry/core	1.3.1	784.66 kB	1.37 MB
@opentelemetry/api	1.4.1	780.32 kB	780.32 kB
msgpack-lite	0.1.26	201.16 kB	272.06 kB
opentracing	0.14.7	194.81 kB	194.81 kB
lru-cache	7.18.3	133.92 kB	133.92 kB
semver	7.5.3	93.39 kB	123.79 kB
@datadog/sketches-js	2.1.0	109.9 kB	109.9 kB
lodash.sortby	4.7.0	75.76 kB	75.76 kB
ipaddr.js	2.0.1	59.52 kB	59.52 kB
ignore	5.2.4	51.22 kB	51.22 kB
int64-buffer	0.1.10	49.18 kB	49.18 kB
import-in-the-middle	1.3.5	34.34 kB	38.81 kB
istanbul-lib-coverage	3.2.0	29.34 kB	29.34 kB
retry	0.10.1	27.44 kB	27.44 kB
lodash.uniq	4.5.0	25.01 kB	25.01 kB
limiter	1.1.5	23.17 kB	23.17 kB
lodash.kebabcase	4.1.1	17.75 kB	17.75 kB
node-abort-controller	3.1.1	16.89 kB	16.89 kB
lodash.pick	4.4.0	16.33 kB	16.33 kB
crypto-randomuuid	1.0.0	11.18 kB	11.18 kB
diagnostics_channel	1.1.0	7.07 kB	7.07 kB
path-to-regexp	0.1.7	6.78 kB	6.78 kB
koalas	1.0.2	6.47 kB	6.47 kB
methods	1.1.2	5.29 kB	5.29 kB
module-details-from-path	1.0.3	4.47 kB	4.47 kB

_{🤖 This report was automatically generated by heaviest-objects-in-the-universe}

codecov · 2023-07-20T15:44:22Z

Codecov Report

Merging #3431 (83387a3) into v2.x (fbc2899) will decrease coverage by 8.96%.
The diff coverage is 0.00%.

❗ Current head 83387a3 differs from pull request most recent head 372b481. Consider uploading reports for the commit 372b481 to get more accurate results

@@            Coverage Diff             @@
##             v2.x    #3431      +/-   ##
==========================================
- Coverage   93.51%   84.55%   -8.96%     
==========================================
  Files         222      218       -4     
  Lines        8829     8591     -238     
  Branches        0       33      +33     
==========================================
- Hits         8256     7264     -992     
- Misses        573     1327     +754

Impacted Files	Coverage Δ
integration-tests/ci-visibility/test/sum.js	`69.19% <ø> (ø)`
packages/datadog-instrumentations/src/mocha.js	`0.00% <0.00%> (-98.64%)`	⬇️
packages/datadog-plugin-fastify/src/index.js	`100.00% <ø> (ø)`

... and 336 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

pr-commenter · 2023-07-20T15:52:27Z

Benchmarks

Benchmark execution time: 2023-07-21 14:40:04

Comparing candidate commit 372b481 in PR branch v2.42.0-proposal with baseline commit 4b02e9e in branch v2.x.

Found 1 performance improvements and 32 performance regressions! Performance is the same for 675 metrics, 27 unstable metrics.

scenario:appsec-iast-no-vulnerability-control-14

🟥 cpu_usage_percentage [+5.403%; +8.708%]

scenario:appsec-iast-no-vulnerability-control-16

🟥 cpu_usage_percentage [+5.174%; +7.968%]

scenario:exporting-pipeline-0.4-12

🟥 cpu_user_time [+57.362ms; +63.007ms] or [+24.054%; +26.421%]
🟥 execution_time [+64.941ms; +68.461ms] or [+23.725%; +25.010%]
🟥 instructions [+120.2M instructions; +120.8M instructions] or [+16.300%; +16.385%]
🟥 max_rss_usage [+3.316KB; +3.543KB] or [+6.135%; +6.555%]

scenario:exporting-pipeline-0.4-14

🟥 cpu_user_time [+48.503ms; +53.477ms] or [+19.209%; +21.179%]
🟥 execution_time [+53.615ms; +58.193ms] or [+18.821%; +20.427%]
🟥 instructions [+120.9M instructions; +122.1M instructions] or [+16.161%; +16.323%]
🟥 max_rss_usage [+5.279KB; +5.761KB] or [+10.479%; +11.435%]

scenario:exporting-pipeline-0.4-16

🟥 cpu_user_time [+39.219ms; +44.188ms] or [+15.888%; +17.902%]
🟥 execution_time [+39.703ms; +44.695ms] or [+14.152%; +15.932%]
🟥 instructions [+98.9M instructions; +100.9M instructions] or [+13.597%; +13.869%]
🟥 max_rss_usage [+4.044KB; +4.112KB] or [+7.262%; +7.385%]

scenario:exporting-pipeline-0.4_with_stats-12

🟥 cpu_user_time [+56.866ms; +62.205ms] or [+23.493%; +25.699%]
🟥 execution_time [+61.648ms; +64.705ms] or [+22.327%; +23.434%]
🟥 instructions [+120.3M instructions; +120.9M instructions] or [+16.240%; +16.323%]
🟥 max_rss_usage [+2.839KB; +2.996KB] or [+5.261%; +5.552%]

scenario:exporting-pipeline-0.4_with_stats-14

🟥 cpu_user_time [+45.028ms; +50.759ms] or [+17.665%; +19.913%]
🟥 execution_time [+52.169ms; +56.811ms] or [+18.143%; +19.757%]
🟥 instructions [+120.0M instructions; +121.4M instructions] or [+15.968%; +16.141%]
🟥 max_rss_usage [+5.192KB; +5.654KB] or [+10.333%; +11.251%]

scenario:exporting-pipeline-0.4_with_stats-16

🟥 cpu_user_time [+47.190ms; +52.581ms] or [+19.748%; +22.004%]
🟥 execution_time [+53.953ms; +58.840ms] or [+19.957%; +21.765%]
🟥 instructions [+120.7M instructions; +123.3M instructions] or [+16.983%; +17.339%]
🟥 max_rss_usage [+3.913KB; +3.995KB] or [+7.008%; +7.155%]

scenario:exporting-pipeline-0.5-12

🟥 instructions [+50.8M instructions; +51.1M instructions] or [+8.260%; +8.304%]

scenario:exporting-pipeline-0.5-14

🟥 instructions [+46.6M instructions; +47.0M instructions] or [+7.658%; +7.728%]

scenario:exporting-pipeline-0.5-16

🟥 instructions [+52.6M instructions; +55.2M instructions] or [+8.924%; +9.372%]

scenario:exporting-pipeline-0.5_with_stats-12

🟥 instructions [+50.7M instructions; +51.8M instructions] or [+8.201%; +8.390%]

scenario:exporting-pipeline-0.5_with_stats-14

🟥 instructions [+46.2M instructions; +46.6M instructions] or [+7.549%; +7.619%]

scenario:exporting-pipeline-0.5_with_stats-16

🟥 instructions [+52.4M instructions; +54.9M instructions] or [+8.825%; +9.250%]

scenario:spans-finish-later-14

🟩 max_rss_usage [-10.541KB; -10.277KB] or [-5.233%; -5.102%]

* try to fix flaky test * Update test-gcloud-function.sh * Update test-gcloud-function.sh * Update test-gcloud-function.sh * Update test-gcloud-function.sh

* Add BITBUCKET_GIT_HTTP_ORIGIN env variable * Update CI spec JSONs

#3440)

juan-fernandez

looks good from CI Visibility's perspective

nsavoire and others added 21 commits July 20, 2023 16:51

Taint request URI (#3302)

917f0e5

* Taint request URI * Add check for safe tainted origins on unvalidated redirect analyzer * Change assertion construction for unvalidated redirect analyzer test * Add metric for uri sourcing * Fix PR comments

Fix erroneous resetting of profiler state to undefined (#3394)

bdcb70e

Update yarn.lock with current pprof-nodejs version (#3398)

5f3e851

Forgot to update yarn.lock when bumping pprof-nodejs version in a previous PR.

openai: be defensive about response object shape (#3408)

b2831f0

delete old build scripts (#3396)

f084c98

Add eslint rule for object-curly-newline (#3406)

99c6df0

Fix a phrase in README (#3405)

dfcf8b0

Fix same dependency detection (#3386)

a91b65f

* Fix same dependency detection * Do not send same dependency:version twice * Clear map on stop * Update savedDependencies to savedDependenciesToSend to improve understanding

fix: Handle case where Lambda handler is using callbacks instead of p…

2e10f31

…romises (#3404)

Ignore esbuild output from integration tests in esbuild (#3410)

07f6cda

[ci-visibility] Do not modify how jest reports timeouts (#3399)

b58689b

Allow https datadog agent url in continuous profiling (#3190) (#3194)

a4c3106

Co-authored-by: Attila Szegedi <[email protected]>

Fix distributed tracing issue caused by single span ingestion changes (…

c7e4716

…#3413)

Ensure OTel span name is copied to resource.name (#3412)

2d91953

Make datadog distributed tracing header take priority over traceconte…

bb66f96

…xt (#3414)

Exclude empty telemetry metrics (#3421)

11990d9

openai: make services entirely optional when initializing (#3420)

578d776

Fix service representation config (#3419)

74f5c4b

* address and fix edge cases * move to better tracer option name for service name flattening

implement naming schema for graphql plugin (#3279)

670471b

* implement naming schema for graphql plugin * implement schema v1 and add tests --------- Co-authored-by: Thomas Hunter II <[email protected]> Co-authored-by: Jordi Bertran de Balanda <[email protected]>

szegedi added the semver-minor label Jul 20, 2023

szegedi requested review from a team as code owners July 20, 2023 15:42

szegedi force-pushed the v2.42.0-proposal branch 2 times, most recently from b126583 to 866e007 Compare July 21, 2023 12:56

Fix flaky serverless integration test (#3418)

5f0218a

* try to fix flaky test * Update test-gcloud-function.sh * Update test-gcloud-function.sh * Update test-gcloud-function.sh * Update test-gcloud-function.sh

uurien and others added 5 commits July 21, 2023 16:31

Fix native-iast-taint-tracking version number (#3433)

773bcdc

openai: remove String#replaceAll for Node.js <=14 (#3432)

61c6f31

Add BITBUCKET_GIT_HTTP_ORIGIN env variable (#3401)

2818d67

* Add BITBUCKET_GIT_HTTP_ORIGIN env variable * Update CI spec JSONs

[ci-visibility] Make integration test for jest compatible with node 12 (

0808128

#3440)

v2.42.0

372b481

szegedi force-pushed the v2.42.0-proposal branch from 866e007 to 372b481 Compare July 21, 2023 14:31

juan-fernandez approved these changes Jul 21, 2023

View reviewed changes

tlhunter approved these changes Jul 21, 2023

View reviewed changes

simon-id approved these changes Jul 21, 2023

View reviewed changes

tlhunter merged commit 149f6f9 into v2.x Jul 21, 2023

szegedi deleted the v2.42.0-proposal branch December 19, 2023 09:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

V2.42.0 proposal #3431

V2.42.0 proposal #3431

Uh oh!

szegedi commented Jul 20, 2023 •

edited

Loading

Uh oh!

github-actions bot commented Jul 20, 2023 •

edited

Loading

Uh oh!

codecov bot commented Jul 20, 2023 •

edited

Loading

Uh oh!

pr-commenter bot commented Jul 20, 2023 •

edited

Loading

Uh oh!

juan-fernandez left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

13 participants

V2.42.0 proposal #3431

V2.42.0 proposal #3431

Uh oh!

Conversation

szegedi commented Jul 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Features

Enhancements

Bug fixes

Minor stuff (probably not notable for release notes)

Uh oh!

github-actions bot commented Jul 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overall package size

Dependency sizes

Uh oh!

codecov bot commented Jul 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

pr-commenter bot commented Jul 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

scenario:appsec-iast-no-vulnerability-control-14

scenario:appsec-iast-no-vulnerability-control-16

scenario:exporting-pipeline-0.4-12

scenario:exporting-pipeline-0.4-14

scenario:exporting-pipeline-0.4-16

scenario:exporting-pipeline-0.4_with_stats-12

scenario:exporting-pipeline-0.4_with_stats-14

scenario:exporting-pipeline-0.4_with_stats-16

scenario:exporting-pipeline-0.5-12

scenario:exporting-pipeline-0.5-14

scenario:exporting-pipeline-0.5-16

scenario:exporting-pipeline-0.5_with_stats-12

scenario:exporting-pipeline-0.5_with_stats-14

scenario:exporting-pipeline-0.5_with_stats-16

scenario:spans-finish-later-14

Uh oh!

juan-fernandez left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

13 participants

szegedi commented Jul 20, 2023 •

edited

Loading

github-actions bot commented Jul 20, 2023 •

edited

Loading

codecov bot commented Jul 20, 2023 •

edited

Loading

pr-commenter bot commented Jul 20, 2023 •

edited

Loading