feat: use dd-trace-go v2 #206

joeyzhao2018 · 2025-08-01T17:10:05Z

What does this PR do?

This PR will update this repo to use dd-trace-go v2 instead of v1
Make the Client-side trace stats computation default off unless DD_TRACE_STATS_COMPUTATION_ENABLED explicitly set.

Motivation

#204

Implementation Details

"gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer" => "github.com/DataDog/dd-trace-go/v2/ddtrace/tracer"
ddtrace.Span => *tracer.Span
Previously an interface now a struct
spanContext.TraceID() returns a string now. Previously uint64
spanContext.SamplingPriority() directly available. Previously not available in Spancontext interface
functionExecutionSpan.Context() still works the same
type FinishConfig struct remains the same. ddtrace.FinishConfig => tracer.FinishConfig
Though tracer.ChildOf(psc) is deprecated, it's still used here because the span is not available here for us to use Span.StartChild. In "Serverless Universal Instrumentation", that span is created in the agent instead of the tracer.

Testing Guidelines

Updated the unittests
Updated the integration tests
Smoke-tested in a real lambda.
- Tested both case for using and not using universal instrumentation for a hello world lambda
- Tested the case where the upstream tracecontext exists. (Note: it seems that there's an existing issue with the upstream lambda's trace. Need to confirm before merging.)

Additional Notes

Types of changes

Bug fix
New feature
Breaking change
Misc (docs, refactoring, dependency upgrade, etc.)

Checklist

This PR's description is comprehensive
This PR contains breaking changes that are documented in the description
This PR introduces new APIs or parameters that are documented and unlikely to change in the foreseeable future
This PR impacts documentation, and it has been updated (or a ticket has been logged)
This PR's changes are covered by the automated tests
This PR collects user input/sensitive content into Datadog

happynancee · 2025-08-01T18:07:00Z

dd-trace-go v2.1.0 enables client-side stats computation by default. Can we turn it off by default?

joeyzhao2018 · 2025-08-01T18:50:01Z

dd-trace-go v2.1.0 enables client-side stats computation by default. Can we turn it off by default?

Updated accordingly.

# Conflicts: # go.mod # go.sum # internal/extension/extension.go

happynancee · 2025-08-04T14:49:23Z

@joeyzhao2018 i think we also need to disable telemetry by default too DataDog/dd-trace-go#3808

magnetikonline · 2025-08-08T03:13:08Z

That would be fantastic @happynancee if we could also land that change 👍

i think we also need to disable telemetry by default too DataDog/dd-trace-go#3808

joeyzhao2018 · 2025-08-27T03:31:47Z

That would be fantastic @happynancee if we could also land that change 👍

i think we also need to disable telemetry by default too DataDog/dd-trace-go#3808

I explicitly set DD_INSTRUMENTATION_TELEMETRY_ENABLED to false in the tracer initialization code block.

magnetikonline · 2025-08-27T06:09:36Z

I explicitly set DD_INSTRUMENTATION_TELEMETRY_ENABLED to false in the tracer initialization code block.

Thanks @joeyzhao2018 - can see the change in 4d70949. 👍

joeyzhao2018 · 2025-08-28T20:29:11Z

internal/extension/extension.go

-			} else {
-				logger.Error(fmt.Errorf("could not get sampling priority from getSamplingPriority()"))
-			}
+			logger.Error(fmt.Errorf("could not get sampling priority from spanContext.SamplingPriority()"))


spanContext.SamplingPriority is directly available in dd-trace-go v2. No longer need the handling for the adapter case

purple4reina

This all looks great!

One question. For a customer who is currently using the v1 tracer in their lambda code, how will upgrading to this new version affect them? Will their spans all of a sudden be orphaned? Will their code even compile?

This is an important question for go because our customers will rely heavily on adding their own custom instrumentation to their functions.

joeyzhao2018 · 2025-08-29T14:25:40Z

One question. For a customer who is currently using the v1 tracer in their lambda code, how will upgrading to this new version affect them? Will their spans all of a sudden be orphaned? Will their code even compile?

To clarify, this is indeed a breaking change as many of the APIs are different in dd-trace-go V2. We will need to bump the major version when releasing it next time and reference the migration guide. But it shouldn't cause any orphaned spans or any feature-wise issues.

This is an important question for go because our customers will rely heavily on adding their own custom instrumentation to their functions.

Again, the customers do need to migrate and adapt to the new APIs. However, since we don't release go tracer as layers, releasing a datadog-lambda-go v2 doesn't block us to do any fix in v1 if needed.

purple4reina · 2025-08-29T19:50:55Z

One question. For a customer who is currently using the v1 tracer in their lambda code, how will upgrading to this new version affect them? Will their spans all of a sudden be orphaned? Will their code even compile?

To clarify, this is indeed a breaking change as many of the APIs are different in dd-trace-go V2. We will need to bump the major version when releasing it next time and reference the migration guide. But it shouldn't cause any orphaned spans or any feature-wise issues.

This is an important question for go because our customers will rely heavily on adding their own custom instrumentation to their functions.

Again, the customers do need to migrate and adapt to the new APIs. However, since we don't release go tracer as layers, releasing a datadog-lambda-go v2 doesn't block us to do any fix in v1 if needed.

FYI, doing a major version bump in golang is VERY different from other languages. Read about it here. It requires copying your entire source into a new v2 directory.... This is why I am thinking it might be best to just move all of our code to dd-trace-go. If you want to use ddtrace v1, you use this repo, but if you want to use ddtrace v2, you've gotta use the lambda contrib found in dd-trace-go.v2.

joeyzhao2018 · 2025-08-29T21:34:40Z

FYI, doing a major version bump in golang is VERY different from other languages. Read about it here.

Thank you for sharing this! This is very helpful. After thinking more about it and some research, I think we actually have to stay as v1 following go's convention. Because there's no breaking change in the public apis for all the lambda wrappers. In other words, it will be backward compatible.

What I said earlier was inaccurate by the go standard/convention. I think what caused the confusion is that our team has been adapting a stricter definition of "breaking change" when using the semantic versioning. Mainly, we want to communicate any "big" changes clearly to the customers, such as behavior changes or dependency updates etc. But for this PR's change, clear warning in the release notes should suffice.

As for migrating the dd-trace-go, we are still doing it and probably going to use Orchestrion as a few customers have already requested it. I don't think this PR is blocked by that migration though.
Let me know if I was wrong, or inaccurate, ... again 🤦‍♂️

purple4reina · 2025-09-02T16:17:45Z

Sadly, bumping to dd-trace-go v2 won't be backward compatible. Consider the following customer app where they are using dd-trace-go v1.73.1:

package main

import (
	"context"
	"time"

	ddlambda "github.com/DataDog/datadog-lambda-go"
	"github.com/aws/aws-lambda-go/events"
	"github.com/aws/aws-lambda-go/lambda"
	"gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer"
)

func main() {
	lambda.Start(ddlambda.WrapFunction(myHandler, nil))
}

func myHandler(ctx context.Context, event events.APIGatewayProxyRequest) (string, error) {
	span, ctx := tracer.StartSpanFromContext(ctx, "hello-rey")
	defer span.Finish()
	time.Sleep(time.Second)
	return "ok", nil
}

Using the current version of datadog-lambda-go, they will see a trace like this, with a single custom span:

However, after updating to use this branch and without any code changes in the handler, there are now two tracers installed: v1 which the customer is using to create their custom span and v2 which is now being used by datadog-lambda-go. This change will lead to orphaned spans and a trace that looks like:

Now, there is some nuance here, which you have already pointed out. If we instead were to create a new v2 of datadog-lambda-go, and the customer uses the same handler file as I shared above, they will still get the same result.

You've also pointed out correctly that the API defined in datadog-lambda-go isn't changing with this PR.

So why then should we do a v2 release? Because the changes proposed in this PR will (most likely) require customers to make code changes to their functions. If they are currently actively using v1 in their code, their traces will be broken if they upgrade to this branch.

You're right. There isn't a breaking change to our api. Customer code will still compile. However, I believe it is best practice for us to introduce a v2 here. If customers want their functions to behave exactly as they did with dd-trace-go v1, then there are code changes they will have to make. If I have to update my code when upgrading a package in order to maintain current functionality, then that is by definition a breaking change which requires a major version bump.

I do additionally think this major version change offers us an opportunity we should take. As we've talked about several times before, we'd like to eventually do away with datadog-lambda-go and migrate everything to dd-trace-go instead. This v2 version bump provides us with an opportunity to do that.

magnetikonline · 2025-09-29T03:39:12Z

So TL;DR @joeyzhao2018 we should expect to see this PR canned/closed and a new upcoming github.com/DataDog/dd-trace-go/contrib/aws/datadog-lambda-go/v2 package to be released, allowing users to migrate over to this package?

This is how I read DataDog/dd-trace-go#3943?

epot · 2025-10-13T09:56:17Z

Is there an update on this? We have a medium CVE that we'd like to get rid of (CVE-2020-8911) but we cannot until this library uses the tracer v2 library.

joeyzhao2018 requested a review from a team as a code owner August 1, 2025 17:10

joeyzhao2018 changed the title ~~adapting to dd-trace-go v2~~ feat: use dd-trace-go v2 Aug 1, 2025

joeyzhao2018 closed this Aug 1, 2025

joeyzhao2018 reopened this Aug 1, 2025

joeyzhao2018 added 6 commits August 1, 2025 20:29

adapting to dd-trace-go v2

2d0172a

# Conflicts: # go.mod # go.sum # internal/extension/extension.go

fix lint

b013185

fix lint

388aceb

fix lint

25512fa

disable client side stats compute by default

81e52b5

use lower64 bit for the trace id

5eaac2d

joeyzhao2018 force-pushed the joey/ddtracegoV2 branch from eed3640 to 5eaac2d Compare August 2, 2025 00:32

log error if SamplingPriority not available

b026979

joeyzhao2018 added 3 commits August 26, 2025 23:07

Merge branch 'main' into joey/ddtracegoV2

212a26b

getSamplingPriority no longer needed

a5ca0c2

disable instrumentation telemetry

4d70949

magnetikonline mentioned this pull request Aug 27, 2025

[BUG]: dd-trace-go/v2 internal/telemetry when used in AWS Lambda environment posts repeatedly to non-exist agent sidecar DataDog/dd-trace-go#3808

Open

joeyzhao2018 commented Aug 28, 2025

View reviewed changes

purple4reina reviewed Aug 28, 2025

View reviewed changes

joeyzhao2018 requested a review from purple4reina August 29, 2025 18:32

joeyzhao2018 added 2 commits August 29, 2025 17:07

change back to go 1.24.0

7f72765

integration tests use 1.24 too

85176e5

joeyzhao2018 mentioned this pull request Sep 24, 2025

feat: datadog-lambda-go DataDog/dd-trace-go#3943

Merged

7 tasks

Uh oh!

feat: use dd-trace-go v2 #206

Are you sure you want to change the base?

feat: use dd-trace-go v2 #206

Uh oh!

Conversation

joeyzhao2018 commented Aug 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Motivation

Implementation Details

Testing Guidelines

Additional Notes

Types of changes

Checklist

Uh oh!

happynancee commented Aug 1, 2025

Uh oh!

joeyzhao2018 commented Aug 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

happynancee commented Aug 4, 2025

Uh oh!

magnetikonline commented Aug 8, 2025

Uh oh!

joeyzhao2018 commented Aug 27, 2025

Uh oh!

magnetikonline commented Aug 27, 2025

Uh oh!

joeyzhao2018 Aug 28, 2025

Choose a reason for hiding this comment

Uh oh!

purple4reina left a comment

Choose a reason for hiding this comment

Uh oh!

joeyzhao2018 commented Aug 29, 2025

Uh oh!

purple4reina commented Aug 29, 2025

Uh oh!

joeyzhao2018 commented Aug 29, 2025

Uh oh!

purple4reina commented Sep 2, 2025

Uh oh!

magnetikonline commented Sep 29, 2025

Uh oh!

epot commented Oct 13, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

joeyzhao2018 commented Aug 1, 2025 •

edited

Loading

joeyzhao2018 commented Aug 1, 2025 •

edited

Loading