[Otel] More document on how to setup application with Sentry and Otel together

[duongcongtoai]

Summary

The doc of sentry-go already tells how to integration sentry-go with open telemetry sdk, but there is something wrong with this setup that sentry UI keeps losing the context of open telemetry traceID (it records both traceID from otel and its random traceID, and the error event somehow belongs to the unknown traceID).

Expected behavior

When I used client.CaptureException, this exception will be reported to Sentry, along with a trace and a spanID. With the integration with Opentelemetry, I expect this traceID is also a valid traceID in Opentelemetry (in my case it's Jaeger), but this is not. After debugging, it is the traceID generated by this code:
sentryhttp then calls StartSpan

I have played around a bit and found a setup that worked:

import (
	"github.com/go-chi/chi/v5"
	sentry "github.com/getsentry/sentry-go"
	sentryhttp "github.com/getsentry/sentry-go/http"
	sentryotel "github.com/getsentry/sentry-go/otel"
	"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
)

func main() {

	r := chi.NewRouter()
	r.Use(otelMW())
	r.Use(inject())
	r.Use(sentryhttp.New(sentryhttp.Options{}).Handle)
	r.Get("/", func(w http.ResponseWriter, r *http.Request) {
		hub := sentry.GetHubFromContext(r.Context())
		cl, scope := hub.Client(), hub.Scope()
		cl.CaptureException(fmt.Errorf("dummy"), &sentry.EventHint{Context: r.Context()}, scope)
		w.Write([]byte("hello"))
	})
	http.ListenAndServe(":3000", r)
}

func otelMW() func(http.Handler) http.Handler {
	return func(next http.Handler) http.Handler {
		h := otelhttp.NewHandler(next, "", otelhttp.WithSpanNameFormatter(func(_ string, req *http.Request) string {
			return "otelhttp mw " + req.URL.Path
		}))
		return h
	}
}

func inject() func(http.Handler) http.Handler {
	return func(next http.Handler) http.Handler {
		return http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
			sentryotel.NewSentryPropagator().Inject(req.Context(), propagation.HeaderCarrier(req.Header))
			next.ServeHTTP(w, req)
		})
	}
}

Quick explanation:

• inside otelhttp middleware, it call tracer.StartSpan, which will trigger sentrySpanProcessor to creates a mapping between otelID (newlycreated) and a transaction its create internally. And as i read the code, there is only one method that allowing my code to get back that transaction, that is via sentryotel.NewSentryPropagator().Inject()

This looks quite hacky to me, because I have to look into the implementation to find out how it works, it would be nice if you can provide a fully working example

Full code

[cleptric]

Do you want to record exceptions, or what is your ask here?

If yes, it's planned getsentry/team-webplatform-meta#31

[duongcongtoai]

When I used client.CaptureException, this exception will be reported to Sentry, along with a trace and a spanID. With the integration with Opentelemetry like the code above, I expect this traceID is also a valid traceID in Opentelemetry (in my case it's Jaeger), but it is not. After debugging, it is the traceID generated by this code:
sentryhttp then calls StartSpan

The library did create a "SentrySpan" containing the traceID of Opentelemetry via spanprocessor here, but the it is not saved into any context

[cleptric]

This was fixed in #580 and will be released in 0.19.0 this week.