Distributed Tracing: Producer OpenTelemetry instrumentation by fedeoliv · Pull Request #1838 · confluentinc/confluent-kafka-dotnet

Fernando de Oliveira (fedeoliv) · 2022-06-14T14:34:22Z

This PR introduces a distributed tracing instrumentation for the Producer by leveraging the use of Activity objects with tags mapping the OpenTelemetry messaging attributes specification.

The implementation is inspired by the great discussion on issue #1269 and the Mikel Blanchard (@CodeBlanch)'s proposal on the PR #1278.

…ing attributes

…hods

…bles

…er activity

CLAassistant · 2022-06-14T14:53:01Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

Fernando de Oliveira (fedeoliv) · 2022-06-22T17:53:28Z

Matt Howlett (@mhowlett) I'd love your feedback on this implementation proposal 😊

Matt Howlett (mhowlett) · 2022-06-22T18:14:57Z

i'd love to have time to give it to you.. i'll ping product about this feature.

Mikel Blanchard (CodeBlanch) · 2022-06-22T20:04:12Z

+                if (activity == null)
+                    return null;
+
+                using (activity)


Why have this using? Won't that cause a stopped Activity to be returned?

Mikel Blanchard (CodeBlanch) · 2022-06-22T20:07:07Z

+
+                using (activity)
+                {
+                    activity?.AddDefaultOpenTelemetryTags(topicPartition, message);


Don't need the null-conditional here (activity?) because line 41 already verified it is non-null.

Mikel Blanchard (CodeBlanch) · 2022-06-22T20:07:37Z

+            TopicPartition topicPartition,
+            Message<TKey, TValue> message)
+        {
+            activity?.AddTag(OpenTelemetryMessaging.SYSTEM, "kafka");


I don't think you need any of the null-conditionals in here because this is only ever called for non-null activity.

Mikel Blanchard (CodeBlanch) · 2022-06-22T20:23:15Z

+            Message<TKey, TValue> message)
+        {
+            activity?.AddTag(OpenTelemetryMessaging.SYSTEM, "kafka");
+            activity?.AddTag(OpenTelemetryMessaging.DESTINATION, topicPartition.Topic);


Generally we use Activity.SetTag as opposed to AddTag. SetTag implements the OTel spec logic for null behavior, de-dupe, etc.

open-telemetry/opentelemetry-dotnet#5173 Based on this, we may be able to suggest AddTag is certain scenarios as well to save some perf.

Mikel Blanchard (CodeBlanch) · 2022-06-22T20:36:55Z

+                int messagePayloadBytes = Encoding.UTF8.GetByteCount(message.Value.ToString());
+                activity?.AddTag(OpenTelemetryMessaging.MESSAGE_PAYLOAD_SIZE_BYTES, messagePayloadBytes.ToString());


Is message.Value.ToString() going to be very expensive? Ideally getting the size would either be cheap or it would be an opt-in thing for users who need it enough to pay for it.

messagePayloadBytes.ToString() why not just push the numeric value? Basically you can add an int as a tag it doesn't need to be forced into a string.

at the very least, check activity.IsAllDataRequested before doing anything which is non-trivial in terms of cost. This ensures that the cost is paid only if the activity has a chance of being exported to some telemetry destination.

Mikel Blanchard (CodeBlanch) · 2022-06-22T20:37:33Z

+
+            internal static Activity Start<TKey, TValue>(TopicPartition topicPartition, Message<TKey, TValue> message)
+            {
+                Activity activity = ActivitySource.StartActivity(ActivityName);


A bit of an unknown here. StartActivity is when sampling happens. The OpenTelemetry specification states...

The API documentation MUST state that adding attributes at span creation is preferred to calling SetAttribute later, as samplers can only consider information already present during span creation.

In this code there are no attributes/tags on start so users won't be able to make interesting sampling decisions based on details about the spans/activity instances. This is kind of an ongoing discussion, not sure if there will eventually be a list of attributes that will be required or it will always be optional. Just sharing the state of things 😄

/cc Cijo Thomas (@cijothomas)

For things which are known already, I'd suggest passing it to StartActivityCall.

One example for sql, where the db system is known (and static), and is passed at startActivity time itself.
https://github.com/open-telemetry/opentelemetry-dotnet/blob/main/src/OpenTelemetry.Instrumentation.SqlClient/Implementation/SqlClientDiagnosticListener.cs#L64

Cijo Thomas (cijothomas) · 2022-06-22T20:46:39Z

+        /// </summary>
+        internal static class Producer
+        {
+            private const string ActivityName = ActivitySourceName + ".MessageProduced";


ActivitySourceName should not be prefixed for ActivityName.

Mikel Blanchard (CodeBlanch) · 2022-06-22T20:47:20Z

+            }
+        }
+
+        private static Activity AddDefaultOpenTelemetryTags<TKey, TValue>(


This is another area with some unknowns. The semantic conventions are still marked as Experimental. Meaning: They are likely to change. There is this whole concept of a schema url that should be attached to instrumentation before it can be declared stable. One option could be we create the Activity here and then provide a callback instrumentation (such as OpenTelemetry) or users can use in order to add the tags that make sense for their domain. The nice thing about that approach would be Confluent Kafka doesn't need to worry about a spec that is changing 😄 That approach (more or less) is what we have done with dotnet itself. The runtime starts things but OpenTelemetry uses hooks in its instrumentation to add the tags according to the spec.

Cijo Thomas (cijothomas) · 2022-06-22T20:47:43Z

+{
+    /// <summary>
+    /// Provides the OpenTelemetry messaging attributes.
+    /// The complete list of messaging attributes specification is available here: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/semantic_conventions/messaging.md#messaging-attributes


the conventions in OTel are still work-in-progress. I'd suggest to mark this overall feature as experimental/non-stable, from the perspective that the name of the tags could change in the future, until the otel conventions become stable.

Cijo Thomas (cijothomas) · 2022-06-22T20:49:35Z

+            // Prepare the activity events listener
+            string activityName = "Confluent.Kafka.MessageProduced";
+            ActivityEventsRecorder eventsRecorder = new(activityName);
+            ActivityListener listener = eventsRecorder.BuildActivityListener();


Another alternative way to test would be to use OpenTelemetry's InMemoryExporter. Example : https://github.com/open-telemetry/opentelemetry-dotnet/blob/main/test/OpenTelemetry.Instrumentation.SqlClient.Tests/SqlClientTests.cs#L90

Mikel Blanchard (CodeBlanch) · 2022-06-22T20:50:25Z

Thanks for this Fernando de Oliveira (@fedeoliv)! I did a quick review and left some comments about where we are with the spec, etc. Happy to help out however I can.

AlphonseKurian1618 · 2022-06-28T18:13:44Z

                    ex);
            }

+            Activity activity = Diagnostics.Producer.Start(topicPartition, message);


Use using and remove activity.stop in finally?

AlphonseKurian1618 · 2022-06-28T18:14:19Z

+                Sample = (ref ActivityCreationOptions<ActivityContext> _) => ActivitySamplingResult.AllData,
+                ActivityStarted = activity =>
+                {
+                    if (activity.DisplayName == activityName)


string.equals for string comparison

Anton (chertby) · 2022-08-24T18:15:57Z

Does it look like development has stopped?
If there's time pressure, I could try to continue if Fernando de Oliveira (@fedeoliv) don't mind?

Juan David Correa (Juandavi1) · 2023-02-04T15:47:06Z

Matt Howlett (@mhowlett) It's been 5 months, is there any update on whether and how Kafka is going to support distributed tracing?

olissa (sookeke) · 2023-02-08T20:05:22Z

Folks, any update on this? it's been months Matt Howlett (@mhowlett)

Sergiy Ivasenko (iiwaasnet) · 2023-02-13T13:52:42Z

Is context propagation implemented as well? Maybe I missed it, but I can't find any code using propagator.Inject()/propagator.ExtractTraceIdAndState()...

Giorgio Galante (GiorgioG) · 2023-05-09T14:12:32Z

Can we get an update?

smurariu · 2023-05-29T15:58:51Z

Anchit Jain (@anchitj) any news on this?

Fernando de Oliveira (fedeoliv) added 10 commits June 10, 2022 10:41

feature: Producer activities with tags targeting OpenTelemetry messag…

48a6968

…ing attributes

refactor: unused activity method removed

8dcca15

feature: OpenTelemetry activity added on Produce and ProduceAsync met…

af3cfa3

…hods

fix: Kafka partition tag value converted to string

e629b6d

test: Integration test for the producer instrumentation

e85c38f

refactor: OpenTelemetry messaging attributes as reusable static varia…

0a71ede

…bles

docs: summary added to Diagnostics and ActivityEventsRecorder methods

a70edf4

feature: OpenTelemetry message payload size attribute added to Produc…

8a8fc71

…er activity

fix: librdkafka path removed from library initialization

55e4745

fix: librdkafka dll inclusion on project binaries removed

d34ba4e

Fernando de Oliveira (fedeoliv) added 2 commits June 14, 2022 13:18

fix: System.Diagnostics.DiagnosticSource package upgrade to 5.0.1

001596d

fix: message value null validation on Producer activity

6fa9ed1

Fernando de Oliveira (fedeoliv) mentioned this pull request Jun 14, 2022

Distributed tracing story with Confluent Kafka client #1269

Open

Matt Howlett (mhowlett) added the enhancement label Jun 22, 2022

Mikel Blanchard (CodeBlanch) reviewed Jun 22, 2022

View reviewed changes

Cijo Thomas (cijothomas) reviewed Jun 22, 2022

View reviewed changes

Mikel Blanchard (CodeBlanch) reviewed Jun 22, 2022

View reviewed changes

Cijo Thomas (cijothomas) reviewed Jun 22, 2022

View reviewed changes

AlphonseKurian1618 reviewed Jun 28, 2022

View reviewed changes

Anton (chertby) mentioned this pull request Aug 29, 2022

[DRAFT] feat: Add distributed tracing #1883

Closed

4 tasks

Eric Erhardt (eerhardt) mentioned this pull request Dec 15, 2023

Add ConfluentKafka instrumentation open-telemetry/opentelemetry-dotnet-contrib#1493

Merged

3 tasks

		int messagePayloadBytes = Encoding.UTF8.GetByteCount(message.Value.ToString());
		activity?.AddTag(OpenTelemetryMessaging.MESSAGE_PAYLOAD_SIZE_BYTES, messagePayloadBytes.ToString());

Conversation

Fernando de Oliveira (fedeoliv) commented Jun 14, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

CLAassistant commented Jun 14, 2022 • edited by cla-assistant Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Fernando de Oliveira (fedeoliv) commented Jun 22, 2022

Uh oh!

Matt Howlett (mhowlett) commented Jun 22, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Mikel Blanchard (CodeBlanch) commented Jun 22, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Anton (chertby) commented Aug 24, 2022

Uh oh!

Juan David Correa (Juandavi1) commented Feb 4, 2023

Uh oh!

olissa (sookeke) commented Feb 8, 2023

Uh oh!

Sergiy Ivasenko (iiwaasnet) commented Feb 13, 2023

Uh oh!

Giorgio Galante (GiorgioG) commented May 9, 2023

Uh oh!

smurariu commented May 29, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

12 participants

Fernando de Oliveira (fedeoliv) commented Jun 14, 2022 •

edited

Loading

CLAassistant commented Jun 14, 2022 •

edited by cla-assistant Bot

Loading