Capture Wire Protocol design goals #21

tigrannajaryan · 2019-07-18T17:48:17Z

This is a follow up from Spec SIG meeting on Jul 18, 2019, recapping
the topics that I mentioned in the meeting.

The document will help us design the right wire protocol.

This is a follow up from Spec SIG meeting on Jul 18, 2019, recapping the topics that I mentioned in the meeting. The document will help us design the right wire protocol.

ccaraman · 2019-07-18T18:00:11Z

specification/proto-design-goals.md

+
+- Be suitable for use between all of the following node types: instrumented applications, telemetry backends, local agents, stand-alone collectors/forwarders.
+
+- Have high reliability of data delivery and clear visibility when the data cannot be delivered.


for "data cannot be delivered" - is this the only type of errors?

This is one of the problems that we encountered in production particularly with OpenCensus protocol. This requirement can be refined further if there is an evidence of other common error types.

songy23 · 2019-07-18T18:45:00Z

specification/proto-design-goals.md

@@ -0,0 +1,19 @@
+# Design Goals for OpenTelemetry Wire Protocol


Should backwards compatibility be part of the goals?

Do you mean backwards compatibility with any existing protocol? I think that should be a non-goal.

If you mean backwards compatibility within its own revisions then I think it goes without saying. That should be a goal of any protocol that can be deployed on a variety of devices at different times by different people. We can call this out specifically if needed.

I mean within its own versions. Agreed it may already be implied and not need to point it specifically.

songy23

LGTM

songy23 · 2019-07-18T19:15:36Z

specification/proto-design-goals.md

@@ -0,0 +1,19 @@
+# Design Goals for OpenTelemetry Wire Protocol


I mean within its own versions. Agreed it may already be implied and not need to point it specifically.

pjanotti · 2019-07-18T19:27:50Z

specification/proto-design-goals.md

+
+- Have high reliability of data delivery and clear visibility when the data cannot be delivered.
+
+- Have low CPU usage for serialization and deserialization.


nit: I prefer resource instead of CPU since we are at high-level.

I think it is worth listing CPU and Memory separately. They are related but somewhat distinct with possibly different solutions. Especially the memory, I wanted to mention it in a separate paragraph so that we realize the specific pattern of allocating and using memory that some protocols tend to require can be quite unfriendly to GCs.

yurishkuro · 2019-07-18T23:49:26Z

specification/proto-design-goals.md

@@ -0,0 +1,19 @@
+# Design Goals for OpenTelemetry Wire Protocol
+
+We want to design a telemetry data exchange protocol that has the following characteristics:


is this intended to be a single protocol for different types of telemetry?

My past experience is with traces and logs (but not metrics). I think the answer should be yes, one protocol for all types, unless we think other telemetry types have significantly different requirements for the protocol. Thoughts are welcome.

I think this makes sense... there is a possible issue with the fact that metrics systems do not consume traces, and there are perhaps emerging standards for wire protocols in that space. How do we play well with prometheus, etc? Or is that out of scope? AKA we will always be doing some data conversion/processing in a gateway node between the telemetry network and the metrics backend.

tedsuo · 2019-07-19T16:03:54Z

specification/proto-design-goals.md

+
+- Allow backpressure signalling.
+
+- Be load-balancer friendly (do not hinder re-balancing).


At what granularity are we load balancing? Per span?

Typically per batch of spans. L7 load balancers trigger rebalancing in specific situations. For gRPC we found that it typically happens when gRPC stream is opened. By controlling what you include in one stream you can control the frequency of rebalancing.
Doing it per span is likely going to be prohibitively expensive.

reyang · 2019-07-19T16:53:52Z

specification/proto-design-goals.md

+
+- Be suitable for use between all of the following node types: instrumented applications, telemetry backends, local agents, stand-alone collectors/forwarders.
+
+- Have high reliability of data delivery and clear visibility when the data cannot be delivered.


How do we think about de-duplicating the data? (e.g. in case of transmission failures, where sender has no idea whether the backend accepted the data or not)

I'll have this in the RFC. In short, I suggest to aim for reliable delivery at the cost of possible duplicates. Details in the RFC.

Thank you, I'll look into the RFC.

tigrannajaryan · 2019-07-22T18:50:39Z

Reviewers, if there are no objections about these goals, let's merge this.

tigrannajaryan · 2019-07-23T14:47:14Z

Reviewers, I need more comments or one more approval to merge this.

reyang

LGTM

tigrannajaryan · 2019-07-23T17:51:33Z

Approvers, please merge.

Capture Wire Protocol design goals

e2e40fc

This is a follow up from Spec SIG meeting on Jul 18, 2019, recapping the topics that I mentioned in the meeting. The document will help us design the right wire protocol.

tigrannajaryan requested review from AloisReitbauer, bogdandrutu, c24t, carlosalberto, iredelmeier, SergeyKanzhelev, songy23, tedsuo and yurishkuro as code owners July 18, 2019 17:48

ccaraman reviewed Jul 18, 2019

View reviewed changes

songy23 reviewed Jul 18, 2019

View reviewed changes

songy23 approved these changes Jul 18, 2019

View reviewed changes

pjanotti approved these changes Jul 18, 2019

View reviewed changes

yurishkuro reviewed Jul 18, 2019

View reviewed changes

tedsuo reviewed Jul 19, 2019

View reviewed changes

reyang reviewed Jul 19, 2019

View reviewed changes

reyang approved these changes Jul 23, 2019

View reviewed changes

yurishkuro merged commit 2ccb7cb into open-telemetry:master Jul 23, 2019

tigrannajaryan deleted the feature/tigran/designgoals branch July 23, 2019 22:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Capture Wire Protocol design goals #21

Capture Wire Protocol design goals #21

tigrannajaryan commented Jul 18, 2019

ccaraman Jul 18, 2019

tigrannajaryan Jul 18, 2019

songy23 Jul 18, 2019

tigrannajaryan Jul 18, 2019

songy23 Jul 18, 2019

songy23 left a comment

songy23 Jul 18, 2019

pjanotti Jul 18, 2019

tigrannajaryan Jul 18, 2019 •

edited

Loading

yurishkuro Jul 18, 2019

tigrannajaryan Jul 19, 2019

tedsuo Jul 19, 2019

tedsuo Jul 19, 2019

tigrannajaryan Jul 19, 2019

reyang Jul 19, 2019

tigrannajaryan Jul 19, 2019

reyang Jul 19, 2019

tigrannajaryan commented Jul 22, 2019

tigrannajaryan commented Jul 23, 2019

reyang left a comment

tigrannajaryan commented Jul 23, 2019


		- Be suitable for use between all of the following node types: instrumented applications, telemetry backends, local agents, stand-alone collectors/forwarders.

		- Have high reliability of data delivery and clear visibility when the data cannot be delivered.

		@@ -0,0 +1,19 @@
		# Design Goals for OpenTelemetry Wire Protocol


		- Have high reliability of data delivery and clear visibility when the data cannot be delivered.

		- Have low CPU usage for serialization and deserialization.

		@@ -0,0 +1,19 @@
		# Design Goals for OpenTelemetry Wire Protocol

		We want to design a telemetry data exchange protocol that has the following characteristics:


		- Allow backpressure signalling.

		- Be load-balancer friendly (do not hinder re-balancing).

Capture Wire Protocol design goals #21

Capture Wire Protocol design goals #21

Conversation

tigrannajaryan commented Jul 18, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

songy23 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tigrannajaryan Jul 18, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tigrannajaryan commented Jul 22, 2019

tigrannajaryan commented Jul 23, 2019

reyang left a comment

Choose a reason for hiding this comment

tigrannajaryan commented Jul 23, 2019

tigrannajaryan Jul 18, 2019 •

edited

Loading