[RFC WIP] Add a new package to enable open tracing

[raymondfeng]

See

• https://github.com/jaegertracing/jaeger-client-node
• https://github.com/opentracing/opentracing-javascript/

The test depends on #3388 (merged).

Checklist

👉 Read and sign the CLA (Contributor License Agreement) 👈

• npm test passes on your machine
• New tests added or existing tests modified to cover all changes
• Code conforms with the style guide
• API Documentation in code was updated
• Documentation in /docs/site was updated
• Affected artifact templates in packages/cli were updated
• Affected example projects in examples/* were updated

👉 Check out how to submit a PR 👈

[emonddr]

Very cool and complicated stuff.

The Basic Use section should provide a description of what default options this component is using, and how it will affect the tracing. Perhaps an example of tracing one request and the output that appears on console or in a log.

[bajtos]

Opentracing provides this introductory material, maybe we should link to that page from our tracing extension's README?

https://opentracing.io/docs/overview/

There is also a guide aimed specifically at framework developers:

https://opentracing.io/docs/best-practices/instrumenting-frameworks/

I need to read those documents first to build a better understanding of the problem domain, before I can provide any meaningful feedback for this pull request.

[bajtos]

This is not in line with OpenTracing guidelines, see e.g. https://opentracing.io/docs/overview/inject-extract/ and https://opentracing-javascript.surge.sh/classes/tracer.html#inject.

[bajtos]

Before we dive into implementation details, I'd like to clarify and reach agreement about the intended user/developer experience and our high-level plan for achieving it.

As I was reading through OpenTracing Overview (including sub-sections), The OpenTracing Semantic Specification and Semantic Conventions and [Best practices for instrumenting frameworks](Instrumenting frameworks), a lot of questions come to my mind.

1. How to extract SpanContext inside our REST server layer? It's a responsibility of Tracer to provide extract function that builds SpanContext from HTTP request header. However, we need to find how to call that code in the right moment of the request handling cycle. Ideally as early as possible. Do we want to bake this into the REST component or allow extensions (like tracing) to implement this part on their own?

2. How to serialize & forward SpanContext when making outgoing calls to backend services, typically via our connectors like loopback-connector-rest and loopback-connector-soap. It's a responsibility of Tracer to provide inject function that builds HTTP request headers to be included in the outgoing request. However, we need to decide how are our connectors going to call that code. Again, are we going to bake this into the connectors or implement an extension point?

3. It's possible to store additional information in SpanContext, this info will be forwarded to backend services, it's called "baggage items". How can we access current Span or SpanContext from Controllers and Repositories, so that we can pick up "baggage items" contributed by callers of our API?

4. OpenTracing specifies a number of well-known span tags, see Semantic Versioning. Do we want to support some of these tags out of the box in the framework? How can users decide which built-in tags they want to set and which should be left out?

   Quoting from Tracing Request Properties:

   Users may also want to track information about the requests without having to manually access the span and set the tags themselves. It’s helpful to provide a way for users to specify properties of the request they want to trace, and then automatically trace these features.

5. How can application developers AND 3rd-party extensions contribute custom span tags, span logs and baggage items; and ensure they are added to spans at the right time of the request life-cycle?

6. How is the application developer going to configure the tracer to be used? Can 3rd-party extensions provide this configuration too, so that the application don't have to? (Think of a company building many microservices that want to share the same tracer setup via a LB4 extension.) How can application and extension code receive the current/global tracer so that they can access the current span and/or create new spans?

7. How can application developers specify which request should be traced and which not? Do we want 3rd-party extensions to be able to contribute this configuration too? Quoting from Specifying Which Requests to Trace:

   Some users may want to trace every request while other may want only specific requests to be traced. You should ideally allow users to set up tracing for either of these scenarios. For example, you could provide @trace annotations/decorators, in which only annotated handler functions have tracing enabled. You can also provide settings for the user to specify whether they’re using these annotations, versus whether they want all requests to be traced automatically.

8. A side note: once we have the first version implemented, do we want to add OpenTracing badge to our README? See Closing remarks.

Now I am not saying we need to answer and implement all of the points listed above. What I am looking for is a high-level overview of our master plan for distributed tracing and then an explanation of where this initial pull request fits into that plan, what is its scope and what's intentionally left for later.

[bajtos]

☝️

[fvisticot]

any news / progress on this feature ?

[lswith]

I've started similar progress to what is here using opentelemetry-js (as it seems to be the current Cloud Native client library).

How to extract SpanContext inside our REST server layer? It's a responsibility of Tracer to provide extract function that builds SpanContext from HTTP request header. However, we need to find how to call that code in the right moment of the request handling cycle. Ideally as early as possible. Do we want to bake this into the REST component or allow extensions (like tracing) to implement this part on their own?

I would suggest that it gets backed into the REST component as the extension points are not called early enough. I've noticed that the spans that are created by the library do not encompass the express middleware and http calls.

How to serialize & forward SpanContext when making outgoing calls to backend services, typically via our connectors like loopback-connector-rest and loopback-connector-soap. It's a responsibility of Tracer to provide inject function that builds HTTP request headers to be included in the outgoing request. However, we need to decide how are our connectors going to call that code. Again, are we going to bake this into the connectors or implement an extension point?

Since opentelemetry-js wraps the http library, any outgoing calls made by this library will forward the SpanContext. Take a look at https://github.com/open-telemetry/opentelemetry-js-contrib/blob/master/examples/postgres/client.js as an example of a client calling code.

[raymondfeng]

I've started similar progress to what is here using opentelemetry-js (as it seems to be the current Cloud Native client library).

Nice. I have been waiting for the opentelemetry to come out.

I would suggest that it gets backed into the REST component as the extension points are not called early enough. I've noticed that the spans that are created by the library do not encompass the express middleware and http calls.

We can simply change the interceptor to be a middleware. It's very straightforward.

Since opentelemetry-js wraps the http library, any outgoing calls made by this library will forward the SpanContext. Take a look at https://github.com/open-telemetry/opentelemetry-js-contrib/blob/master/examples/postgres/client.js as an example of a client calling code.

That's cool as we don't have to mess around individual connectors.

@lswith Are you willing to contribute a patch for proposed enhancements? I have too many things on my plate at the moment.

[lswith]

@raymondfeng I'm happy to take over. I'm going to wait for this issue: open-telemetry/opentelemetry-js#752 to resolve though, as it is required before we can use spans across async/await.

[stale]

This pull request has been marked stale because it has not seen activity within two months. It will be closed within 14 days of being stale unless there is new activity.

[stale]

This pull request has been closed due to continued inactivity. If you are interested in finishing the proposed changes, then feel free to re-open this pull request or open a new one.

[shashwatmi]

Any Update on this? Or we Would love to pick it up as we need this for our internal Project. @lswith @raymondfeng

[nflaig]

Any Update on this? Or we Would love to pick it up as we need this for our internal Project. @lswith @raymondfeng

I have just published loopback4-tracing, you might wanna take a look at it.

Few notes about the package

• opentelemetry dependencies are a bit outdated
• currently only supports jaeger as it is the only exporter, should be straight forward to adds others like otlp grpc exporter
• tests are still all passing, but haven't tested it further since I last worked / used it (over 1 year ago)

With some additional testing and a few minor updates this should be easy to make production ready again. I am definitely open to contributions to the package, you can fork it, or even just get some inspiration from different parts.

Anyways, hope this helps one way or the other to improve loopback <> tracing support.

[shashwatmi]

@nflaig Hey! thanks for sharing, I am looking into, i'll dive in and push any PR, if needed.
Thanks again for it! i will keep you posted.