Support for opentelemetry

[hohwille]

As a devon4j user, I want to have opentracing support so that I can monitor my IT landscape better and rely on open standards supported by many tools rather than proprietary correlation-ID.

In devonfw we have created the concept of the correlation-ID which is nice. However devonfw is comitted to open-standards and meanwhile something simlar and even better has be come a kind of standard called opentracing opentelemetry:

• https://opentracing.io/
• https://opentelemetry.io/
• https://www.datadoghq.com/blog/instrument-opencensus-opentracing-and-openmetrics/

We can have a look at existing libs like jaeger but the functionally is so simple that we can implement this oursevles in the existing code.
Finally, we should provide some downward compatibility for correlation-ID, so e.g. the X-Correlation-ID header should still be honored if present and no opentracing headers are present.

[hohwille]

Links to current implementation and behaviour:

• devon4j/modules/logging/src/main/java/com/devonfw/module/logging/common/impl/DiagnosticContextFilter.java

  Line 90 in c8f25cb

  private void setCorrelationId(ServletRequest request) {

• devon4j/modules/logging/src/main/java/com/devonfw/module/logging/common/impl/DiagnosticContextFacadeImpl.java

  Line 31 in 9b8f946

  MDC.put(LoggingConstants.CORRELATION_ID, correlationId);

• devon4j/modules/service/src/main/java/com/devonfw/module/service/common/impl/header/ServiceHeaderCustomizerCorrelationId.java

  Line 42 in 9b8f946

  String correlationId = MDC.get(LoggingConstants.CORRELATION_ID);

• You can even have a look at the log-files of any devon4j app

For open-tracing there is already something in kafka:

• https://github.com/devonfw/devon4j/blob/bdd0c44c53c1ff91fb853e1aa5c12f245cb256c1/modules/kafka/src/main/java/com/devonfw/module/kafka/common/messaging/trace/impl/MessageTraceSupport.java

[maihacke]

I did some more research on this, there are multiple solutions available. I propose that we incorporate the relativly new Opentracing API.

1. Related to this there is out-of-the-box support in Apache CXF: http://cxf.apache.org/docs/using-opentracing.html
2. Addtionally we need to investigate if we really need a fully featured opentracing implementation like Jaeger. I think a simplicstic approach which puts the tracing information into the logs / MDC would be suffient. Users of Devon4j should still be able to replace the opentracing implementation with something more complex, if they need the features.
3. If found an alternative to spring-cloud-sleuth (which I think we were using in devon4j-kafka), which might fit better to an overal approach: https://github.com/opentracing-contrib/java-spring-cloud

I think a good next step would be to create a POC for that. The goal would be to reuse existing implementations (1-3) and minimze our own effort in implementing a tracing solution. The solution should be as simple as possible.
And showcasing:

1. Client doing http/rest call to server
2. Server accepts call, and creates kafka message
3. Server consumes kafka message

From 1-3 the tracing headers should be created/keept out-of-the-box, following the opentracing spec. E.g. we have 1 trace with at least 3 spans here (one for each step).

[hohwille]

Besides Kafka there are other places where we also should integrate and test open-tracing. At least the service-client is an important such place and it has multiple implementations (one using CXF - see link from @maihacke in 1. above, another using Java HTTP Client). Further we can investigate how other projects use OpenAPI to generate clients - they might also then need a way to support opentracing. Might be nice to have, but at least we should aks in the community and collect experience/ideas if available.

[maihacke]

Yes, the service client should be definitely part of the PoC. My idea was to use the Kafka example app as base for the PoC. It has already testcases which use rest, but I‘m not sure if they use the service client. But I‘m sure that could be easily achieved if its not already the case.

[hohwille]

IMHO we need more alignment and clarification on this (see also PR #278):

• shall we use a library for open-tracing or implement our own minimal solution?
• what about libraries already in use that already support open-tracing?
• shall we write the data directly via logging (MDC) or do we need an abstraction that can be replaced so data can be send to jaeger or other systems.
• how and where should we demonstrate the new feature?

[hohwille]

devonfw / devon4j is comitted to open-standards so we should consider this:
https://github.com/eclipse/microprofile-opentracing#microprofile-opentracing

[maihacke]

IMHO we need more alignment and clarification on this (see also PR #278):

• shall we use a library for open-tracing or implement our own minimal solution?

--> Use opentracing-api and library (probably, see below)

• what about libraries already in use that already support open-tracing?

--> there are some and we are already using one in the kafka modul (brave)

• shall we write the data directly via logging (MDC) or do we need an abstraction that can be replaced so data can be send to jaeger or other systems.

--> out of the box definitly yes. But the goal would be to use a library which allows to change this by (spring) configuration. My idea is that our modules/examples use the opentracing-api and out of the box, the tracing information will go to the logs, and mdc is set correctly. This is the standard simple solution. for more advanced (cloud) deployments where a tracing system like zipkin or jaeger is in place it should be easy to change the configuration so that mdc is still set, but tracing events go to the tracing system.

• how and where should we demonstrate the new feature?

-->It should be part of every example if we make it right, since the logs should contain the tracing information then. additionally we could provide an example for integration jaeger.

[hohwille]

actually not so bad that we have been slow on this one. OpenTracing is meanwhile superseded by https://opentelemetry.io/

[maihacke]

That‘s quite interesting. Thanks for the hint.

[baumeister25]

In my opinion we should have something for opentelemetry at least in our guide.
Do you think a module or starter is necessary?
We have already a solution for opentelemetry. Do you think that this one is feasible? https://devonfw.com/website/pages/solutions/solution/monitoring_openTelemetry/