Proposal: otel4s-sdk repo

[armanbilge]

I propose we create a sibling repository "otel4s-sdk" (name ideas happily accepted) to develop pure Scala implementations of the otel4s API, such as the exporter suggested in #171. An independent repository is necessary to avoid creating a circular dependency between otel4s and http4s.

This SDK would be:

1. lightweight, without Java deps i.e. gRPC/Netty;
2. high-performance, by leveraging the Cats Effect runtime; and
3. cross-platform, and support Scala.js and Scala Native.

@sherriesyt will work on implementing the Tracing API (her GSoC project was just accepted 🎉 ) and @kovstas has volunteered to work on the Metrics API. @iRevive, @ChristopherDavenport, @bpholt, @mpilquist, @christiankjaer, and @fredshonorio have all expressed interest in the project as well and helping to mentor our new contributors :)

[rossabaker]

Is this the end state? (Boxes are repos.) I'm assuming the circular dependency comes from http4s backends being instrumented by otel4s-api, and otel4s-sdk uses an http4s backend in its HTTP exporters.

graph BT;
    A[otel4s]
    B[http4s] --> A 
    C[otel4s-sdk] --> B
    C --> A
    D[otel4s-java] --> A

Loading

It's hard to implement a moving target like otel4s-api in a separate repo. If we can defer instrumenting http4s backends with otel4s-api until the latter is more stable, there's no circular dependency, and it will be more efficient to get the SDKs bootstrapped in the otel4s repo.

graph BT;
    A[otel4s] --> B
    B[http4s] 

Loading

Then, once otel4s-api is more stable, we can spin off both SDKs and get to the first diagram?

[armanbilge]

I'm assuming the circular dependency comes from http4s backends being instrumented by otel4s-api, and otel4s-sdk uses an http4s backend in its HTTP exporters.

Yes, that's right.

It's hard to implement a moving target like otel4s-api in a separate repo. If we can defer instrumenting http4s backends with otel4s-api until the latter is more stable, there's no circular dependency, and it will be more efficient to get the SDKs bootstrapped in the otel4s repo.

I 💯 agree and glad you think so as well. If everyone is in agreement, I think this will work out much better!

[ChristopherDavenport]

Here is a minimal exemplar of a pure backend.

https://github.com/ChristopherDavenport/otel4s-local

Happy to PR it in if that's the direction desired for a pure SDK.

[ChristopherDavenport]

In a follow-up. I have working and useful references in my fork of natchez-http4s-otel for otel4s. I am happy to find a community space for us to work with, as I'd like to get it published so folks can work with otel4s in real-world scenarios.

https://github.com/christopherdavenport/natchez-http4s-otel/tree/otel2

It seems like a very useful tool both in the interrim as this gains adoption(prior to otel4s embedding into backends), and as a reference of what an implementation for Client/Server tracing and TextMapPropagation via clients.

Thoughts?

[rossabaker]

It could fit as a module here with the proposed interim dependency tree (e.g., this repo -> http4s). It touches a lot of bases for a general library instrumentation and can help test the API for fitness. It would then spin off at the same time as the SDKs.

[rossabaker]

I'm traveling this week and not keeping up well, but what's the rationale for the sdk branch? Particularly if it's touching non-sdk code (e.g., #240), we risk developing two incompatible cores.

[armanbilge]

but what's the rationale for the sdk branch

Well, it started as a tracing-sdk branch, then the scope increased 😅 The intention is to give @sherriesyt a branch to target with smaller, incremental PRs for her project. I guess we could send those to main, but enable the NoPublishPlugin on her modules until they are ready?

[rossabaker]

Yeah, as long as we're keeping it green, small PRs to the main will be easier in the long run, and NoPublishPlugin is a good compromise if we're not ready to publish.

[iRevive]

The current dependency graph:

graph BT;
    A[otel4s-core]
    B[http4s] -..-> A 
    C[otel4s-sdk] 
    E[otel4s-sdk-exporter] --> C
    E[otel4s-sdk-exporter] --> B
    C --> A
    D[otel4s-oteljava] --> A

Loading

Only otel4s-sdk-exporter requires http4s.