feat: make OpenTelemetry an optional dependency

[r0x0d]

Move opentelemetry-api from required dependencies to optional extras, allowing docket to be used without OpenTelemetry installed. When OTel is not available, the library silently uses noop stub implementations that maintain the same interface but perform no operations.

This change is particularly helpful for Linux distribution packagers, especially in Fedora, where the OpenTelemetry Python packages have faced significant packaging challenges. As discussed in the Fedora devel mailing list thread [1], the opentelemetry-* packages have been difficult to maintain and may be orphaned or retired, making it problematic for packages that hard-depend on them.

By making OpenTelemetry optional, docket can be packaged and used in distributions without requiring the full OpenTelemetry stack, while users who want observability features can still install them via:

pip install pydocket[telemetry]   # for tracing only
pip install pydocket[metrics]     # for metrics + tracing

Implementation details:

• Add _otel_stubs.py with noop implementations of OTel interfaces
• Add _otel.py as unified import point with automatic fallback
• Update all source files to import from _otel module
• Move opentelemetry-api to new 'telemetry' optional extra
• Update 'metrics' extra to depend on 'telemetry'
• Add conftest.py to skip instrumentation tests when OTel unavailable
• All 601 tests pass, 90% coverage maintained

[1] https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/3LHITWOBP76JNSQICDDX5BHFVFCFS4NA/

[r0x0d]

Hi, @chrisguidry! Not sure if this is something that you will find useful or valuable, but this patch would help a lot of packaging docket in downstream distributions.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 25f376be0f

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chrisguidry]

Hi @r0x0d, thank you so much for the PR! I'm trying to decide how I feel about this. It was definitely problematic when Docket had a dependency on the opentelemetry-sdk, which was true up until a few releases ago. But now Docket's only default dependency is on opentelemetry-api, whose whole point is to be the lightweight no-op in-memory stub implementation.

Could you help me understand more about the Fedora packaging issue? Would relaxing the version constraint help? I'm not opposed to vendoring some OTEL stubs, but it feels like a pretty disappointing step to take.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 35.35354% with 128 lines in your changes missing coverage. Please review.
✅ Project coverage is 96.88%. Comparing base (00d78c8) to head (f7d288d).

Files with missing lines	Patch %	Lines
src/docket/_otel_stubs.py	0.00%	124 Missing ⚠️
src/docket/_otel.py	88.57%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #320      +/-   ##
==========================================
- Coverage   98.65%   96.88%   -1.77%     
==========================================
  Files         102      104       +2     
  Lines       10161    10311     +150     
  Branches      491      491              
==========================================
- Hits        10024     9990      -34     
- Misses        121      303     +182     
- Partials       16       18       +2     

Flag	Coverage Δ
python-3.10	96.19% <35.35%> (-2.47%)	⬇️
python-3.11	96.09% <35.35%> (-1.22%)	⬇️
python-3.12	96.19% <35.35%> (-2.47%)	⬇️
python-3.13	96.19% <35.35%> (-2.47%)	⬇️
python-3.14	96.17% <35.35%> (-2.47%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
src/docket/_telemetry.py	100.00% <100.00%> (ø)
src/docket/docket.py	100.00% <100.00%> (ø)
src/docket/execution.py	100.00% <100.00%> (ø)
src/docket/instrumentation.py	100.00% <100.00%> (ø)
src/docket/worker.py	100.00% <100.00%> (ø)
tests/instrumentation/test_counters.py	100.00% <100.00%> (ø)
tests/instrumentation/test_export.py	100.00% <100.00%> (ø)
tests/instrumentation/test_tracing.py	100.00% <ø> (ø)
src/docket/_otel.py	88.57% <88.57%> (ø)
src/docket/_otel_stubs.py	0.00% <0.00%> (ø)

... and 5 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[r0x0d]

Hi @r0x0d, thank you so much for the PR! I'm trying to decide how I feel about this. It was definitely problematic when Docket had a dependency on the opentelemetry-sdk, which was true up until a few releases ago. But now Docket's only default dependency is on opentelemetry-api, whose whole point is to be the lightweight no-op in-memory stub implementation.

Could you help me understand more about the Fedora packaging issue? Would relaxing the version constraint help? I'm not opposed to vendoring some OTEL stubs, but it feels like a pretty disappointing step to take.

Hi, @chrisguidry! Thanks for being open on the idea.

So, the difficulties we have with OTEL right now is managing the updates from version to version, especially the protobuf ones that are super outdated, and it seems that folks have a hard time trying to update it to newer versions 1.

My goal here (and maybe my PR is not reflecting it directly), is to have an option to disable OTEL entirely when building in Fedora until things normalize, and we can unretire the OTEL packages again. Making it optional, for us, would decrease the complexity of packaging.

The upstream docket could (and should) still make OTEL default and that how it should be, but for downstream, a way to toggle it off would be much appreciated.

Giving a bit of context here; I'm trying to get https://github.com/rhel-lightspeed/linux-mcp-server/ to Fedora repositories, in which, it depends on fastmcp==2.14.x, that depends on docket.

Maybe there's an easier way to handle this, and if so, please let me know and I would love to work on it.

Thanks for your attention!

[chrisguidry]

Ahh okay, I think you may be seeing the dependencies from before I introduced the metrics extra. Here's the current tree:

$ uv tree --no-dev
Using CPython 3.12.8 interpreter at: /home/chris/.pyenv/versions/3.12.8/bin/python
Resolved 127 packages in 0.59ms
pydocket
├── cloudpickle v3.1.1
├── croniter v6.0.0
│   ├── python-dateutil v2.9.0.post0
│   │   └── six v1.17.0
│   └── pytz v2025.2
├── fakeredis[lua] v2.33.0
│   ├── redis v7.1.0
│   ├── sortedcontainers v2.4.0
│   └── lupa v2.6 (extra: lua)
├── opentelemetry-api v1.39.1
│   ├── importlib-metadata v8.7.0
│   │   └── zipp v3.23.0
│   └── typing-extensions v4.15.0
├── prometheus-client v0.24.1
├── py-key-value-aio[memory, redis] v0.3.0
│   ├── beartype v0.22.5
│   ├── py-key-value-shared v0.3.0
│   │   ├── beartype v0.22.5
│   │   └── typing-extensions v4.15.0
│   ├── cachetools v6.2.1 (extra: memory)
│   └── redis v7.1.0 (extra: redis)
├── python-json-logger v4.0.0
├── redis v7.1.0
├── rich v14.2.0
│   ├── markdown-it-py v4.0.0
│   │   └── mdurl v0.1.2
│   └── pygments v2.19.2
├── typer v0.21.1
│   ├── click v8.3.0
│   ├── rich v14.2.0 (*)
│   ├── shellingham v1.5.4
│   └── typing-extensions v4.15.0
├── typing-extensions v4.15.0
└── opentelemetry-sdk v1.39.1 (extra: metrics)
    ├── opentelemetry-api v1.39.1 (*)
    ├── opentelemetry-semantic-conventions v0.60b1
    │   ├── opentelemetry-api v1.39.1 (*)
    │   └── typing-extensions v4.15.0
    └── typing-extensions v4.15.0
(*) Package tree already displayed

This doesn't bring in the heavier-weight opentelemetry-* packages like opentelemetry-distribution, which brings in our old friend enemy protobuf. Even the metrics extra only brings in a few slim opentelemetry packages. The main reason I split it off was because opentelemetry-sdk still has that problematic exact pin for opentelemetry-api that can cause problems for people.

Are you testing with the current version of Docket (0.17.5)? I wonder if the problem is that we still have a <0.17 pin in fastmcp? Can you show me what you get for a uv tree --no-dev on your project?

[r0x0d]

Are you testing with the current version of Docket (0.17.5)?

Yeah, Fedora requires that we always ship the latest version of a library/software.

I wonder if the problem is that we still have a <0.17 pin in fastmcp?

Seems like it. https://github.com/jlowin/fastmcp/blob/v2.14.4/pyproject.toml#L13, but, in the upcoming 3.0, pydocket will become optional, which kinda helps, but they have introduced opentelemetry-api as required dependency 😅.

I will need to have the same conversation with fastmcp community once 3.0 lands, so 😅… It's a more general problem. Unfortunately, I don't think there is a timeline for when it will come out of beta, that's why I'm pursuing this update in here, because if I can get pydocket build/shipping in Fedora, I can get fastmcp 2.14 (current stable) shipped as well.

Ahh okay, I think you may be seeing the dependencies from before I introduced the metrics extra

It's not that making the opentelemetry-sdk optional is not helpful, it is just that, opentelemetry-python-* was indeed a package in Fedora before, but due to the complications of maintaining it (due to the protobuf lib) the former maintainer just retired the package. Due to the composition of https://github.com/open-telemetry/opentelemetry-python being a monorepo, we package it as a single source, but splitting on each workspace member (e.g; python-opentelemetry-api, python-opentelemetry-proto, python-opentelemetry-...)

If I unretire the package in Fedora, that means I will need to support all that was shipped before, and that includes the opentelemetry-{*,proto} etc... The build of opentelemetry seems a bit complicated, and I don't think that my team has the capacity right now to support it, unfortunately, that's why I'm coming here to see the possibility of making the whole opentelemetry-python-* optional, so that we can bring this library to more people in downstream distributions.

[chrisguidry]

Ha, well that "community" that introduced opentelemetry-api to fastmcp is actually me :D

What we're doing in fastmcp and docket here are very standard, innocuous, and correct uses of the OTEL python libraries. The whole idea is that library maintainers build directly against opentelemetry-api (which is intentionally lightweight and has low dependencies) and then people deploying systems to be instrumented bring opentelemetry-sdk, the exporters they need for their environment, etc.

I'd like to help you get this fixed at the source, which seems to be Fedora's packaging of the opentelemetry Python libraries. If we don't fix this at the source, then I think you'll just keep having this problem in general. Solving it here and making Docket and FastMCP more complex is a last resort.

It sounds to me like the core problem is that Fedora's packages for opentelemetry-python are bundled as a monolith, so you can't get opentelemetry-api without also getting opentelemetry-distro (which has the OTLP implementation, which needs protobuf).

So first, is every Python library in existence supposed to be represented separately in Fedora? Like, did someone make a docket or fastmcp Fedora package? If not, how is your project establishing its dependency on fastmcp (and thus docket)? Would that not also work for opentelemetry-api, to have like a "private" dependency of your project? Why do you need to depend on the Fedora opentelemetry-python packages at all? Forgive my ignorance about Fedora packaging, I'm more of a Debian/Ubuntu user so I'm not as familiar.

[chrisguidry]

Maybe let's back up and start at the core problem you're hitting when you try to build the linux-mcp-server Fedora package?

[chrisguidry]

This release Jeremiah just cut of FastMCP 2.x should definitely help: https://github.com/jlowin/fastmcp/releases/tag/v2.14.5

[r0x0d]

Hi, @chrisguidry! I took it back with the former maintainer of OTEL package in Fedora, and he gave me a workaround that I could probably just package otel-api instead of the whole workspace.

I have opened a package review in Fedora for it. If that works, then this won't be needed anymore.

[chrisguidry]

Ooo that's great news! Mind linking to it here?

[r0x0d]

https://bugzilla.redhat.com/show_bug.cgi?id=2436472 here is the bugzilla for the package review. Hopefully someone will be able to take a look at this soon.

[chrisguidry]

Oo it looks like you're making some progress with the Fedora packages!

Thanks for raising this issue and thanks for hearing me out. Let me know (you can reach out to me directly by the email on my profile too) if I can help with anything else relating to this. For now I'm going to close this PR.

[r0x0d]

Oh, thanks! I forgot to come here to close this.