[saltas888]

FastAPI 0.143.0 traces requests on its own, and Infrahub must not record extra spans

This PR upgrades FastAPI from 0.141.1 to 0.143.0. Operators who turn tracing on in Infrahub receive the same spans from the API server and the task workers as before the upgrade.

Jira: INBOX-364

How tracing works in Infrahub, and what FastAPI 0.142.0 changes

When an operator turns tracing on, Infrahub records a span for each step of its work and sends the spans to a trace collector. A span is a timed record of one step: what ran, when it started, how long it took and whether it failed. For example, each API request records a request span.

• Up to FastAPI 0.141.1, FastAPI recorded no spans itself. The Infrahub API is traced by a separate OpenTelemetry package, opentelemetry-instrumentation-fastapi, which server.py adds to the API when it starts.
• Since FastAPI 0.142.0, FastAPI also traces requests itself (fastapi/fastapi#16403). It does this in every FastAPI app that runs in a process where tracing is turned on. For each request, it records one span for the request and one span for each step inside it: resolving dependencies, running the endpoint and, when the endpoint returns data, serializing the response.

The Release Radar bot reported that the upgrade would double every request's spans in Infrahub. The effect differs for each Infrahub process:

• API server: no change in spans. FastAPI does not trace a request when it finds the separate OpenTelemetry package in the app, and server.py adds that package before the API handles its first request. This PR also turns FastAPI's own telemetry off in server.py, so the API no longer depends on FastAPI finding the package. Each API request still records one request span, and FastAPI records none. The new tests in test_server.py check this, also with the separate package removed from the app.
• Task workers: three new spans for each health check, which this PR drops. With --with-healthcheck, each task worker also runs a small web server from Prefect that answers GET /health (prefect/workers/server.py). Prefect creates that server as a plain FastAPI app, so Infrahub cannot add the separate package to it or turn FastAPI's tracing off on it. When tracing is on, the worker sets it up for its whole process, so on 0.143.0 every health check would record three spans: the request, resolving dependencies and running the endpoint.
• Prefect server: no change. It runs in its own container. Infrahub does not turn tracing on in that container, and Prefect 3.8.6 does not either, so FastAPI records no spans there.

Changes

• fastapi 0.141.1 → 0.143.0 in pyproject.toml and uv.lock. No package is added or removed.
• The API turns FastAPI's built-in tracing, metrics and logs off: server.py passes telemetry={"tracing": False, "metrics": False, "logs": False} to FastAPI(...), with a comment saying why.
• Task workers drop the spans of Prefect's health checks.
  • A sampler is the part of the tracing setup that decides which spans to keep. A new sampler, ExcludeServerPathsSampler in backend/infrahub/trace.py, drops the request span of every GET /health.
  • It also drops every span whose parent span in the same process was dropped, so the spans inside a health check are dropped too, whatever OTEL_TRACES_SAMPLER is set to.
  • Every other decision still follows OTEL_TRACES_SAMPLER, so operators who set that variable keep their sampling.
  • Only the task workers use this sampler.
• New knowledge page dev/knowledge/backend/observability.md: how the API server and the task workers produce traces, metrics and logs, why FastAPI's built-in telemetry stays off, and how the local observability stack collects them. It is linked from backend/AGENTS.md and dev/README.md.
• New tests:
  • backend/tests/unit/test_server.py checks that one API request records one request span from the separate OpenTelemetry package and no span from FastAPI's own tracing, and that FastAPI records no span even when the separate package is removed from the app.
  • backend/tests/unit/test_trace.py checks that a /health request records no spans and that other requests record all their spans, with the default sampler, always_on and traceidratio.

No changelog fragment: traces contain the same spans as on 0.141.1.

How this was verified

• The API test passes on both 0.141.1 and 0.143.0. It fails when the instrument_app call in server.py is removed.
• The test with the separate package removed from the app fails when the telemetry setting is removed from server.py.
• The sampler tests fail when the sampler is changed to never drop. The always_on and traceidratio cases also fail without the check on the parent span.
• The 0.143.0 breaking change does not apply (fastapi/fastapi#16476). Since 0.143.0, FastAPI sets up exporters, the part that sends spans to a collector, only when FASTAPI_OTEL_AUTO_CONFIGURE is set, and Infrahub does not set it.
• Prefect 3.8.6 accepts fastapi>=0.139.0,<1.0.0.
• Local checks passed on the stable-based branch: ruff, mypy, the generated-file checks and 2782 backend unit tests.

uv run pytest backend/tests/unit/test_server.py backend/tests/unit/test_trace.py -v

Spans recorded in a local stack, before and after the upgrade

The development stack was run with tracing turned on and the spans sent to Tempo, with an image built from this branch. Only two things changed between runs: the FastAPI version, switched with uv pip install inside the server and task-worker containers, and the Infrahub code, either this branch or the commit it was then based on. The runs were done before this branch moved from develop to stable, on top of develop commit 62b99b2507, and before server.py turned FastAPI's telemetry off. stable and develop pin the same FastAPI, Prefect and OpenTelemetry versions, and the branch's commits apply to stable without changes. That setting changes no span, because FastAPI already skipped its tracing in the API. Each run sent 5 GET /api/config requests and 5 GraphQL queries (BuiltinTag { count }) to the API, and 5 health checks to each of the 2 task workers. The spans were then read back from Tempo.

Run	FastAPI	Infrahub code	Spans per GET /api/config	Spans per GraphQL query	Spans per task-worker health check
Before the upgrade	0.141.1	base	4	19	0
Upgrade without this PR's code	0.143.0	base	4	19	3
This PR	0.143.0	this branch	4	19	0

• The API recorded the same spans in all three runs, and every span came from the separate OpenTelemetry package or from Infrahub's own tracing. For GET /api/config, these are the request span and three http send spans. FastAPI recorded no span of its own in the API.

• Without this PR's code, every health check on a task worker recorded three spans from FastAPI. Prefect's health check returns its response directly, so FastAPI records no serialization span for it.

  GET /health
  ├── fastapi.dependencies
  └── fastapi.endpoint
  

• With OTEL_TRACES_SAMPLER=always_on on the task workers and FastAPI 0.143.0, this PR before the check on the parent span (commit 981a11f4a2) still recorded fastapi.dependencies and fastapi.endpoint for every health check, without their request span. With the check (commit 73555e5a48), health checks record no span. In that run, the worker's background tasks, such as "Sync Git Repositories" and "Detect failed merges", were recorded with the same spans as before the upgrade.

Follow-up, not in this PR

A longer-term option, moving the API to FastAPI's built-in tracing and removing the separate package, is described in this comment.

🤖 Generated with Claude Code

[saltas888]

🤖 platform-health-agent · CI green, awaiting human review · 2026-10-31

details

• Jira card: INBOX-364. The card holds the full analysis.
• This comment is updated in place on later passes. Review threads are answered inline.

History:

• 2026-10-31: head a0d768b460 unchanged. The Oct 16 re-run of backend-docker-integration (a) passed, so CI is green: 46 passed, 12 skipped, none failed. The stale workflow labelled the PR after 14 days without activity; that workflow never closes PRs. No human review yet.
• 2026-10-16 10:00 UTC: head a0d768b460. The re-run of backend-docker-integration (a) failed on test_composite_artifacts, a timing race in the test that this PR does not touch. Re-ran the job again. Details on the Jira card. No human review yet.
• 2026-10-16 09:00 UTC: head a0d768b460. backend-docker-integration (a) failed before any test ran: the Docker stack did not start (RabbitMQ queue state in that runner). Not caused by this PR; shard (b) passed on the same commit. Re-ran the job. Details on the Jira card. No human review yet.
• 2026-10-16 08:00 UTC: head a0d768b460. The card owner rebased the branch onto stable (the PR now targets stable) and added 73555e5a48 (health-check spans dropped whatever OTEL_TRACES_SAMPLER is), 1346822cd2 (telemetry= off for the API) and a0d768b460 (dev/knowledge/backend/observability.md). Verified locally: ruff, mypy and ty clean; 25 server and trace tests pass; the new telemetry test fails when the setting is removed. CI: 27 passed, 12 skipped, 16 still running, none failed. No human review yet.
• 2026-10-15 14:01 UTC: head a25d2c6ded unchanged. CI finished: 44 passed, 13 skipped, cubic neutral, none failed. The PR description was rewritten at 13:08 UTC and now includes the local Tempo results. No human review yet.
• 2026-10-15 12:00 UTC: head a25d2c6ded (new commit from the card owner: the sampler also drops spans whose local parent was not sampled). Verified locally: ruff, mypy and ty clean, 24 trace and server tests pass, and the new tests fail when the parent check is removed. CI: 40 passed, 14 skipped, 3 still running, none failed. No human review yet.
• 2026-10-12 12:00 UTC: head 6123b5693d. 43 checks passed; one integration shard did not run (runner unavailable) and was re-run. cubic: no issues. No human review yet.
• 2026-10-12: rebased on develop (11 new commits). Resolved one conflict in backend/infrahub/workers/infrahub_async.py by keeping both constants: SUBMISSION_WINDOW_CAPACITY from develop and PREFECT_WORKER_HEALTHCHECK_PATH from this PR. 3527 backend unit tests passed locally. Head 6123b5693d.
• 2026-10-12: pushed d2cab0c49b and cffb6920cb, requested by the card owner. Task workers now drop the trace spans of Prefect's health checks. The API test now also asserts that FastAPI records no spans of its own. The PR description is updated. Local /pre-ci passed (3508 backend unit tests).
• 2026-10-12 09:00 UTC: checked head 7359e61e37. 36 checks passed, 15 skipped, 8 still running (component, integration and E2E shards), none failed. The cubic review found no issues. No human review yet.
• 2026-10-12: PR opened. Local /pre-ci passed, including 3502 backend unit tests.

[cubic-dev-ai]

No issues found across 3 files

Confidence score: 5/5

• Automated review surfaced no issues in the provided summaries.
• No files require special attention.

_{You’re at about 92% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.}

<sub>Shadow auto-approve: would require human review. Upgrades FastAPI to 0.143.0 and pins one server span per API request with a test; however the upgrade introduces new built-in tracing spans on Prefect worker health checks and leaves open whether to disable that tracing. Human sign-off is needed for that operational telemetry decision.

View guided diff | Re-trigger cubic</sub>

[codspeed]

Merging this PR will not alter performance

✅ 13 untouched benchmarks

---

_{Comparing pha/INBOX-364 (a0d768b) with stable (73e8303)}

[cubic-dev-ai]

0 issues found across 4 files (changes from recent commits).

Confidence score: 5/5

• Automated review surfaced no issues in the provided summaries.
• No files require special attention.

_{You’re at about 93% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.}

<sub>Shadow auto-approve: would auto-approve. Upgrades FastAPI to 0.143.0 while preserving prior tracing: a new sampler drops built-in spans on task-worker /health checks, and tests pin one contrib server span per API request. Bounded, restores old span behavior, with unit tests.

View guided diff | Re-trigger cubic</sub>

[github-actions]

This PR has been inactive for 2 weeks and has been marked as stale.

If you're still working on this, please:

• Push new commits, or
• Leave a comment to indicate this PR is still active

The stale label will be removed automatically when there's new activity.