feat(observability): add OpenTelemetry + Jaeger distributed tracing (Rec #8) - #1232
Conversation
Rec #8 - Distributed Tracing - OTLP receivers on gRPC :4317 and HTTP :4318 - Batch processor (1024 spans, 5s timeout) - Memory limiter (400MiB limit, 100MiB spike) - Jaeger exporter for trace visualization - Prometheus exporter for trace metrics (:8889) - Traces and metrics pipelines configured
Rec #8 - Distributed Tracing - Jaeger all-in-one service (UI :16686, collector :14268/:14250) - OpenTelemetry Collector service (OTLP gRPC :4317, HTTP :4318) - Both services on pmoves_monitoring network - Standard tier resource limits (1.0 CPU, 512M memory) - HEALTHCHECK on both services - Security: read_only, tmpfs, no-new-privileges
Rec #8 - Distributed Tracing - setup_tracing() initializes OTel SDK with OTLP gRPC exporter - get_tracer() returns named tracer (no-op fallback when OTel not installed) - trace_nats_message() context manager for NATS message processing spans - trace_http_request() context manager for outbound HTTP request spans - inject_trace_headers() / extract_trace_headers() for W3C trace propagation - All OTel imports wrapped in try/except ImportError for opt-in behavior - No-op tracer/span classes for graceful degradation
Rec #8 - Distributed Tracing - TracingMiddleware creates OTel spans for every inbound HTTP request - Extracts traceparent from incoming headers for upstream span linking - Records http.method, http.url, http.status_code, duration - Sets ERROR status on 5xx responses and exceptions - Adds X-Trace-Id response header for debugging - No-op when opentelemetry packages not installed - Compatible with existing Prometheus metrics
Rec #8 - Distributed Tracing - Jaeger datasource pointing to jaeger:16686 query service - Proxy access mode for Grafana-to-Jaeger communication - Traces-to-metrics linkage with Prometheus datasource - Node graph visualization enabled - Span bar shows http.method tag
Rec #8 - Distributed Tracing - traced_publish() injects W3C traceparent into NATS message headers - make_traced_callback() wraps subscription handlers with trace extraction - Extracts trace context from incoming NATS headers, creates child spans - Links consumer spans to producer spans across service boundaries - Backward compatible — headers are optional, tracing imports guarded - Updated __all__ with new exports (traced_publish, make_traced_callback)
|
Warning Rate limit exceeded
Your organization is not enrolled in usage-based pricing. Contact your admin to enable usage-based pricing to continue reviews beyond the rate limit, or try again in 57 minutes and 1 seconds. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (6)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Docker Hardening ValidationHardening Validation ReportValidated: Tue Apr 14 00:58:24 UTC 2026Services CheckedPMOVES.AI Docker Hardening Validation[INFO] Checking: pmoves/docker-compose.hardened.yml [INFO] Validating: hi-rag-gateway-v2 [INFO] Validating: extract-worker [INFO] Validating: langextract [INFO] Validating: presign [INFO] Validating: render-webhook [INFO] Validating: retrieval-eval [INFO] Validating: pdf-ingest [INFO] Validating: jellyfin-bridge [INFO] Validating: invidious-companion-proxy [INFO] Validating: ffmpeg-whisper [INFO] Validating: media-video [INFO] Validating: media-audio [INFO] Validating: hi-rag-gateway-v2-gpu [INFO] Validating: hi-rag-gateway-gpu [INFO] Validating: deepresearch [INFO] Validating: supaserch [INFO] Validating: publisher-discord [INFO] Validating: mesh-agent [INFO] Validating: nats-echo-req [INFO] Validating: nats-echo-res [INFO] Validating: publisher [INFO] Validating: analysis-echo [INFO] Validating: graph-linker [INFO] Validating: comfy-watcher [INFO] Validating: grayjay-plugin-host [INFO] Validating: agent-zero [INFO] Validating: archon [INFO] Validating: channel-monitor [INFO] Validating: pmoves-yt [INFO] Validating: notebook-sync [INFO] Validating: supabase_service_role_key [INFO] Validating: supabase_jwt_secret ====================================== |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f5206ebb21
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| # Inside a request handler the middleware creates the parent span; | ||
| # child spans are created automatically via ``tracer.start_span()``. | ||
| "" |
There was a problem hiding this comment.
The opening module docstring is closed with "" instead of """, which leaves the file syntactically invalid (python -m py_compile fails) and prevents services.common.tracing from being imported at all, so none of the new tracing utilities can run.
Useful? React with 👍 / 👎.
|
|
||
| Compatible with existing Prometheus metrics — this middleware does not | ||
| interfere with ``prometheus_fastapi_instrumentator`` or similar. | ||
| "" |
There was a problem hiding this comment.
This module has the same malformed top-level docstring terminator (""), causing an unterminated triple-quoted string syntax error on import; as a result, FastAPI services cannot load TracingMiddleware when this file is referenced.
Useful? React with 👍 / 👎.
| volumes: | ||
| - ./config/otel/otel-collector-config.yaml:/etc/otelcol/config.yaml:ro | ||
| ports: | ||
| - "${OTEL_GRPC_PORT:-4317}:4317" |
There was a problem hiding this comment.
The collector defaults to publishing gRPC on host port 4317, but Jaeger is already hard-bound to 4317 in the same compose overlay; with default env values, docker compose -f docker-compose.yml -f docker-compose.tracing.yml up -d hits a port-allocation conflict and one service fails to start.
Useful? React with 👍 / 👎.
| ctx = extract_trace_headers(msg_headers) | ||
| tracer = get_tracer(service_name) | ||
|
|
||
| with trace_nats_message(subject, msg.data) as span: |
There was a problem hiding this comment.
make_traced_callback extracts traceparent into ctx, but the callback starts a new span without ever attaching or passing that context, so consumer spans are not parented to producer spans and cross-service NATS traces fragment into separate traces.
Useful? React with 👍 / 👎.
- Fix malformed docstrings in tracing.py and tracing_middleware.py (terminated with "" instead of """, causing SyntaxError) - Resolve OTLP gRPC port conflict between Jaeger (4317) and otel-collector (now defaults to 4327) in compose overlay - Pass extracted trace context to span creation in nats_client.py so consumer spans are properly parented to producer spans Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Docker Hardening ValidationHardening Validation ReportValidated: Tue Apr 14 17:17:37 UTC 2026Services CheckedPMOVES.AI Docker Hardening Validation[INFO] Checking: pmoves/docker-compose.hardened.yml [INFO] Validating: hi-rag-gateway-v2 [INFO] Validating: extract-worker [INFO] Validating: langextract [INFO] Validating: presign [INFO] Validating: render-webhook [INFO] Validating: retrieval-eval [INFO] Validating: pdf-ingest [INFO] Validating: jellyfin-bridge [INFO] Validating: invidious-companion-proxy [INFO] Validating: ffmpeg-whisper [INFO] Validating: media-video [INFO] Validating: media-audio [INFO] Validating: hi-rag-gateway-v2-gpu [INFO] Validating: hi-rag-gateway-gpu [INFO] Validating: deepresearch [INFO] Validating: supaserch [INFO] Validating: publisher-discord [INFO] Validating: mesh-agent [INFO] Validating: nats-echo-req [INFO] Validating: nats-echo-res [INFO] Validating: publisher [INFO] Validating: analysis-echo [INFO] Validating: graph-linker [INFO] Validating: comfy-watcher [INFO] Validating: grayjay-plugin-host [INFO] Validating: agent-zero [INFO] Validating: archon [INFO] Validating: channel-monitor [INFO] Validating: pmoves-yt [INFO] Validating: notebook-sync [INFO] Validating: supabase_service_role_key [INFO] Validating: supabase_jwt_secret ====================================== |
1 participant
Summary
Implements Recommendation #8 — Add distributed tracing to the PMOVES.AI platform.
Adds full OpenTelemetry + Jaeger stack with opt-in instrumentation modules.
Components Added
tracing.pytracing_middleware.pyDeploy
Design
Related: Review Recommendation #8, PMOVES-P6