Repository navigation
feat(observability): bootstrap OTel SDK + Prometheus metrics + Pino-style logger #4974
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
KooshaPari
wants to merge
1
commit into
diegosouzapw:release/v3.8.34
from
KooshaPari:koosha/feat/observability-otel-bootstrap
Closed
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,28 @@ | ||
| # OpenTelemetry env vars (PR-001 default) | ||
| # Leave OTEL_EXPORTER_OTLP_ENDPOINT unset to disable span export (default). | ||
| # OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318 | ||
| # OTEL_EXPORTER_OTLP_PROTOCOL=http/json | ||
| # OTEL_EXPORTER_OTLP_HEADERS=x-api-key=secret,x-tenant=42 | ||
| # OTEL_TRACES_SAMPLER_ARG=1 | ||
| # OTEL_BSP_MAX_QUEUE_SIZE=2048 | ||
| # OTEL_BSP_SCHEDULE_DELAY=5000 | ||
| # OTEL_SERVICE_NAME=omniroute | ||
| # OTEL_RESOURCE_ATTRIBUTES=deployment.region=us-east-1,deployment.environment=production | ||
| # OTEL_SDK_DISABLED=false | ||
|
|
||
| # LiveWS LAN allow-list (PR #4877 — already merged) | ||
| # Comma-separated host list. Loopback is always allowed. | ||
| # LIVE_WS_ALLOWED_HOSTS=100.96.135.160,192.168.1.20 | ||
|
|
||
| # Authz LAN recognition (PR #4876 — closed by design) | ||
| # Tailscale CGNAT (100.64.0.0/10) is private-LAN by default. | ||
| # TAILSCALE_CIDR=disabled # opt out | ||
| # Comma-separated extra IPv4 CIDRs for LAN overlays (ZeroTier, Nebula, ...). | ||
| # PRIVATE_LAN_CIDR=10.241.0.0/16,100.64.0.0/10 | ||
|
|
||
| # Structured logger | ||
| OMNIROUTE_LOG_LEVEL=info | ||
| # OMNIROUTE_LOG_FORMAT=pretty | ||
|
|
||
| # Misc | ||
| DEPLOYMENT_ENVIRONMENT=development | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,100 @@ | ||
| # OmniRoute Observability — Quickstart | ||
|
|
||
| This document describes the operator-facing observability stack that ships | ||
| with OmniRoute v3.8.34. Three subsystems: | ||
|
|
||
| 1. **OpenTelemetry traces** (`src/lib/observability/otel.ts`) — OTLP/HTTP | ||
| exporter, per-request span emission, idempotent bootstrap. | ||
| 2. **Prometheus `/metrics`** (`src/lib/observability/metrics.ts`) — | ||
| RED-method HTTP counter/histogram, provider-attempt counters, quota | ||
| gauges, cache hit/miss counters. Cardinality-capped label allow-list. | ||
| 3. **Structured logger** (`src/lib/observability/logger.ts`) — pino-style | ||
| NDJSON with `AsyncLocalStorage` request-scoped context, trace/span | ||
| correlation, optional pretty format for dev. | ||
|
|
||
| All three are **off by default** (zero-dep, zero-cost). Operators opt in | ||
| via env vars (see below) — when disabled, every function returns a | ||
| no-op stub in O(1) with no allocations on the hot path. | ||
|
|
||
| ## OpenTelemetry | ||
|
|
||
| | Variable | Default | Purpose | | ||
| |----------|---------|---------| | ||
| | `OTEL_SDK_DISABLED` | `false` | Set to `1` / `true` / `yes` / `on` to force-disable. | | ||
| | `OTEL_EXPORTER_OTLP_ENDPOINT` | _unset_ | Collector URL, e.g. `http://otel-collector:4318`. | | ||
| | `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` | _unset_ | Override traces-only endpoint. | | ||
| | `OTEL_EXPORTER_OTLP_HEADERS` | _unset_ | Comma-separated `k=v,k=v` headers (auth tokens). | | ||
| | `OTEL_EXPORTER_OTLP_PROTOCOL` | `http/json` | We implement HTTP/JSON; `grpc` falls back with a warning. | | ||
| | `OTEL_TRACES_SAMPLER_ARG` | `1` | Sample ratio `[0,1]`. | | ||
| | `OTEL_BSP_MAX_QUEUE_SIZE` | `2048` | Spans buffered between flushes. | | ||
| | `OTEL_BSP_SCHEDULE_DELAY` | `5000` | Flush interval (ms). | | ||
| | `OTEL_SERVICE_NAME` / `OMNIROUTE_SERVICE_NAME` | `omniroute` | `service.name` resource attribute. | | ||
| | `OTEL_RESOURCE_ATTRIBUTES` | _unset_ | Comma-separated `k=v,k=v` extras (e.g. `deployment.region=us-east-1`). | | ||
| | `DEPLOYMENT_ENVIRONMENT` | auto-detect | `deployment.environment` resource attribute. | | ||
|
|
||
| Bootstrap once at startup (already wired in `src/instrumentation-node.ts`): | ||
|
|
||
| ```ts | ||
| import { initTelemetry, shutdownTelemetry } from "@/lib/observability"; | ||
| await initTelemetry(); | ||
|
|
||
| process.on("SIGTERM", () => { | ||
| void shutdownTelemetry().finally(() => process.exit(0)); | ||
| }); | ||
| ``` | ||
|
|
||
| Spans: | ||
|
|
||
| ```ts | ||
| import { getTracer } from "@/lib/observability"; | ||
|
|
||
| const tracer = getTracer("my-component"); | ||
| await tracer.withSpan("upstream.call", async (span) => { | ||
| span.setAttribute("provider", "openai"); | ||
| span.setAttribute("model", "gpt-4o"); | ||
| // throws? → status=ERROR + exception event auto-recorded | ||
| }); | ||
| ``` | ||
|
|
||
| ## Prometheus /metrics | ||
|
|
||
| Endpoint: `GET /api/system/metrics` (added in PR-001). | ||
|
|
||
| | Metric | Type | Labels | | ||
| |--------|------|--------| | ||
| | `omniroute_http_requests_total` | counter | `route`, `method`, `status` | | ||
| | `omniroute_http_request_duration_seconds` | histogram | `route`, `method`, `status` | | ||
| | `omniroute_provider_upstream_attempts_total` | counter | `provider`, `model`, `outcome` | | ||
| | `omniroute_provider_upstream_duration_seconds` | histogram | `provider`, `model`, `outcome` | | ||
| | `omniroute_quota_remaining` | gauge | `provider`, `model` | | ||
| | `omniroute_quota_limit` | gauge | `provider`, `model` | | ||
| | `omniroute_cache_hits_total` | counter | `layer` | | ||
| | `omniroute_cache_misses_total` | counter | `layer` | | ||
|
|
||
| Cardinality cap: each family has a fixed label allow-list | ||
| (`src/lib/observability/metrics.ts :: LABEL_ALLOWLIST`). Unrecognised | ||
| labels are dropped at write time with a `console.warn` so operators see | ||
| the mismatch. | ||
|
|
||
| ## Structured logger | ||
|
|
||
| | Variable | Default | Purpose | | ||
| |----------|---------|---------| | ||
| | `OMNIROUTE_LOG_LEVEL` | `info` | `trace` / `debug` / `info` / `warn` / `error` / `fatal`. | | ||
| | `OMNIROUTE_LOG_FORMAT` | `json` | `json` (NDJSON) or `pretty` (human-readable). | | ||
|
|
||
| Usage: | ||
|
|
||
| ```ts | ||
| import { log, withLogContext } from "@/lib/observability/logger"; | ||
|
|
||
| withLogContext({ tenantId: "t-1", requestId: req.id }, () => { | ||
| log.info("combo_created", { comboId, tierCount }); | ||
| log.error("upstream_failed", err, { provider, model }); | ||
| }); | ||
| ``` | ||
|
|
||
| When an OTel span is active in the current async scope, every log | ||
| record automatically carries `traceId` and `spanId`. Logs at `warn` / | ||
| `error` / `fatal` go to **stderr**; everything else to **stdout** (so | ||
| container log shippers can split severity streams). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The file
.env.example.observabilityis placed in the project root. According to the Repository Style Guide (Section 1: File Placement & Organization), the project root must only contain specific allowed files, and.env.example.observabilityis not on that list. Please merge these environment variables into the existing.env.examplefile in the root to keep the root clean and adhere to the organization rules.References