feat(observability): Prometheus alert rules + SLO definitions (PR-008)

[KooshaPari]

Summary

[gemini-code-assist]

Code Review

This pull request introduces a comprehensive, dependency-free observability stack for OmniRoute, featuring OpenTelemetry tracing, Prometheus metrics, structured logging, auto-instrumentation helpers, and Prometheus alert/recording rules with accompanying runbooks and validation scripts. The code review identified several critical issues that must be addressed: broken trace context propagation where child spans fail to inherit parent trace IDs; a missing end() method on the Span interface and stub spans; a bug in measureEventLoopLagMs that leaves the event loop lag metric static after startup; multiple TypeScript compilation errors from passing an unaccepted durationMs property to metrics functions; potential application crashes in the logger when serializing circular references or BigInt values; compilation failures in unit tests due to importing non-existent functions and using incorrect signatures; and logical flaws in Prometheus alert rules that could mask single-provider outages or trigger false positives during horizontal scaling.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

In OpenTelemetry, child spans must inherit the traceId of their parent span to maintain trace context propagation. Currently, a brand new traceId is generated for every span regardless of whether a parent exists, which completely breaks distributed tracing. Additionally, the Span interface is missing an end() method, which is required to properly pop spans from the active stack and avoid memory leaks.

Suggested change

	const traceId = generateId(16);
	const spanId = generateId(8);
	const parent = activeSpanStack[activeSpanStack.length - 1];
	const span: Span = {
	name,
	kind: opts?.kind ?? "INTERNAL",
	traceId,
	spanId,
	parentSpanId: parent?.spanId,
	startTimeUnixNano: process.hrtime.bigint(),
	endTimeUnixNano: 0n,
	attributes: { ...(opts?.attributes ?? {}) },
	status: { code: "UNSET" },
	resource: serviceResource(),
	const parent = activeSpanStack[activeSpanStack.length - 1];
	const traceId = parent ? parent.traceId : generateId(16);
	const spanId = generateId(8);
	const span: Span = {
	name,
	kind: opts?.kind ?? "INTERNAL",
	traceId,
	spanId,
	parentSpanId: parent?.spanId,
	startTimeUnixNano: process.hrtime.bigint(),
	endTimeUnixNano: 0n,
	attributes: { ...(opts?.attributes ?? {}) },
	status: { code: "UNSET" },
	resource: serviceResource(),
	end() { endSpan(this); }
	};

[gemini-code-assist]

The measureEventLoopLagMs function currently only schedules setImmediate calls when lagSamples is empty (at startup). Once lagSamples has 5 elements, it never schedules any new measurements, returning the exact same cached average forever. This makes the event loop lag metric completely static and useless after startup. Update it to schedule a new measurement on each call and maintain a rolling window of the last 5 samples.

Suggested change

	function measureEventLoopLagMs(): number {
	// Coarse but free: schedule an immediate and time how long it takes to
	// fire. Averaged across the last 5 samples.
	if (!lagSamples.length) {
	for (let i = 0; i < 5; i++) {
	const start = Date.now();
	setImmediate(() => lagSamples.push(Date.now() - start));
	}
	return 0;
	}
	return lagSamples.reduce((a, b) => a + b, 0) / lagSamples.length;
	}
	const lagSamples: number[] = [];
	function measureEventLoopLagMs(): number {
	const start = Date.now();
	setImmediate(() => {
	lagSamples.push(Date.now() - start);
	if (lagSamples.length > 5) {
	lagSamples.shift();
	}
	});
	if (!lagSamples.length) return 0;
	return lagSamples.reduce((a, b) => a + b, 0) / lagSamples.length;
	}
	const lagSamples: number[] = [];

[gemini-code-assist]

The recordProviderAttempt function does not accept durationMs in its signature. Passing it here will cause a TypeScript compilation error due to excess property checks on object literals.

        recordProviderAttempt({
          provider: safeProvider,
          model: safeModel,
          outcome: "success",
          durationSeconds: durationMs / 1000,
        });

[gemini-code-assist]

The recordProviderAttempt function does not accept durationMs in its signature. Passing it here will cause a TypeScript compilation error due to excess property checks on object literals.

        recordProviderAttempt({
          provider: safeProvider,
          model: safeModel,
          outcome,
          durationSeconds: durationMs / 1000,
        });

[gemini-code-assist]

The httpMetricsMiddleware function does not accept durationMs in its signature. Passing it here will cause a TypeScript compilation error due to excess property checks on object literals.

        httpMetricsMiddleware({
          method,
          route: path,
          status,
          durationSeconds: durationMs / 1000,
        });

[gemini-code-assist]

Add the end() method to the stub span to satisfy the updated Span interface and ensure consistency when telemetry is disabled.

Suggested change

	function stubSpan(
	name: string,
	opts?: { kind?: SpanKind; attributes?: Record<string, AttributeValue> }
	): Span {
	return {
	name,
	kind: opts?.kind ?? "INTERNAL",
	traceId: "00000000000000000000000000000000",
	spanId: "0000000000000000",
	startTimeUnixNano: 0n,
	endTimeUnixNano: 0n,
	attributes: { ...(opts?.attributes ?? {}) },
	status: { code: "UNSET" },
	};
	}
	function stubSpan(
	name: string,
	opts?: { kind?: SpanKind; attributes?: Record<string, AttributeValue> }
	): Span {
	return {
	name,
	kind: opts?.kind ?? "INTERNAL",
	traceId: "00000000000000000000000000000000",
	spanId: "0000000000000000",
	startTimeUnixNano: 0n,
	endTimeUnixNano: 0n,
	attributes: { ...(opts?.attributes ?? {}) },
	status: { code: "UNSET" },
	end() { endSpan(this); }
	};
	}

[gemini-code-assist]

The test file imports and tests non-existent functions and constants (extractRoutePattern, withProxySpan, PROXY_SPAN_ROUTE_HEADER, and PROXY_SPAN_DURATION_HEADER) from @/lib/observability/proxySpan, which will cause compilation and test execution failures. Update the imports and refactor the test cases to only test the actual exported functions (wrapProxy and propagateTraceParent).

import {
  propagateTraceParent,
  wrapProxy,
} from "@/lib/observability/proxySpan";

[gemini-code-assist]

The test file imports non-existent functions instrumentHttp and instrumentProvider from @/lib/observability/auto, which will cause compilation and test execution failures. Additionally, the test cases call instrumentFetch and instrumentProvider with signatures that do not match the actual implementations in auto.ts (e.g., passing 3 arguments instead of the expected parameters). Update the imports and refactor the test cases to match the actual API signatures.

Suggested change

	import {
	instrumentHttp,
	instrumentFetch,
	instrumentProvider,
	instrumentCache,
	instrumentDb,
	sanitizeProvider,
	sanitizeModel,
	isAutoEnabled,
	currentTraceId,
	} from "@/lib/observability/auto";
	import {
	instrumentFetch,
	instrumentProviderCall,
	instrumentCache,
	instrumentDb,
	sanitizeProvider,
	sanitizeModel,
	isAutoEnabled,
	currentTraceId,
	} from "@/lib/observability/auto";
	import { wrapProxy } from "@/lib/observability/proxySpan";

[gemini-code-assist]

The OmniRouteBifrostTimeoutRate alert uses a global sum(...) across all providers. If a low-traffic provider is completely broken (100% timeout rate), its timeouts will be masked by high-traffic healthy providers, and the alert will never fire. Group the expression by provider to ensure single-provider outages are caught.

      - alert: OmniRouteBifrostTimeoutRate
        expr: |
          sum by (provider) (rate(omniroute_provider_upstream_attempts_total{outcome="timeout"}[5m]))
          /
          sum by (provider) (rate(omniroute_provider_upstream_attempts_total[5m])) > 0.1
        for: 5m

[gemini-code-assist]

The OmniRouteHighConcurrency alert is intended to detect single-node saturation (soft-concurrency ceiling for a single OmniRoute node). However, the PromQL expression uses a global sum(...) without grouping by instance. As the cluster scales horizontally, the aggregate request rate will naturally exceed 500 req/min, triggering false positive alerts even when individual nodes are completely healthy. Group the rate by instance.

      - alert: OmniRouteHighConcurrency
        expr: |
          sum by (instance) (rate(omniroute_http_requests_total[1m])) * 60 > 500
        for: 5m

[KooshaPari]

Diagnosis: CI is RED with Fast Quality Gates failing.

[diegosouzapw]

Thanks for this work, @KooshaPari 🙏 — closing after a maintainer review of all 46 open refactor PRs in this batch (yours included). This is not a rejection of the effort.

Why closed (applies to the whole batch): every branch here is 44–50 commits behind release/v3.8.38 and carries old versions of 500 files, so the 3-dot diff against the release reads as a **+71k/−20k rollback** rather than your actual change — merging as-is would revert ~48 already-merged commits. This branch is additionally orphaned from the v1.0.0 tree (no merge-base with the release), so it can't even be rebased — the work would have to be re-applied file by file. So none are mergeable as they stand, independent of the idea's merit.

This PR specifically: Needs rework — well-calibrated alerts/SLOs, but they reference metrics that only exist in the orphaned observability stack.

What happens to it: 📓 Captured as a documented lesson for our refactor roadmap — the approach is recorded (with your credit) and will guide the matching phase of our internal plan. The SLO thresholds + burn-rate definitions + the offline validate-prometheus-rules.mjs gate are recorded as references.

We ran a full case-by-case triage. The approaches worth keeping are being recorded internally with author credit, so when we pick them up the attribution travels with the code. Thank you for pushing on this. 🙌