feat(layer-4): Prometheus metrics + provisioned Grafana dashboard

CLAUDE.md

@@ -24,6 +24,7 @@ COVENANT_BEHAVIOR_DRIFT=1 covenant check # body-only drift (schema identical); p
 | 1 proxy + quarantine | `covenant/proxy/` | `fastapi`/`httpx`/`uvicorn` are the `[proxy]` extra, lazily imported in `cli.py`. |
 | 2 store | `covenant/store/` | `asyncpg` is the `[store]` extra. Store writes are best-effort: log and swallow, never fail the request path. |
 | 3 probes + judge | `covenant/fingerprint.py`, `covenant/judge/` | Probes *execute* tools at snapshot/check time (list read-only tools only). `anthropic`/`google-genai` are the `[judge]` extra, imported on use; the model-name prefix picks the provider. |
+| 4 observability | `covenant/proxy/metrics.py`, `deploy/` | `prometheus-client` rides the `[proxy]` extra. One `CollectorRegistry` per app, never the global registry (tests create many apps). Metric writes are in-process and non-throwing, never on the store path. |
 
 Design specs (rationale, rule tables, named decisions) live in `docs/superpowers/specs/` — read the relevant spec before changing classifier or proxy behavior.
 

README.md

@@ -134,6 +134,7 @@ covenant proxy --upstream http://localhost:8000/mcp --port 9000
 - `POST /covenant/refresh` — Covenant re-lists the upstream itself and re-checks. Detection is proxy-owned by design: a client's `tools/list` can arrive *after* the call it should have protected, so enforcement never depends on client behavior.
 - `GET /covenant/status` — currently quarantined tools and why.
 - `GET /covenant/calls` — recent call log with latency and outcomes.
+- `GET /covenant/metrics` — Prometheus metrics: per-tool call counters (ok/error/blocked), latency histograms, drift events, quarantine gauge. `docker compose up -d prometheus grafana` gives a provisioned dashboard at `http://localhost:3000` — the quarantine stat flips green→red within one scrape of a drift.
 
 Try it end-to-end with a live agent-style client: [examples/demo_layer1.py](examples/demo_layer1.py).
 
@@ -160,7 +161,7 @@ Covenant is built in dependency-ordered layers; each ships alone and each higher
 | 1 | Transparent proxy + quarantine | ✅ shipped |
 | 2 | Postgres contract store (call log, drift events, durable quarantine) | ✅ shipped |
 | 3 | Behavioral probes — response fingerprints + LLM judge for semantic drift | ✅ shipped |
-| 4 | Observability — OTel spans, Prometheus, dashboard | roadmap |
+| 4 | Observability — Prometheus metrics + Grafana dashboard (OTel deferred) | ✅ shipped |
 | 5 | K8s operator + Helm — `MCPContract` CRD, probes as Jobs | roadmap |
 
 Design specs for the shipped layers live in [docs/superpowers/specs](docs/superpowers/specs).

covenant/proxy/metrics.py

@@ -0,0 +1,55 @@
+"""Layer 4 observability: Prometheus metrics for the proxy.
+
+One ``Metrics`` instance per app, each with its own ``CollectorRegistry`` — the
+prometheus_client default registry is process-global and would collide when tests
+(or embedders) create several proxy apps in one process.
+
+Exposed at ``GET /covenant/metrics`` in the standard text exposition format.
+"""
+
+from __future__ import annotations
+
+from prometheus_client import (
+    CONTENT_TYPE_LATEST,
+    CollectorRegistry,
+    Counter,
+    Gauge,
+    Histogram,
+    generate_latest,
+)
+
+
+class Metrics:
+    def __init__(self) -> None:
+        self.registry = CollectorRegistry()
+        self.calls = Counter(
+            "covenant_calls_total",
+            "tools/call requests through the proxy, by tool and outcome",
+            ["tool", "outcome"],  # outcome: ok | error | blocked
+            registry=self.registry,
+        )
+        self.latency = Histogram(
+            "covenant_call_latency_seconds",
+            "Upstream latency of forwarded tools/call requests",
+            ["tool"],
+            registry=self.registry,
+        )
+        self.drift = Counter(
+            "covenant_drift_total",
+            "Drift events detected, by severity",
+            ["severity"],
+            registry=self.registry,
+        )
+        self.quarantined = Gauge(
+            "covenant_quarantined_tools",
+            "Tools currently quarantined",
+            registry=self.registry,
+        )
+
+    def record_call(self, tool: str, outcome: str, latency_s: float | None = None) -> None:
+        self.calls.labels(tool=tool, outcome=outcome).inc()
+        if latency_s is not None:
+            self.latency.labels(tool=tool).observe(latency_s)
+
+    def render(self) -> tuple[bytes, str]:
+        return generate_latest(self.registry), CONTENT_TYPE_LATEST

covenant/proxy/server.py

@@ -31,6 +31,7 @@
 from ..store.base import Store
 from ..store.memory import InMemoryStore
 from .detect import detect
+from .metrics import Metrics
 from .quarantine import Quarantine
 
 log = logging.getLogger("covenant.proxy")
@@ -111,6 +112,7 @@ def _is_error(resp_json: object) -> bool:
 async def _proxy(app: FastAPI, request: Request) -> Response:
     q: Quarantine = app.state.q
     store: Store = app.state.store
+    metrics: Metrics = app.state.metrics
     body = await request.body()
 
     rpc = None
@@ -131,6 +133,7 @@ async def _proxy(app: FastAPI, request: Request) -> Response:
             f"tool unavailable - '{tool}' quarantined by Covenant "
             f"(contract drift: {q.reason(tool)})",
         )
+        metrics.record_call(tool, "blocked")
         await _safe(store.record_call(tool, method, 0, True, True))
         return Response(content=json.dumps(blocked), media_type="application/json")
 
@@ -168,10 +171,13 @@ async def _passthrough() -> AsyncIterator[bytes]:
         tools = ((resp_json.get("result") or {}).get("tools")) or []
         breaking = detect(app.state.baseline, tools)
         q.sync(breaking)
+        metrics.quarantined.set(len(q.all()))
         await _safe(store.sync_quarantine(breaking))
 
     if method == "tools/call" and isinstance(tool, str):
-        await _safe(store.record_call(tool, method, latency_ms, _is_error(resp_json), False))
+        is_err = _is_error(resp_json)
+        metrics.record_call(tool, "error" if is_err else "ok", latency_ms / 1000)
+        await _safe(store.record_call(tool, method, latency_ms, is_err, False))
 
     return Response(
         content=raw, status_code=up_resp.status_code,
@@ -207,6 +213,7 @@ async def lifespan(app: FastAPI) -> AsyncIterator[None]:
     app.state.http = http_client or httpx.AsyncClient(timeout=30.0)
     app.state.lister = lister
     app.state.store = store or InMemoryStore()
+    app.state.metrics = Metrics()
 
     @app.get("/covenant/status")
     async def status() -> JsonDict:
@@ -226,11 +233,18 @@ async def refresh() -> JsonDict:
             raise HTTPException(status_code=502, detail=f"upstream list failed: {e}") from e
         breaking = detect(app.state.baseline, tools)
         app.state.q.sync(breaking)
+        app.state.metrics.quarantined.set(len(app.state.q.all()))
         await _safe(app.state.store.sync_quarantine(breaking))
         for tool, reason in breaking.items():
+            app.state.metrics.drift.labels(severity="breaking").inc()
             await _safe(app.state.store.record_drift(tool, "breaking", [{"message": reason}]))
         return {"quarantined": app.state.q.all(), "checked": len(tools)}
 
+    @app.get("/covenant/metrics")
+    async def metrics() -> Response:
+        payload, content_type = app.state.metrics.render()
+        return Response(content=payload, media_type=content_type)
+
     @app.api_route("/mcp", methods=["GET", "POST", "DELETE"])
     async def mcp(request: Request) -> Response:
         return await _proxy(app, request)

deploy/grafana/dashboards/covenant.json

@@ -0,0 +1,66 @@
+{
+  "title": "Covenant — MCP contract firewall",
+  "uid": "covenant",
+  "timezone": "browser",
+  "refresh": "5s",
+  "time": { "from": "now-15m", "to": "now" },
+  "panels": [
+    {
+      "id": 1,
+      "title": "Calls by outcome (rate/min)",
+      "type": "timeseries",
+      "gridPos": { "x": 0, "y": 0, "w": 12, "h": 8 },
+      "targets": [
+        {
+          "expr": "sum by (tool, outcome) (rate(covenant_calls_total[1m])) * 60",
+          "legendFormat": "{{tool}} · {{outcome}}"
+        }
+      ]
+    },
+    {
+      "id": 2,
+      "title": "p95 call latency",
+      "type": "timeseries",
+      "gridPos": { "x": 12, "y": 0, "w": 12, "h": 8 },
+      "fieldConfig": { "defaults": { "unit": "s" }, "overrides": [] },
+      "targets": [
+        {
+          "expr": "histogram_quantile(0.95, sum by (tool, le) (rate(covenant_call_latency_seconds_bucket[5m])))",
+          "legendFormat": "{{tool}}"
+        }
+      ]
+    },
+    {
+      "id": 3,
+      "title": "Quarantined tools",
+      "type": "stat",
+      "gridPos": { "x": 0, "y": 8, "w": 12, "h": 6 },
+      "fieldConfig": {
+        "defaults": {
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              { "color": "green", "value": null },
+              { "color": "red", "value": 1 }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "targets": [{ "expr": "covenant_quarantined_tools" }]
+    },
+    {
+      "id": 4,
+      "title": "Drift events",
+      "type": "timeseries",
+      "gridPos": { "x": 12, "y": 8, "w": 12, "h": 6 },
+      "targets": [
+        {
+          "expr": "sum by (severity) (increase(covenant_drift_total[5m]))",
+          "legendFormat": "{{severity}}"
+        }
+      ]
+    }
+  ],
+  "schemaVersion": 39
+}

deploy/grafana/provisioning/dashboards/provider.yml

@@ -0,0 +1,7 @@
+apiVersion: 1
+providers:
+  - name: covenant
+    folder: ""
+    type: file
+    options:
+      path: /var/lib/grafana/dashboards

deploy/grafana/provisioning/datasources/prometheus.yml

@@ -0,0 +1,7 @@
+apiVersion: 1
+datasources:
+  - name: Prometheus
+    type: prometheus
+    access: proxy
+    url: http://prometheus:9090
+    isDefault: true

deploy/prometheus.yml

@@ -0,0 +1,6 @@
+scrape_configs:
+  - job_name: covenant
+    metrics_path: /covenant/metrics
+    scrape_interval: 5s
+    static_configs:
+      - targets: ["host.docker.internal:9000"]

docker-compose.yml

@@ -1,7 +1,9 @@
-# Local Postgres for Covenant's Layer 2 contract store.
-#   docker compose up -d db
+# Local services for Covenant.
+#   docker compose up -d db                    # Layer 2 contract store
 #   DATABASE_URL=postgresql://covenant:covenant@127.0.0.1:5432/covenant
-# Reversible: `docker compose down -v` removes the container and its volume.
+#   docker compose up -d prometheus grafana    # Layer 4 observability
+#   Grafana: http://localhost:3000 (anonymous) - scrapes the proxy at host port 9000
+# Reversible: `docker compose down -v` removes the containers and volumes.
 
 services:
   db:
@@ -20,5 +22,27 @@ services:
       timeout: 3s
       retries: 10
 
+  prometheus:
+    image: prom/prometheus:latest
+    ports:
+      - "9090:9090"
+    volumes:
+      - ./deploy/prometheus.yml:/etc/prometheus/prometheus.yml:ro
+    extra_hosts:
+      - "host.docker.internal:host-gateway"  # reach the proxy on the host (Linux parity)
+
+  grafana:
+    image: grafana/grafana:latest
+    ports:
+      - "3000:3000"
+    environment:
+      GF_AUTH_ANONYMOUS_ENABLED: "true"
+      GF_AUTH_ANONYMOUS_ORG_ROLE: Admin
+    volumes:
+      - ./deploy/grafana/provisioning:/etc/grafana/provisioning:ro
+      - ./deploy/grafana/dashboards:/var/lib/grafana/dashboards:ro
+    depends_on:
+      - prometheus
+
 volumes:
   pgdata: {}

docs/superpowers/specs/2026-07-03-covenant-layer4-observability-design.md

@@ -0,0 +1,54 @@
+# Layer 4 — observability (Prometheus metrics + Grafana dashboard)
+
+## Scope
+
+Prometheus metrics on the proxy, exposed at `GET /covenant/metrics`, plus a
+provisioned Grafana dashboard in docker-compose. **OTel spans are deferred**: the
+OTel SDK is a heavy dependency tree, and every signal the demo needs (per-tool
+traffic, latency, drift, quarantine) is a metric, not a trace. Revisit when a
+real multi-hop fleet exists to trace.
+
+## Named decisions
+
+- **Per-app `CollectorRegistry`** — the prometheus_client default registry is
+  process-global; two `create_app` calls in one process (every test run) would
+  collide with `Duplicated timeseries`. Each app gets its own registry;
+  `/covenant/metrics` renders only its own.
+- **`prometheus-client` lives in the `[proxy]` extra** — metrics are meaningless
+  without the proxy process, and the core CLI must stay dependency-light
+  (Layer 0 rule). No new extra for one pure-Python package.
+- **Endpoint is `/covenant/metrics`, not `/metrics`** — everything Covenant owns
+  sits under `/covenant/*`; the proxy is transparent everywhere else. Prometheus
+  sets `metrics_path` per job, so the non-default path costs one config line.
+- **Metrics are in-process, not store-backed** — Prometheus scrapes cumulative
+  counters and handles restarts (`rate()` is reset-aware). The Layer 2 store
+  remains the durable record; metrics are the live signal. No double-write.
+- **Best-effort principle carries over** — metric mutations are non-throwing
+  in-memory operations on the request path; no awaits, no store timeout needed.
+
+## Metrics
+
+| Metric | Type | Labels | Incremented |
+|---|---|---|---|
+| `covenant_calls_total` | Counter | `tool`, `outcome` (`ok`/`error`/`blocked`) | every `tools/call` through the proxy |
+| `covenant_call_latency_seconds` | Histogram | `tool` | forwarded calls only (blocked calls never reach upstream) |
+| `covenant_drift_total` | Counter | `severity` | per breaking tool on `/covenant/refresh` |
+| `covenant_quarantined_tools` | Gauge | — | set after every quarantine sync (refresh + in-band) |
+
+## Dashboard
+
+`deploy/grafana/dashboards/covenant.json`, provisioned automatically:
+calls-by-outcome rate, p95 latency (`histogram_quantile` over buckets),
+quarantined-tools stat (green 0 / red ≥1), drift events. Compose runs
+Prometheus (scrapes the host-run proxy via `host.docker.internal`, 5s interval)
+and anonymous-admin Grafana on :3000.
+
+## Demo
+
+```bash
+docker compose up -d prometheus grafana
+covenant proxy --upstream http://localhost:8000/mcp   # + traffic
+curl -X POST http://localhost:9000/covenant/refresh   # after drifting the server
+```
+
+Quarantine stat flips green→red on the dashboard within one scrape.

pyproject.toml

@@ -41,6 +41,7 @@ proxy = [
     "fastapi>=0.115",
     "uvicorn>=0.30",
     "httpx>=0.27",
+    "prometheus-client>=0.20",
 ]
 store = [
     "asyncpg>=0.29",
@@ -57,6 +58,7 @@ dev = [
     "fastapi>=0.115",
     "uvicorn>=0.30",
     "httpx>=0.27",
+    "prometheus-client>=0.20",
     "asyncpg>=0.29",
     "anthropic>=0.40",
     "google-genai>=1.0",