Skip to content
This repository was archived by the owner on May 26, 2026. It is now read-only.

feat(kora): KR-FEAT-HEARTBEAT ST2 — endpoint flip + Doppler env update #118

Merged
rafe-walker merged 1 commit into from
May 22, 2026
Merged

feat(kora): KR-FEAT-HEARTBEAT ST2 — endpoint flip + Doppler env update #118

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
117 changes: 62 additions & 55 deletions kora_cli/web_server.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4662,63 +4662,70 @@ async def _iter():
async def get_heartbeat_services():
"""Return per-service heartbeat status for Joshua's backend stack.

v1 stub. Replace body with a projection of the live
HeartbeatPoller state once KR-FEAT-HEARTBEAT lands.

Service status enum: ``healthy`` | ``degraded`` | ``unhealthy``.
Each service surfaces a small ``details`` dict — shape varies per
service (e.g. Sentry carries ``unresolved_issues``; Supabase
carries ``connections_pct``); FE renders as expandable key/value.
KR-FEAT-HEARTBEAT ST2: flipped from stub to live read via
:func:`kora_cli.heartbeat_probes.current_service_snapshots`.
The heartbeat scheduler populates the snapshot cache every
``KORA_HEARTBEAT_PROBE_INTERVAL_SEC`` seconds (default 300).

Two-branch shape:

- Live path: ``stub=False`` + ``cache_warming=False`` +
``services`` projected from the snapshot cache.
- Cache-warming path: ``stub=False`` + ``cache_warming=True``
+ ``services=[]``. Returned when the daemon has just
started and the first probe cycle hasn't completed —
FE renders "Probes warming up..." instead of an empty
state. Suppresses any false "all services down" alert
heuristic.

Service status enum: ``healthy`` | ``degraded`` | ``unhealthy``
| ``unknown`` (the latter added in this flip; see TS
``HeartbeatStatus`` in ``web/src/lib/api.ts``).
"""
from datetime import datetime, timezone

from kora_cli.heartbeat_probes import current_service_snapshots

snapshots = current_service_snapshots()
now_iso = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")

if not snapshots:
return {
"services": [],
"generated_at": now_iso,
"stub": False,
"cache_warming": True,
}

# Stable ordering so FE doesn't re-shuffle cards between
# refreshes: render in the default-probe registration order
# (vercel → sentry → doppler → supabase → fly), then any
# extras (operator-added probes via a future config-driven
# extension) by alphabetical name.
canonical_order = ("vercel", "sentry", "doppler", "supabase", "fly")
ordered_names = [n for n in canonical_order if n in snapshots] + sorted(
name for name in snapshots if name not in canonical_order
)

services: list[dict[str, Any]] = []
for name in ordered_names:
snapshot = snapshots[name]
services.append({
"name": snapshot.name,
"status": snapshot.status,
"last_check_at": snapshot.last_check_at.strftime(
"%Y-%m-%dT%H:%M:%SZ"
),
"latency_ms": snapshot.latency_ms,
"details": dict(snapshot.details),
"error": snapshot.error,
})

return {
"services": [
{
"name": "vercel",
"status": "healthy",
"last_check_at": "2026-05-22T18:00:00Z",
"latency_ms": 142,
"details": {
"deployments_last_24h": 8,
"error_rate_24h": 0.0,
},
},
{
"name": "sentry",
"status": "degraded",
"last_check_at": "2026-05-22T18:00:00Z",
"latency_ms": 230,
"details": {"unresolved_issues": 12},
},
{
"name": "doppler",
"status": "healthy",
"last_check_at": "2026-05-22T18:00:00Z",
"latency_ms": 95,
"details": {
"projects_total": 3,
"oldest_secret_age_days": 47,
},
},
{
"name": "supabase",
"status": "healthy",
"last_check_at": "2026-05-22T18:00:00Z",
"latency_ms": 38,
"details": {"connections_pct": 14},
},
{
"name": "fly",
"status": "healthy",
"last_check_at": "2026-05-22T18:00:00Z",
"latency_ms": 88,
"details": {
"apps_running": 2,
"deploys_last_24h": 1,
},
},
],
"generated_at": "2026-05-22T18:00:05Z",
"stub": True,
"services": services,
"generated_at": now_iso,
"stub": False,
"cache_warming": False,
}


Expand Down
42 changes: 42 additions & 0 deletions kora_docs/15_status_and_roadmap/kora_runtime_doppler_env_mapping.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,34 @@ the project — never even as a stale fallback.
| `SLACK_SIGNING_SECRET` | Legacy Slack Bolt gateway | Same gating as SLACK_APP_TOKEN. **Same value as `KORA_SLACK_SIGNING_SECRET`** — both come from the same Slack app's Basic Information page; the env-var split exists because the legacy gateway and the new webhook listener consume the secret via different code paths. Operator sets both to the same value until a follow-on refactor consolidates them. | Same as `KORA_SLACK_SIGNING_SECRET` |
| `SLACK_GATEWAY_ENABLED` | Toggle for the legacy Bolt gateway | Defaults to `true` in `docker/entrypoint.sh`. Set to `false` if the legacy gateway should stay dormant (e.g. running webhook-only on the daemon). | `true` / `false` |

#### Phase 2 Feature 2 — Heartbeat probes (KR-FEAT-HEARTBEAT)

The daemon's heartbeat scheduler probes 5 backend services every
`KORA_HEARTBEAT_PROBE_INTERVAL_SEC` (default 300s). Each probe needs
a Doppler-injected service token. A probe with its auth env unset
degrades gracefully (status: `unknown` in the panel + zero outbound
calls) — these secrets are NOT deploy-blocking, but the heartbeat
dashboard will show "auth env unset" until they're configured.

All 5 live in `kora-runtime-gateways` (same project as the legacy
gateway tokens — gateways = "tokens the runtime uses to reach
outbound services on Joshua's behalf").

| Secret | Probe | Mint via | Scope | Example shape |
|---|---|---|---|---|
| `KORA_VERCEL_API_TOKEN` | Vercel — recent deployments + error rate | <https://vercel.com/account/tokens> | Read-only scope sufficient (lists `/v6/deployments`). | `<32+ char opaque>` |
| `KORA_SENTRY_API_TOKEN` | Sentry — unresolved issue count | <https://sentry.io/settings/account/api/auth-tokens/> | `org:read` scope minimum (`event:read` if probe extension wants project breakdown later). | `<64-hex>` |
| `KORA_SENTRY_ORG` | Sentry — org slug for the issues query | Operator-known org slug (e.g. `stormhaven`). | — | `stormhaven` |
| `KORA_DOPPLER_API_TOKEN` | Doppler — workplace reachability | <https://dashboard.doppler.com/workplace/.../tokens> → **Service Token** (NOT a project token). Workplace read-only scope. Mint a dedicated service token for the probe — keep separate from any per-project tokens. | Workplace read-only | `dp.st.<scope>.<opaque>` |
| `KORA_SUPABASE_ANON_KEY` | Supabase — PostgREST endpoint reachability | Supabase project → Settings → API → **anon key** (NOT the service_role key). | `anon` (public) | `<JWT-shaped>` |
| `KORA_SUPABASE_URL` | Supabase — project URL | Same Project Settings page. | — | `https://<project-ref>.supabase.co` |
| `KORA_FLY_API_TOKEN` | Fly — `kora-runtime` machines state | `flyctl auth token` (operator workstation, deploy token) or Fly dashboard org tokens page. | Read access to the kora-runtime app(s). | `fly_<opaque>` |
| `KORA_FLY_STAGING_APP_NAME` | Fly — optional staging app name | Optional. Set if the operator wants the probe to ALSO check the staging app. Leave unset to probe prod only. | — | `kora-runtime-staging` |

**Validation tip**: after setting these, restart the daemon (or wait
≤5 min for the next probe cycle); `GET /api/heartbeat/services`
should flip each service from `unknown` to `healthy` / `degraded`.

---

## fly.toml `[env]` values (NOT in Doppler)
Expand Down Expand Up @@ -97,6 +125,8 @@ does not own them.
|---|---|---|
| `KORA_WEBHOOK_RATE_LIMIT` | `60/minute` | Tighten via Doppler-gateways if dead-letter rate spikes suggest a flood. slowapi syntax. |
| `KORA_HEALTH_PROBE_CADENCE_SECONDS` | `300` | Lower if dashboard freshness suffers under default 5min cadence. |
| `KORA_HEARTBEAT_PROBE_INTERVAL_SEC` | `300` | KR-FEAT-HEARTBEAT — backend-service probe cadence (Vercel/Sentry/Doppler/Supabase/Fly). Distinct from `KORA_HEALTH_PROBE_CADENCE_SECONDS` (the MCP-client health-check task); both default to 5min, registered as DISTINCT scheduler tasks so one slow cycle doesn't block the other. |
| `KORA_MCP_HEALTH_CHECK_INTERVAL_SEC` | `300` | KR-MCP-CONSUMPTION ST2 — MCP-client-pool health check cadence. Same default as above; same isolation rationale. |
| `KORA_LOG_LEVEL` | `INFO` | `DEBUG` for first-deploy investigation; revert to `INFO` afterwards. |
| `KORA_DEV` | unset | Set to `1` ONLY for local-dev `kora daemon` invocation (bypasses Doppler wrap + lets `KORA_DEPLOY_ENV` default to `dev`). Never set in Fly. |

Expand Down Expand Up @@ -127,6 +157,18 @@ for SECRET in KORA_MCP_BEARER_TOKEN KORA_SLACK_SIGNING_SECRET SLACK_APP_TOKEN SL
|| echo "MISS gateways:$SECRET"
done

# KR-FEAT-HEARTBEAT probe tokens — NOT deploy-blocking. The
# heartbeat panel surfaces "auth env unset" on missing probes
# rather than failing the boot. Run this section opt-in to verify
# the heartbeat-panel data path is fully configured.
for SECRET in KORA_VERCEL_API_TOKEN KORA_SENTRY_API_TOKEN KORA_SENTRY_ORG \
KORA_DOPPLER_API_TOKEN KORA_SUPABASE_ANON_KEY KORA_SUPABASE_URL \
KORA_FLY_API_TOKEN; do
doppler secrets get "$SECRET" -p kora-runtime-gateways -c "$CONFIG" --plain >/dev/null \
&& echo "OK gateways:$SECRET (heartbeat probe)" \
|| echo "MISS gateways:$SECRET (heartbeat probe — panel shows unknown)"
done

# Anti-secret check — these MUST be absent.
for ANTI in ANTHROPIC_API_KEY ANTHROPIC_AUTH_TOKEN; do
doppler secrets get "$ANTI" -p kora-runtime-anthropic -c "$CONFIG" --plain 2>/dev/null \
Expand Down
Loading