Skip to content

Phase 3: self-healing telemetry v2 (rolling z-score + typed playbooks) #150

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
KooshaPari merged 1 commit into from
Jul 2, 2026
Merged

Phase 3: self-healing telemetry v2 (rolling z-score + typed playbooks) #150

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 35 additions & 1 deletion docs/architecture/RESILIENCE_GUIDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
title: "Resilience Guide"
version: 3.8.31
lastUpdated: 2026-06-20
lastUpdated: 2026-06-28
---

# Resilience Guide
Expand Down Expand Up @@ -241,6 +241,40 @@ Orquestrados por `.github/workflows/nightly-resilience.yml` (cron + dispatch). N

---

## 4. Self-Healing Telemetry (Phase 3, v2)

**Scope:** every provider participating in an `autoCombo` request.

**Purpose:** continuously sample per-provider latency and error rate, detect statistically anomalous samples via a rolling-window z-score, and dispatch a typed playbook (`force-proxy-rotation`, `degrade-provider`, `drop-cooldown`, etc.) against the provider manager — without operator intervention.

**When it fires:** a sample whose z-score against the provider's rolling window exceeds the configured threshold. The threshold is tunable via the `ResilienceTab → Self-Healing` card (`windowSize`, `zThreshold`, `cooloffMs`, `minSamples`, `dryRun`).

**Implementation:**

| Layer | Path |
|---|---|
| DB schema | `src/lib/db/migrations/100_provider_health_history.sql` |
| Persistence | `src/lib/db/providerHealthHistory.ts` |
| Detector | `src/lib/resilience/anomalyDetector.ts` |
| Settings | `src/lib/resilience/selfHealingSettings.ts` |
| Catalog | `src/lib/resilience/playbooks.ts` |
| Coordinator | `src/lib/resilience/selfHealingManager.ts` |
| Hook | `src/lib/resilience/anomalyHook.ts` (singleton) |
| Boot wiring | `src/server-init.ts` (lazy hydration on first sample) |
| UI | `src/app/(dashboard)/dashboard/settings/components/ResilienceTab.tsx` |
| E2E | `tests/e2e/selfHealing.test.ts` |

**Master switches (must both be on for the loop to act):**

1. `ResilienceTab → Self-Healing → Enabled` (or via `PATCH /api/resilience { selfHealing: { enabled: true, ... } }`)
2. `OMNIROUTE_SELF_HEALING_ENABLED` feature flag in `featureFlagDefinitions.ts` (`category: "health"`, default `false`)

**Retention:** DB rows older than `retentionSeconds` (default 86_400 = 24h) are pruned by `SelfHealingManager.prune()`, called on boot and at each detection cycle.

**Dry-run:** setting `dryRun: true` records the action to the anomaly ledger but skips the side-effect on the provider manager. Use this to tune thresholds safely before letting the loop act on real traffic.

---

## See Also

- [Architecture Guide](./ARCHITECTURE.md) — System architecture and internals
Expand Down
13 changes: 12 additions & 1 deletion docs/routing/AUTO-COMBO.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
title: "OmniRoute Auto-Combo Engine"
version: 3.8.31
lastUpdated: 2026-06-20
lastUpdated: 2026-06-28
---

# OmniRoute Auto-Combo Engine
Expand Down Expand Up @@ -583,3 +583,14 @@ See `docs/marketing/TIERS.md` for tier definitions and provider classification.
| `open-sse/services/autoCombo/providerRegistryAccessor.ts` | Test hook for mocking provider registry |
| `src/shared/constants/routingStrategies.ts` | `ROUTING_STRATEGY_VALUES` (17 strategies) |
| `src/sse/handlers/chat.ts` | Integration: auto-prefix short-circuit |

## Self-Healing Telemetry (Phase 3, v2)

The `selfHealing.ts` file is being extended in Phase 3 to consume a rolling-window anomaly detector and dispatch typed playbooks against the provider manager. New surface:

- **`src/lib/resilience/anomalyHook.ts`** — singleton hook: `recordHealthSample({ providerId, latencyMs, errorRate, nowMs })` is called from the auto-combo engine after every completed request
- **`src/lib/resilience/anomalyDetector.ts`** — pure-function detector (z-score against the rolling window)
- **`src/lib/resilience/playbooks.ts`** — typed action catalog: `force-proxy-rotation`, `degrade-provider`, `drop-cooldown`, `noop`
- **`src/lib/db/providerHealthHistory.ts`** + migration `100_provider_health_history.sql` — SQLite log of samples + anomalies + playbook dispatches, with TTL prune

This makes the auto-combo engine self-tuning: a provider whose recent latency spikes beyond its rolling-window baseline is automatically rotated or degraded without operator intervention. See the [Resilience Guide §4](../architecture/RESILIENCE_GUIDE.md#4-self-healing-telemetry-phase-3-v2) for tunables.
192 changes: 192 additions & 0 deletions src/app/(dashboard)/dashboard/settings/components/ResilienceTab.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,15 @@
maxRetryCooldownMs: number;
};

type SelfHealingSettings = {
enabled: boolean;
windowSize: number;
zThreshold: number;
cooloffMs: number;
minSamples: number;
dryRun: boolean;
Comment on lines +59 to +62

@chatgpt-codex-connector chatgpt-codex-connector Bot Jun 29, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Use the backend self-healing field names in the UI

These fields do not match the backend SelfHealingSettings shape (warnThreshold, criticalThreshold, minSamplesForDetection, retentionSeconds, playbookEnabled, etc.). As a result, the GET response renders values like zThreshold, cooloffMs, minSamples, and dryRun as undefined, and even after the PATCH schema is fixed the server normalizer will ignore those keys, so threshold/min-sample/dry-run edits will not persist or affect runtime behavior.

Useful? React with 👍 / 👎.

};

type ResilienceResponse = {
requestQueue: RequestQueueSettings;
connectionCooldown: {
Expand All @@ -67,6 +76,7 @@
comboCooldownWait: ComboCooldownWaitSettings;
quotaShareConcurrencyLimit: QuotaShareConcurrencyLimitSettings;
providerCooldown: ProviderCooldownSettings;
selfHealing?: SelfHealingSettings;
};

function formatMs(value: number | null | undefined) {
Expand Down Expand Up @@ -1031,6 +1041,177 @@
);
}

function SelfHealingCard({
value,
onSave,
saving,
}: {
value: SelfHealingSettings | undefined;
onSave: (next: SelfHealingSettings) => Promise<void>;
saving: boolean;
}) {

Check warning on line 1052 in src/app/(dashboard)/dashboard/settings/components/ResilienceTab.tsx

View check run for this annotation

SonarQubeCloud / SonarCloud Code Analysis

Mark the props of the component as read-only. 


See more on https://sonarcloud.io/project/issues?id=KooshaPari_OmniRoute&issues=AZ8RnZnPUniKS_Rlpk9z&open=AZ8RnZnPUniKS_Rlpk9z&pullRequest=150
const t = useTranslations("settings");
const fallback: SelfHealingSettings = {
enabled: false,
windowSize: 20,
zThreshold: 2.5,
cooloffMs: 60_000,
minSamples: 10,
dryRun: false,
};
const initial = value ?? fallback;
const [editing, setEditing] = useState(false);
const [draft, setDraft] = useState(initial);

useEffect(() => {
setDraft(initial);
}, [value]);

const title =
t("resilienceSelfHealingTitle") || "Self-healing telemetry (auto-detect provider incidents)";
const desc =
t("resilienceSelfHealingDesc") ||
"When enabled, the server continuously samples per-provider latency and error rate. If a provider's recent sample is more than the configured z-score above its rolling window mean, the matched playbook is dispatched (cool-off prevents flapping). dryRun records the action but skips the side-effect.";

return (
<Card className="p-6">
<div className="mb-4 flex items-start justify-between gap-4">
<div className="space-y-2">
<div className="flex items-center gap-2">
<span className="material-symbols-outlined text-xl text-primary">healing</span>
<h2 className="text-lg font-bold">{title}</h2>
</div>
<SectionDescription
scope={t("resilienceSelfHealingScope") || "All autoCombo providers"}
trigger={t("resilienceSelfHealingTrigger") || "z-score > threshold on rolling window"}
effect={
t("resilienceSelfHealingEffect") ||
"Dispatches matched playbook (cooldown, retry, fail-open, etc.)"
}
/>
</div>
<ActionRow
editing={editing}
saving={saving}
onEdit={() => setEditing(true)}
onCancel={() => {
setDraft(initial);
setEditing(false);
}}
onSave={async () => {
await onSave(draft);
setEditing(false);
}}
/>
</div>

<p className="mb-4 text-sm text-text-muted">{desc}</p>

<div className="grid grid-cols-1 gap-3 lg:grid-cols-2">
{editing ? (
<>
<BooleanField
label={t("resilienceSelfHealingEnabled") || "Enable self-healing"}
description={
t("resilienceSelfHealingEnabledDesc") ||
"Also requires the OMNIROUTE_SELF_HEALING_ENABLED feature flag to be on."
}
checked={draft.enabled}
onChange={(enabled) => setDraft((prev) => ({ ...prev, enabled }))}
/>
<NumberField
label={t("resilienceSelfHealingWindowSize") || "Rolling window size"}
value={draft.windowSize}
min={2}
onChange={(windowSize) => setDraft((prev) => ({ ...prev, windowSize }))}
/>
<NumberField
label={t("resilienceSelfHealingZThreshold") || "Z-score threshold"}
value={draft.zThreshold}
min={1}
suffix="σ"
onChange={(zThreshold) => setDraft((prev) => ({ ...prev, zThreshold }))}
/>
<NumberField
label={t("resilienceSelfHealingCooloffMs") || "Cool-off between dispatches"}
value={draft.cooloffMs}
min={0}
suffix="ms"
onChange={(cooloffMs) => setDraft((prev) => ({ ...prev, cooloffMs }))}
/>
<NumberField
label={t("resilienceSelfHealingMinSamples") || "Min samples before detection"}
value={draft.minSamples}
min={1}
onChange={(minSamples) => setDraft((prev) => ({ ...prev, minSamples }))}
/>
<BooleanField
label={t("resilienceSelfHealingDryRun") || "Dry-run (log, do not act)"}
description={
t("resilienceSelfHealingDryRunDesc") ||
"Useful for tuning thresholds without side-effects on real traffic."
}
checked={draft.dryRun}
onChange={(dryRun) => setDraft((prev) => ({ ...prev, dryRun }))}
/>
</>
) : (
<>
<div className="rounded-xl border border-border bg-bg-subtle p-4">
<div className="text-xs text-text-muted">
{t("resilienceSelfHealingEnabled") || "Enabled"}
</div>
<div className="mt-1 text-sm font-semibold text-text-main">
{initial.enabled ? t("statusEnabled") : t("statusDisabled")}
</div>
</div>
<div className="rounded-xl border border-border bg-bg-subtle p-4">
<div className="text-xs text-text-muted">
{t("resilienceSelfHealingWindowSize") || "Window"}
</div>
<div className="mt-1 text-sm font-semibold text-text-main">
{initial.windowSize}
</div>
</div>
<div className="rounded-xl border border-border bg-bg-subtle p-4">
<div className="text-xs text-text-muted">
{t("resilienceSelfHealingZThreshold") || "Z-score"}
</div>
<div className="mt-1 text-sm font-semibold text-text-main">
{initial.zThreshold}σ
</div>
</div>
<div className="rounded-xl border border-border bg-bg-subtle p-4">
<div className="text-xs text-text-muted">
{t("resilienceSelfHealingCooloffMs") || "Cool-off"}
</div>
<div className="mt-1 text-sm font-semibold text-text-main">
{formatMs(initial.cooloffMs)}
</div>
</div>
<div className="rounded-xl border border-border bg-bg-subtle p-4">
<div className="text-xs text-text-muted">
{t("resilienceSelfHealingMinSamples") || "Min samples"}
</div>
<div className="mt-1 text-sm font-semibold text-text-main">
{initial.minSamples}
</div>
</div>
<div className="rounded-xl border border-border bg-bg-subtle p-4">
<div className="text-xs text-text-muted">
{t("resilienceSelfHealingDryRun") || "Dry-run"}
</div>
<div className="mt-1 text-sm font-semibold text-text-main">
{initial.dryRun ? t("yes") : t("no")}
</div>
</div>
</>
)}
</div>
</Card>
);
}

export default function ResilienceTab() {
const notify = useNotificationStore();
const t = useTranslations("settings");
Expand Down Expand Up @@ -1063,7 +1244,10 @@
connectionCooldown: json.connectionCooldown,
providerBreaker: json.providerBreaker,
waitForCooldown: json.waitForCooldown,
comboCooldownWait: json.comboCooldownWait,
quotaShareConcurrencyLimit: json.quotaShareConcurrencyLimit,
providerCooldown: json.providerCooldown,
selfHealing: json.selfHealing,
});
} catch (error) {
notify.error(
Expand Down Expand Up @@ -1099,7 +1283,10 @@
connectionCooldown: json.connectionCooldown,
providerBreaker: json.providerBreaker,
waitForCooldown: json.waitForCooldown,
comboCooldownWait: json.comboCooldownWait,
quotaShareConcurrencyLimit: json.quotaShareConcurrencyLimit,
providerCooldown: json.providerCooldown,
selfHealing: json.selfHealing,
});
notify.success(tx("savedSuccessfully", "Resilience settings updated."));
} catch (error) {
Expand Down Expand Up @@ -1176,6 +1363,11 @@
saving={savingSection === "providerCooldown"}
onSave={(providerCooldown) => savePatch("providerCooldown", { providerCooldown })}
/>
<SelfHealingCard
value={data.selfHealing}
saving={savingSection === "selfHealing"}
onSave={(selfHealing) => savePatch("selfHealing", { selfHealing })}
/>
<ModelLockoutCard />
</div>
);
Expand Down
5 changes: 5 additions & 0 deletions src/app/api/resilience/route.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -136,6 +136,7 @@ export async function GET() {
comboCooldownWait: resilience.comboCooldownWait,
quotaShareConcurrencyLimit: resilience.quotaShareConcurrencyLimit,
providerCooldown: resilience.providerCooldown,
selfHealing: resilience.selfHealing,
legacy: buildLegacyResilienceCompat(resilience),
});
} catch (err: unknown) {
Expand Down Expand Up @@ -208,6 +209,9 @@ export async function PATCH(request) {
providerCooldown: body.providerCooldown as ResilienceSettingsPatch["providerCooldown"],
}
: {}),
...(body.selfHealing
? { selfHealing: body.selfHealing as ResilienceSettingsPatch["selfHealing"] }
Comment on lines +212 to +213

@chatgpt-codex-connector chatgpt-codex-connector Bot Jun 29, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Allow selfHealing through the resilience PATCH schema

This body.selfHealing branch is unreachable for the dashboard payload because validateBody(updateResilienceSchema, rawBody) uses a strict schema that has not been extended with a selfHealing property. When the new Self-Healing card calls PATCH /api/resilience with { selfHealing: ... }, validation returns 400 before this merge runs, so users cannot save the new settings.

Useful? React with 👍 / 👎.

: {}),
...normalizeLegacyPatch(body),
});

Expand Down Expand Up @@ -244,6 +248,7 @@ export async function PATCH(request) {
},
comboCooldownWait: nextResilience.comboCooldownWait,
providerCooldown: nextResilience.providerCooldown,
selfHealing: nextResilience.selfHealing,
legacy: buildLegacyResilienceCompat(nextResilience),
});
} catch (err: unknown) {
Expand Down
Loading
Loading