[hartmark]

What Problem This Solves

Debugging a live 503 storm on omniroute-dev (currently serving real production traffic) surfaced two real gaps in the resource-pressure admission guard.

1. Rejections were nearly invisible. The admission-layer guard (services/admission/runtime.ts) rejects with a 503 before the request ever reaches chatCore/attemptLogging.ts, the only place that writes call_logs rows — so a resource-pressure rejection never shows up in /dashboard/logs, by architecture, not by bug. The only console signal was [resourcePressure] critical pressure guard tripped (reason=psi_some); returning 503 — a bare reason code with no numbers, so there was nothing to act on when deciding whether the guard is mistuned or genuinely saturated.

2. Recovery lagged more than necessary. PSI's own someAvg10/fullAvg10 are kernel-computed 10-second rolling averages, so real recovery already lags behind the actual event ending. Requiring 3 consecutive samples on top of that (at the ~1s sample cadence) stacked another 2-3 seconds of guard-still-shedding time after the process was genuinely fine again.

Why This Change Was Made

Logging: buildCriticalGuard now takes an optional detail object and logs the actual numbers at the moment of rejection — heapUsedMb/thresholdMb for the immediate heap-trip path, and psiSomeAvg10/psiFullAvg10/cgroupCurrentMb/cgroupMaxMb/recoveryStreak/sampleAgeMs for the cached-critical-state reuse path. Every rejection logs this, not just the first sample that flips state into critical.

Recovery speed: isRecovered() already requires every tracked ratio/PSI value to clear the separate, more conservative recoveryRatio/recoveryPsiAvg10 thresholds (not just dip under the critical ones) before a sample counts as recovered at all — that's real signal, not noise. The sustainedSamplesRecovery streak requirement on top of that conservative bar was redundant delay. Reduced from 3 to 1.

User Impact

An operator watching podman logs/console output during a real pressure event now sees the actual heap/PSI/cgroup numbers driving the rejection, instead of just a reason code. The guard also exits its rejecting state roughly 2-3 seconds sooner once the underlying pressure genuinely subsides, reducing user-visible failed-request duration during a real spike.

Evidence

Full resource-pressure test suite, run against this exact change:

$ node --import tsx/esm ... --test tests/unit/resource-pressure.test.ts tests/unit/resource-pressure-policy.test.ts tests/unit/resource-pressure-runtime.test.ts tests/unit/resource-pressure-sampler.test.ts
ℹ tests 36
ℹ pass 36
ℹ fail 0

$ node --import tsx/esm ... --test tests/unit/execute-chat-resource-pressure-breaker.test.ts
ℹ tests 3
ℹ pass 3
ℹ fail 0

39/39. Sample of the new log output (from the test run itself):

[resourcePressure] critical pressure guard tripped (reason=v8_heap_absolute heapUsedMb=201 thresholdMb=200); returning 503
[resourcePressure] critical pressure guard tripped (reason=v8_heap_ratio psiSomeAvg10=null psiFullAvg10=null cgroupCurrentMb=null cgroupMaxMb=null recoveryStreak=0 sampleAgeMs=11); returning 503

eslint clean on both touched files. tsc --noEmit shows 2 pre-existing errors unrelated to this change (confirmed present on the unmodified base too, in an unrelated auth/tlsFingerprint file).

Applied directly to the live omniroute-dev deployment first via hot-reload, since this was actively affecting real production traffic during debugging — confirmed the dashboard stayed healthy (200s, no compile/HMR errors) after the change. This PR is the matching clean-history change against release/v3.8.51.

[hartmark]

Follow-up: the guard tripped again in production even with the faster-recovery fix, while OmniRoute's own container usage was trivial (282MB/33GB). Root-caused: /proc/pressure/memory reflects host-wide PSI, not this process's own cgroup pressure (confirmed by comparing it against /sys/fs/cgroup/memory.pressure from inside a running container — the two differ). On a shared host, contention from unrelated processes was enough to trip the guard.

Bumped highPsiAvg10/criticalPsiAvg10/recoveryPsiAvg10 by 50% (20/40/10 → 30/60/15) to tolerate that noise. The ratio-based thresholds (this process's own real OOM safety margin) are untouched. resource-pressure-policy.test.ts + resource-pressure.test.ts (16/16) and the complexity ratchet both pass.