[HouMinXi]

fix(sse): default crash-guard logger to console.warn, not console

Problem

installProcessCrashGuard() is called with no argument from
apiBridgeServer.ts, liveServer.ts and embedWsProxy.ts. The default
logger was log ?? console, and the swallow path calls
logger("warn", ...). console is an object, not a function, so the first
client abort that reached the process-level guard threw TypeError: b is not a function inside the uncaughtException handler and took the whole server
down.

Observed in production on 2026-08-29 under load: a burst of client aborts
(ECONNRESET from cancelled streaming requests) hit the guard, the TypeError
killed the process, and systemd restarted it — twice in three minutes
(exit status 7, "uncaughtException: Error: aborted" immediately followed by
"TypeError: b is not a function" in the crash log). The guard introduced by
#11556 to prevent exactly this class of crash had become the crash itself.

Fix

Default the logger to console.warn.bind(console) — a function — when the
caller passes nothing or a non-function:

const logger = typeof log === "function" ? log : console.warn.bind(console);

Verification

• New regression test exercises the no-argument path end to end: installs the
  guard, locates the registered handler, feeds it a real ECONNRESET-shaped
  error, and asserts the abort is swallowed and logged once (the broken
  default throws TypeError right there).
• Bug-injection round trip: reverting to log ?? console makes the new test
  fail; restoring the function default passes.
• Existing guard suite (re-export identity, idempotent install, swallow
  classification) stays green: 9/9.

[HouMinXi]

Load test results: OmniRoute under a 90-agent workload

I ran a staged load test against a production OmniRoute instance (podman,
cgroup memory limit 5 GiB, V8 heap capped at 4096 MB, MAX_HEAVY_IN_FLIGHT=6,
queue 30 s) and measured where the local layer actually breaks. Test plan was
frozen before the first run: stage ladder 1/3/6/12/24/45/60/75/90 concurrent
agents, each stage a mix of light/medium/heavy requests (50/30/20), success
gate >=95% protocol success, no OOM, memory ratio under 75% after load.

The load generator runs real requests through the full local chain
(auth, admission, routing, translation, executor, SSE). For the capacity
numbers below the upstream is a local mock server answering in ~50 ms, so the
numbers isolate OmniRoute itself rather than upstream latency. A separate
run against real upstreams (DeepSeek/SenseNova free tiers) is described
afterwards.

Local-layer capacity

concurrency	success	rejected by admission	peak memory
1-45	100% (405/405)	0	2.6-3.0 GiB
60	99.4%	0.6%	2.88 GiB
75	95.6%	4.4%	2.77 GiB
90	92.6%	7.4%	2.73 GiB

Every single failure past 45 concurrent agents is a chat_admission_busy
503: the heavy-request admission gate doing its job, not a crash, not memory
pressure, not OOM. Stream TTFB stayed between 2.0 and 2.5 s at every level.
End-to-end latency grows with queueing (p50 5.4 s at 12 agents, 38 s at 90),
which is the price of the same backpressure.

Two things stand out to me in this data. Peak memory does not move with
concurrency: 2.6-3.0 GiB at 1 agent and at 90 agents alike. And the instance
survived 27 consecutive runs at up to 90 agents with zero restarts. On the
5 GiB contract, the practical sweet spot is 60-75 concurrent agents with the
current admission settings; 90 is a working backpressure zone, not a failure
zone.

Two production bugs the load test flushed out

Both were found because the load generator does something real traffic mostly
doesn't: abort connections mid-stream.

1. The resource-pressure detector counts reclaimable page cache as memory

resourcePressurePolicy.ts computed its cgroup ratio from raw
memory.current. Under load, page cache grows; at some point the ratio
crossed 0.92 and the guard latched at critical, returning 503 to everything.
The catch: idle memory.current was 4.34 GiB, of which 3.01 GiB was file
cache and only 1.62 GiB anonymous. memory.events all zero, PSI 0.00. There
was no real memory pressure at all.

The recovery path made it worse. isRecovered() compares the same
cache-inclusive ratio against 0.75. I dropped memory.reclaim on the host
until the ratio read 0.16, and the 503s continued for another five minutes
until a container restart. In production this was a 26-minute outage that
started from a single 655-message request plus natural page-cache growth.

Fix: compute the ratio from the working set (memory.current minus the
file field of memory.stat), with a fallback to raw current when the stat
read is unavailable or skewed across the non-atomic reads, and memory.high
keeps comparing the raw total because the kernel throttles on total charge.
PR #12017.

Before/after on the identical stage-24 load: 10/10 requests rejected with
503 and a stop-rule abort, versus 24/24 success at 49% peak memory after the
fix.

2. The client-abort guard crashes the process it protects

#11556 added installProcessCrashGuard() to swallow benign client aborts.
It's called with no argument from apiBridgeServer.ts, liveServer.ts and
embedWsProxy.ts. The default logger was log ?? console, and the swallow
path calls logger("warn", ...). console is an object, not a function, so
the first aborted connection that reached the process-level guard threw
TypeError: b is not a function inside the uncaughtException handler and
took the server down. Twice in three minutes during my test run, each time
recovered by systemd. The guard introduced to prevent exactly this crash was
the crash.

Fix is one line, defaulting the logger to console.warn.bind(console), plus
a regression test that exercises the no-argument path with a real
ECONNRESET-shaped error. After deploying it, 27 runs at up to 90 agents with
deliberate mid-stream aborts produced zero uncaught exceptions and zero
restarts. PR #12042.

Real-upstream runs (context, not the headline)

The same harness ran stages 1-45 against real upstreams. Stage 12 was 72/72.
From stage 24 on, the dominant failure mode shifted to upstream timeouts:
128/144 at 24 agents (SenseNova accounted for 15 of 16 timeouts; DeepSeek was
71/72), 78-84% at 45 agents. The local layer showed zero 503s, zero admission
rejections beyond the expected, zero OOM at every level. The bottleneck from
24 agents up was upstream throughput on free-tier accounts, not OmniRoute.

Setup notes for anyone reproducing

• Memory contract: quadlet sets NODE_OPTIONS=--max-old-space-size=4096 and
  --memory=5g; check that no .env OMNIROUTE_MEMORY_MB shadows it,
  because the startup script appends heap flags last-wins.
• The load harness reserves its scratch dir per run and owns the stop-file
  lifecycle; a metrics collector that hard-exits when a stop file exists will
  race a fast stop-rule trip (mine did, and lost the memory timeline for one
  run until I made it warn and continue).
• In rootless podman, a mock upstream on the host is reachable from the
  container only via host.containers.internal (169.254.1.2), not via
  127.0.0.1 or the bridge subnet address.

The full run artifacts (per-request JSONL, 2 s-interval cgroup metrics,
pre-registration and incident RCAs) are kept with the deployment; the two fix
PRs (#12017, #12042) carry the code-level evidence.