Skip to content

fix(resilience): self-probe wedge watchdog + pack-artifact closure allowlist - #12851

Closed
alvinveroy wants to merge 3 commits into
cheaper-inference:release/v3.8.51from
alvinveroy:pr/g2-self-probe-watchdog
Closed

alvinveroy wants to merge 3 commits into
cheaper-inference:release/v3.8.51from
alvinveroy:pr/g2-self-probe-watchdog

Conversation

@alvinveroy

@alvinveroy alvinveroy commented Sep 6, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Add a self-probe wedge watchdog to server-ws.mjs and the standalone-server-ws wrapper, plus the matching pack-artifact allowlist entry so the new module survives the npm tarball prepublish prune.

Also carries the upstream-merge-gate drift fixes (TypeScript api-typecheck baseline + docs migration count + i18n mirrors + changelog fragment normalization) inherited from origin/release/v3.8.51. These drift fixes are required for ANY fork PR into release/v3.8.51 to pass CI until the upstream base absorbs them.

Related Issues

Validation

Change type: build-deploy / resilience. Focused loop from CONTRIBUTION_GOLDEN_PATH:

  • Change type: build-deploy
  • Focused tests and category gates from the golden path (pack-artifact closure tests + watchdog regression test)
  • npm run lint
  • Reconciled with the current active release base (origin/release/v3.8.51); focused checks rerun afterward
  • Production-code changes include a new automated test in this PR

Commands run on the VPS (ai.servers.com, deploy@23.111.17.228):

git worktree add /home/deploy/g2-final-check pr/g2-self-probe-watchdog
ln -sfn /home/deploy/OmniRoute/node_modules /home/deploy/g2-final-check/node_modules
export PATH=/home/deploy/node24/bin:/usr/local/bin:/usr/bin:/bin

for f in tests/unit/self-probe-watchdog.test.mjs \
         tests/unit/pack-artifact-server-ws-closure.test.ts \
         tests/unit/pack-artifact-entrypoint-closures.test.ts \
         tests/unit/build/assemble-standalone.test.ts; do
  node --import tsx/esm --import ./open-sse/utils/setupPolyfill.ts \
       --import ./tests/_setup/isolateDataDir.ts --test "$f"
done

npm run check:api-typecheck        # PASS (290 errors, all within frozen baseline)
npm run check:file-size            # PASS
npm run check:changelog-integrity  # PASS
npm run check:docs-all             # PASS

Final result: 4/4 focused tests green, all 4 gate categories green on VPS.

Tests Added Or Updated

  • tests/unit/self-probe-watchdog.test.mjs — pure decision helpers + loopback probe simulator; covers evaluateSelfProbeConfig parsing, shouldConsiderAlive semantics (any HTTP response, including 401, counts as alive), and the OMNIROUTE_SELF_PROBE=off escape hatch.
  • tests/unit/pack-artifact-server-ws-closure.test.ts — new assertion: self-probe-watchdog.mjs in allowlist.
  • tests/unit/pack-artifact-entrypoint-closures.test.ts — new assertion: dist/self-probe-watchdog.mjs in required paths.
  • tests/unit/build/assemble-standalone.test.ts — new assertion: self-probe-watchdog.mjs in EXTRA_MODULE_ENTRIES.

Coverage Notes

  • Production code changed: scripts/dev/standalone-server-ws.mjs (one import + one watchdog call) and scripts/dev/self-probe-watchdog.mjs (new file).
  • Coverage gate (npm run test:coverage, 60% statements/lines/functions/branches) remains green: the new watchdog's pure helpers and the loopback-probe wiring are exercised by the new tests.

Reviewer Notes

  • No config break: OMNIROUTE_SELF_PROBE=off is the only escape hatch; defaults to enabled, threshold 3 failures × 15s timeout × 60s interval = ~3 min worst case before forced restart.
  • Why a watchdog instead of metric-gated restart: the listener keeps accepting TCP during the wedge, so any external liveness probe (k8s livenessProbe, AWS ALB target group health check) sees the port as live. The probe runs on the same event loop it monitors — only a wedged loop can fail to answer a localhost HTTP probe.
  • No external network surface: the probe hits 127.0.0.1:<port>/api/status only. Same-loop answer = alive. Counter increments only on connect-refused or timeout, not on any HTTP response (401 counts as alive — OmniRoute answers 401 when healthy and keyless).
  • Supervised process: relies on launchd KeepAlive / systemd Restart=always / docker restart=always to relaunch after exit(1). Documented in the watchdog JSDoc.
  • Drift-fix prerequisite commit: fix(ci): clear merge-gate drift inherited from upstream release/v3.8.51 carries the api-typecheck baseline entry for open-sse/executors/glm.ts (pre-existing TS2554 base-red), the 169→170 migration-count updates in README.md/AGENTS.md/llm.txt + 41 i18n mirrors, and the changelog fragment normalization for reset-aware-model-family.md. All three are inherited from the upstream release base; every fork PR into release/v3.8.51 trips them until the upstream base absorbs them.

@alvinveroy

Copy link
Copy Markdown
Contributor Author

/retest

…lowlist

- Adds scripts/dev/self-probe-watchdog.mjs and wires it from standalone-server-ws.mjs
  after the listener accepts. Loopback probe of /api/status on the same event loop —
  a wedged loop cannot answer it. N consecutive connect/timeout failures (any HTTP
  response counts as alive, incl. 401) → process.exit(1) so launchd KeepAlive /
  systemd / Docker restart.
- Config: OMNIROUTE_SELF_PROBE=off disables; OMNIROUTE_SELF_PROBE_INTERVAL_MS
  (default 60000), OMNIROUTE_SELF_PROBE_TIMEOUT_MS (default 15000),
  OMNIROUTE_SELF_PROBE_THRESHOLD (default 3).
- Pack-artifact closures (app staging + npm-shipped required) and the standalone
  build's EXTRA_MODULE_ENTRIES updated to ship scripts/dev/self-probe-watchdog.mjs
  alongside the other server-ws.mjs deps.

Regression tests in tests/unit/self-probe-watchdog.test.mjs.
Carries the upstream-merge-gate drift fixes inherited from origin/release/v3.8.51
that every fork PR trips until the upstream base absorbs them:

- README.md/AGENTS.md/llm.txt (3 files): 169 -> 170 migrations (real count from
  src/lib/db/migrations/).
- docs/i18n/<41 locales>/llm.txt (41 files): same substitution in each mirror
  body so the docs-sync check (which compares every mirror's body to
  stripTopHeading(root llm.txt)) passes.
- config/quality/api-typecheck-baseline.json: register the pre-existing TS2554
  base-red in open-sse/executors/glm.ts (16th positional arg to
  createSSETransformStreamWithLogger expected-but-not-declared on upstream).
- changelog.d/fixes/reset-aware-model-family.md: normalize to bullet form per
  the changelog fragment convention.
- skills/cli-tunnel/SKILL.md: regenerate to match the CLI-tool catalog
  (check:agent-skills-sync exit 2 on the dry-run diff).
check:agent-skills-sync exits 2 on the dry-run diff when the regenerated
SKILL.md differs from the catalog-derived content. The upstream release batch
modified CLI-tool signatures whose SKILL.md reflection was not regenerated.
@alvinveroy
alvinveroy force-pushed the pr/g2-self-probe-watchdog branch from b91bd75 to 1775e19 Compare September 6, 2026 10:20
@alvinveroy

Copy link
Copy Markdown
Contributor Author

/retest

@alvinveroy

Copy link
Copy Markdown
Contributor Author

Closing this PR for now to unblock the queue. The watchdog + pack-closure additions themselves are clean — local and CI gates for changes owned by this PR all pass:

  • ✅ Docs Gates (fast-path)
  • ✅ API Route Typecheck (290 errors, all within frozen baseline)
  • ✅ Merge integrity (changelog + generated skills)
  • ✅ No new ESLint warnings
  • ✅ Vitest (fast-path)
  • ✅ Semgrep / Semgrep-cloud-platform
  • ✅ Change Classification

But three Unit Tests fast-path shards (1/4, 2/4, 3/4) fail with pre-existing reds inherited from upstream origin/release/v3.8.51, not from this PR's diff. Concretely:

These test files exist on release/v3.8.51 but the production-code companions live in the in-flight fix/v3851-* / security/v3851-* branches (huggingchat-stream-error-boundary, dashboard-failure-event-redaction, zai-web-stream-error-boundary, streamhandler-public-errors, public-error-boundaries, codex-response-failed-boundary, error-sanitizer-sk-key-qv45, chat-rejects-image-only-model, kiro-tool-call-validation, model-sync-route). None of those have landed on release/v3.8.51 yet, so the test files assert behaviors the running code doesn't yet satisfy.

Verified upstream-side: on a clean checkout of origin/release/v3.8.51 (f9a1cc8a9), the same 8 tests fail with byte-identical assertion errors. Same on origin/main. These are not regressions introduced by this PR — the existing /api/status-returning 401 means the daemon is healthy, but the test suite is in a half-shipped state.

To keep CI green, the campaign/green-prs pipeline (CONTRIBUTING.md + the mergify/merge-queue workflow) needs to land the upstream fix-branches into release/v3.8.51 first; this PR can then reopen and pass cleanly without modifying the failed test files or their production companions.

The two prerequisite commits I added here (docs-count drift + changelog fragment normalization + api-typecheck baseline entry for the pre-existing glm.ts TS2554 + cli-tunnel SKILL.md regen) are still useful — they close a per-PR gate-noise class that affects every fork PR until the upstream base absorbs them. They will be carried into the reopened PR (or absorbed by a separate infra-drift PR).

Closing now to keep the queue clean for the maintainer. Reopen when the green-prs campaign has landed the fix-branches.

@alvinveroy alvinveroy closed this Sep 6, 2026
@alvinveroy alvinveroy reopened this Sep 6, 2026
@alvinveroy alvinveroy closed this Sep 6, 2026
@alvinveroy
alvinveroy deleted the pr/g2-self-probe-watchdog branch September 6, 2026 10:48
@amorkezouati774-cyber

Copy link
Copy Markdown

kali

@amorkezouati774-cyber

Copy link
Copy Markdown

connorblack added a commit to connorblack/OmniRoute that referenced this pull request Sep 12, 2026
…r/model

getComboTargetModelId() validated each candidate prefix via prefixRoutesToProvider(),
which round-trips through parseModel()'s global ALIAS_TO_PROVIDER_ID table. That
table deliberately canonicalizes "agy" to "antigravity" for auth/execution (cheaper-inference#8013),
so the round trip failed for the "agy/" prefix even though it was exactly how the
target was qualified. The "agy/" qualifier was never stripped, and the unstripped
modelId then resolved against the wrong provider/model pair, so getCanonicalModelMetadata
found no registry/spec/synced-capability match and combo leaves lost their context_length
(and every other derived field) entirely (cheaper-inference#12851).

Trust rawProvider/providerId as strip candidates directly; they are exactly how the
target was already qualified and canonicalized via this same local alias map, so they
need no extra parseModel() round trip.

Also broaden the combo per-target fail-closed gate in catalog.ts: a known context
window (static/registry, synced, or an operator model_context_overrides entry) is
evidence on its own and must not be discarded just because reasoning-effort evidence
for that leaf is unknown.

Claude-Session: https://claude.ai/code/session_01R5zXR8ogxoKtKdxTTJTVBv
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants