Repository navigation
fix(ops): clamp embedding inputs to model context; egress-IP lockout provider override for paid plans; glm-5.3 vision fragment - #12662
Closed
alvinveroy wants to merge 9 commits into
Conversation
…ode background identity headers + staged resilience work # Conflicts: # open-sse/executors/base.ts # open-sse/handlers/embeddings.ts # open-sse/services/combo.ts # open-sse/services/opencodeQuotaFetcher.ts # src/app/(dashboard)/dashboard/settings/components/ResilienceTab.tsx # src/i18n/messages/in.json # src/lib/usage/callLogArtifacts.ts # src/shared/constants/visionModels.ts # src/shared/utils/httpClientAbortGuard.mjs # tests/unit/call-log-cap.test.ts # tests/unit/httpClientAbortGuard.test.mjs
…10, with a first-byte watchdog Root cause of the 2026-09-03/04 outage: with ENABLE_TLS_FINGERPRINT=true every direct inference dispatch rode the wreq-js (browser-JA3) transport, whose responses resolved but streamed ZERO bytes for 90-600s on opencode-go / command-code / bailian (agents aborted -> 499 storms; the then-live 110s headers-wait default turned the hangs into 504s). Plain node fetch streamed the same requests in seconds and control-plane calls were unaffected — the stall is transport+streaming-specific, not DNS/CF slow-walking. - wreq is no longer eager by default: plain direct fetch is the fast path; a Cloudflare fingerprint rejection (403 + 1010 / browser_signature_banned) arms a 1h per-provider cache and retries once via wreq (replay-safe requests only). TLS_FINGERPRINT_PROVIDERS keeps its legacy eager semantics, now guarded. - Every wreq response (direct, ladder retry, proxied overlay) carries a first-byte watchdog (TLS_FINGERPRINT_FIRST_BYTE_WATCHDOG_MS, default 10s): no first byte -> cancel and fall back to the direct dispatcher instead of hanging the caller. Verified live: fingerprint off + ladder armed-on-demand restores TTFB 3-6s on opencode-go (was 90-600s of zero bytes), zero 1010 recurrence in the window.
…out provider allowlist override; glm-5.3 vision fragment - embeddings handler: string inputs are clamped per-item to 20k chars before upstream dispatch — oversized documents no longer fail with 'Invalid input: maximum context length is 8192 tokens' (121 production 400s in one day on openrouter/openai/text-embedding-3-small). Same constant/rationale as the memory-retrieval clamp in src/lib/memory/embedding/remote.ts. - providerErrorRules: OMNIROUTE_EGRESS_IP_LOCK_PROVIDERS env override for the egress-bucketed lockout family (cheaper-inference#9611). PAID plans are account-bucketed, not IP-bucketed — one account's plan 429 cascading 'Shared egress IP quota exhausted' across ALL sibling connections for days is a misclassification for paid operators (production: 1419x 503 ALL_TARGETS_SKIPPED storm). none/off/false/0 disables; CSV replaces the family. - visionModels: glm-5.3 family fragment supersedes the narrow glm-5.3-flash entry so /v1/models marks every glm-5.3* variant vision-capable.
… watchdog 2026-09-05 wedge incident: opencode h2 stream resets (ERR_HTTP2_STREAM_ERROR, UND_ERR_SOCKET) fanned out through every combo target (8 fallbacks / 18 decisions per request), then a client abort during combo cleanup left the event loop wedged — listener accepting, nothing served, nothing logged, for hours. The existing supervisor only restarts on process exit. - Bounded transport abort (both dispatchers): consecutive transport-class network failures share the same egress — after DEFAULT_TRANSPORT_ABORT_BOUND (3), abort the combo with an explicit 502 instead of burning the remaining fan-out. Non-transport failures reset the counter. New predicates isTransportClassNetworkError / shouldAbortComboForTransportFailures in comboPredicates.ts. - Self-probe wedge watchdog (scripts/dev/self-probe-watchdog.mjs, wired in standalone-server-ws.mjs after the listener accepts): loopback probe of /api/status on the same event loop — a wedged loop cannot answer it. N consecutive connect/timeout failures (any HTTP response counts as alive, incl. 401) exit(1) so launchd KeepAlive / systemd / Docker relaunch. Config: OMNIROUTE_SELF_PROBE(_INTERVAL_MS|_TIMEOUT_MS|_THRESHOLD), default 60s/15s/3; OMNIROUTE_SELF_PROBE=off disables. - file-size rebaseline for the touched frozen files with justification.
…in TLS watchdog; stryker tap registration - dashboard/combos/page.tsx: defer the post-hydration localStorage correction one microtask so setShowUsageGuide is not a synchronous setState inside the effect body (react-hooks/set-state-in-effect base red on release/v3.8.51 from cheaper-inference#12448); eslint:json gate green. - proxyFetch.ts withTlsFirstByteWatchdog: the watchdog race consumed the first body chunk from the reader but never re-emitted it — the first SSE frame was silently dropped. The rest stream now re-emits the captured chunk before continuing the original stream. - stryker.conf.json: register 10281-combo-reasoning-probe + transport-bounded-abort in tap.testFiles (they cover mutated modules; --strict gate). - callLogArtifacts.ts / earlier dashboard stray: restore pristine remote state for files untouched by this branch (resolves unused-var lint).
Contributor
Author
|
Superseded by #12531 — consolidating the 3.8.51 fork batch. This head branch carried the same cumulative content (this PR's topic + the TLS-fingerprint, opencode-identity, and ops work) on an older base and is CONFLICTING with current release/v3.8.51. #12531 now holds the full updated batch merged onto the latest release/v3.8.51 (17 upstream fixes), all gates green locally (changelog integrity, docs-all, agent-skills sync, file-size), with per-area change notes and tests. Closing as duplicate; reopen only if you want this topic split out separately. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #12658
1. Embeddings input clamp
/v1/embeddingsproxied oversized documents verbatim → upstream 400 "Invalid 'input': maximum context length is 8192 tokens" (121× in one day onopenrouter/openai/text-embedding-3-small). String inputs are now clamped per-item to 20k chars before dispatch — same constant and rationale as the memory-retrieval clamp insrc/lib/memory/embedding/remote.ts. A truncated input keeps embeddings working; an unclipped one guarantees a 400.2. Egress-IP lockout override
applyEgressIpLockout()assumes opencode's free tier is IP-bucketed (#9611). PAID plans are account/workspace-bucketed, so one account's plan-level 429 cascaded "Shared egress IP quota exhausted" across ALL sibling connections for days → 1419× 503ALL_TARGETS_SKIPPEDin one hour.New
OMNIROUTE_EGRESS_IP_LOCK_PROVIDERS:none|off|false|0→ egress-IP lockout disabled;a,b,c→ exact replacement set.3. visionModels:
glm-5.3family fragmentSupersedes the narrow
glm-5.3-flashentry so/v1/modelsmarks everyglm-5.3*variant vision-capable for agent model pickers.Testing
tests/unit/embeddings-input-clamp.test.ts(4) +tests/unit/egress-ip-lock-env-override.test.ts(4) — all green.