fix(sse): rotation attribution diagnostics (skipped accounts, serving id, request correlation)

changelog.d/fixes/14223-rotation-attribution-skipped-served-correlation.md

@@ -0,0 +1 @@
+- **fix(sse):** skipped-account diagnostics for multi-account rotation: accounts passed over while cooling down are now named (masked id + remaining time, one info line each per request) and their per-account state is readable from the resilience connections endpoint, so an operator can tell skipped accounts apart from unused ones; proxy log entries also carry the masked serving account and the request correlation id when the rotation attribution flag is on (off by default, no selection change) ([#14223](https://github.com/diegosouzapw/OmniRoute/pull/14223)) — thanks @maxmad64bis

config/quality/file-size-baseline.json

@@ -1,4 +1,5 @@
 {
+  "_rebaseline_2026_09_20_14223_rotation_attribution_growth": "PR #14223 own growth: rotation attribution diagnostics (masked serving-account id + request correlation id on proxy log rows, per-account rotation state, skip lines). Re-measured on the tree reconciled with release/v3.8.51 @59de50e4 (which brought #14149 -- the MuseSpark block and the per-request format/session context moved out of opencode.ts -- #14226 and #14464): open-sse/executors/opencode.ts 1251->1318 gate count (re-measured again after #14353 moved the member list off the instance) (snapshotEntries + logSkippedCooldownAccounts + attribution wiring at the nine existing rotation exits incl. the two park-and-replay returns; logic kept at the rotation seam), src/sse/handlers/chat.ts 2547->2559 (attribution sink type + flag read + two forwarded fields at the existing log call-site), src/sse/handlers/chatHelpers.ts 1253->1257 (+4, two additive optional params forwarded to the journal row), open-sse/utils/proxyFetch.ts 1268->1287 (+19, AppliedProxySink rotationAccount field + noteRotationAccount helper). Irreducible spec wiring at existing chokepoints, additive and flag-off inert. Covered by 6 new test files (17 tests).",
   "_rebaseline_2026_09_20_prettier_frozen": "Prettier-only pass over frozen proxy files (verified green on the base with split(\"\\n\").length counting): open-sse/utils/proxyFetch.ts 1276->1268 (inherited shrink ratchet, re-measured on the reconciled release/v3.8.51 tip @373c31f3 -- not this PR's growth); src/sse/handlers/chat.ts 2547= ; src/app/(dashboard)/dashboard/settings/components/ProxyRegistryManager.tsx 1477= (iso-LOC format).",
   "_rebaseline_2026_09_21_14250_member_egress_lines": "PR #14250 own growth: src/app/(dashboard)/dashboard/settings/components/ProxyRegistryManager.tsx 1477->1479 (+2 = the PoolMemberEgressLines import and its one-line mount under the pool members label, next to PoolEgressObservation). The member-egress observation itself lives outside the frozen file, all under cap: PoolMemberEgressLines.tsx, the dedicated GET /api/settings/proxies/pool/member-egress route, readPoolMemberEgressObservation in src/lib/proxyPoolEgressObservation.ts and getRecentEgressIpForProxy in src/lib/db/proxyLogs.ts. Only the mount point is irreducible. Covered by tests/unit/proxy-pool-member-egress-route.test.ts and tests/unit/ui/PoolMemberEgressLines.test.tsx.",
   "_rebaseline_2026_09_22_opencode_train10c_drift": "Release-tip drift: open-sse/executors/opencode.ts 1247->1251, left by the train-10c merge wave (2026-09-22) — the PR->release fast-gates do not run check:file-size, so the tip went red for every train boarding afterwards. Absorbed once at the tip under the owner-approved train-rebaseline policy (see _rebaseline_2026_09_18_merge_train_8_frozen_growth). Structural shrink tracked in #3501.",
@@ -369,7 +370,7 @@
   "_rebaseline_2026_07_27_3850_relax_filesize_cap": "OWNER-APPROVED TEMPORARY relax for v3.8.50-3.8.54 PREPARE phase (docs/ROADMAP.md). cap 800->900 (+100), testCap 800->900 (+100). Targets: decompose-existing-frozen unchanged (frozen still only-shrink); this only relaxes the cap for NEW files in the decompose/extract-while-PREPARE phase (.51='executor registry in-place' and .52='combo.ts decomposition' create new leaf modules above 800). RE-TIGHTENING MANDATORY in v3.8.51: cap target 850 = 850 once decomposition wave stabilizes. SUPERSEDED by _rebaseline_2026_07_27_3850_relax_filesize_cap_v2_20pct (v1 +20% buffer) — retained for audit. Tracked via same roadmap issue.",
   "_rebaseline_2026_07_27_v3849_train1h": "Merge-train 1H (31 PRs) — owner-approved 2026-07-27. Two distinct causes, kept separate on purpose: (1) GENUINE irreducible growth at existing chokepoints — providerLimits/auth (#8632 Kimi quota-reset recovery), rateLimitManager (#8616 idle wedged limiters), models-catalog-route.test (#8610 OpenCode Go effort aliases); (2) COLLISION with #8585, which banked shrinks measured on the pre-train release tip while 30 sibling PRs in the SAME train grew those files again — chat/accountFallback (#8628), chatCore (#8613), videoGeneration (#8581), imageGeneration. The zero-headroom frozen entries cannot absorb either. Ceilings re-pinned to the post-merge tip; #8612 (also in this train) automates shrink-banking so this self-inflicted drift stops recurring. Detail: src/lib/usage/providerLimits.ts 1006->1013 (#8632); src/sse/services/auth.ts 2492->2508 (#8632); open-sse/services/rateLimitManager.ts 1014->1060 (#8616); src/sse/handlers/chat.ts 1842->1845 (#8628); open-sse/handlers/chatCore.ts 4939->4955 (#8613); open-sse/handlers/imageGeneration.ts 3100->3101 ((sem PR — teto do #8585)); open-sse/handlers/videoGeneration.ts 1038->1063 (#8581); open-sse/services/accountFallback.ts 1965->1966 (#8628); tests/unit/models-catalog-route.test.ts 1608->1636 (#8610)",
   "frozen": {
-    "src/sse/handlers/chatHelpers.ts": 1253,
+    "src/sse/handlers/chatHelpers.ts": 1257,
     "_rebaseline_2026_09_17_13720_merge_release_v3851": "Merge de release/v3.8.51 na #13720 (2026-09-17). src/sse/handlers/chatHelpers.ts 1246 -> 1253, decomposto: 1246 -> 1250 e crescimento INHERITED do tip (base-red ja presente em origin/release/v3.8.51 no commit 9688032451fc, arquivo com 1250 linhas contra cap 1246 — nao e desta PR e nao foi introduzido por este merge); 1250 -> 1253 sao as MESMAS +3 linhas da propria #13720 ja auditadas e aprovadas pelo dono na entrada _rebaseline_2026_09_16_13720_suffix_effort_propagation abaixo (threading de resolvedThinkingEffort). Nenhum outro teto foi tocado por este merge; tests/unit/chatcore-translation-paths.test.ts (3449 > 3447) permanece vermelho de proposito — e base-red herdado e a PR nao toca o arquivo.",
     "_rebaseline_2026_09_16_13720_suffix_effort_propagation": "OWNER-APPROVED 2026-09-16 (explicit exception for this unit only, chatHelpers.ts only). PR #13720 (HouMinXi, suffix-effort propagation across model attempts): src/sse/handlers/chatHelpers.ts merge-base (before PR's own commit) was 1164; the release tip independently grew it to 1213 (+49, unrelated merged PRs) while the frozen cap sat at 1214 to cover exactly that tip growth. The PR's own diff on this file is +3 lines only (threading resolvedThinkingEffort: one field on resolveModelOrError's return object, one destructured param and one passthrough call-site argument in executeChatWithBreaker — see commit 4fdb0c5851b7f645efbe5627cd0babb0c3d230c3), taking the merged result to 1216 (1217 per check-file-size.mjs's countLines, which counts the trailing newline as an extra split segment). All 3 added lines are single-property additions inside existing multi-line object literals/signatures; there is no redundant or duplicated line in the PR's own hunks to trim, and none of the +3 lines are outside the PR's own diff. Covered by tests/unit/suffix-effort-propagation.test.ts (27/27), tests/unit/chatcore-upstream-body.test.ts + tests/unit/request-dedup-tenant-isolation.test.ts (57/57), all green against this exact head.",
     "_rebaseline_2026_09_17_13947_tip_growth": "Base-red drain da PR #13947 (Refs #13866) — crescimento de PRODUCAO que chegou pelo tip e nunca foi rebaselinado; nenhum destes arquivos e tocado por esta PR. #12906 (d70f43d4, retry empty_response 502 + timeout de inicio de resposta ciente de reasoning): src/sse/handlers/chat.ts 2498->2500, src/sse/handlers/chatHelpers.ts 1231->1245, open-sse/utils/proxyFetch.ts 1275->1276, open-sse/utils/stream.ts 3098->3123. #12904 (f3acf4f8, injecao unica do system prompt global pos-traducao) + #12910 (051576fd, finalizacao de cache semantico por request id exato): open-sse/handlers/chatCore.ts 6181->6203. Anteriores ao lote, ja acima do cap na base 3d5baf13: open-sse/handlers/imageGeneration.ts 3293->3304 (#13748, b97338a8) e open-sse/services/combo/roundRobinCombo.ts 1213->1221 (#13776, aeba6b1a). Registrado contra o estado mergeado; nenhum outro cap e tocado.",
@@ -492,7 +493,7 @@
     "open-sse/services/combo/executeTargetAttempt.ts": 1273,
     "open-sse/translator/response/openai-responses.ts": 1518,
     "open-sse/utils/cursorAgentProtobuf.ts": 1588,
-    "open-sse/utils/proxyFetch.ts": 1268,
+    "open-sse/utils/proxyFetch.ts": 1287,
     "open-sse/utils/stream.ts": 3239,
     "open-sse/vendor/codex-chatgpt-web/adapters/chatgpt-web/browser-worker.ts": 4398,
     "open-sse/vendor/codex-chatgpt-web/bridge.ts": 1335,
@@ -521,7 +522,7 @@
     "src/shared/components/RequestLoggerV2.tsx": 1748,
     "src/shared/constants/providers/apikey/gateways.ts": 1535,
     "src/shared/services/cliRuntime.ts": 1296,
-    "src/sse/handlers/chat.ts": 2547,
+    "src/sse/handlers/chat.ts": 2559,
     "src/sse/services/auth.ts": 3592,
     "tests/unit/account-fallback-service.test.ts": 2453,
     "tests/unit/provider-validation-specialty.test.ts": 4656,
@@ -532,7 +533,7 @@
     "open-sse/executors/deepseek-web.ts": 1224,
     "open-sse/executors/default.ts": 1205,
     "open-sse/services/rateLimitManager.ts": 1329,
-    "open-sse/executors/opencode.ts": 1303
+    "open-sse/executors/opencode.ts": 1318
   },
   "_rebaseline_2026_09_15_roundrobin_dashboard_events": "Fix #13089 (Combo Studio Live dashboard shows an empty backlog for round-robin combos): open-sse/services/combo/roundRobinCombo.ts 1205->1213. Round-robin is the only combo strategy that bypasses handleComboChat/executeTargetAttempt.ts, the path that publishes the combo.target.attempt/succeeded/failed EventBus events the Live dashboard listens for — so round-robin completions never showed up. The new call-site wiring (createRRDashboardEvents(...) instantiated once per target, one-line .attempt()/.succeeded()/.failed() calls at the 6 existing dispatch/outcome points) is the emitter logic actually extracted into a new module, open-sse/services/combo/rrDashboardEvents.ts — this is the minimum irreducible footprint for wiring 6 required call sites into 6 fixed control-flow points of the frozen file. Covered by tests/unit/issue-13089-roundrobin-live-ws-events.test.ts (2 tests: success + failure paths).",
   "_rebaseline_base_2026_08_10_proxyfetch": "Base-red fix (green-prs sweep, issue #9985): open-sse/utils/proxyFetch.ts 1207 > cap 1000 — new proxied-TLS fetch helper introduced by the Fal reference-image work. Owner-authorized quick rebaseline to green; structural slim tracked for v3.9.0.",

docs/reference/FEATURE_FLAGS.md

@@ -46,7 +46,7 @@ A boolean flag is considered **enabled** when its effective value is `"true"`,
 
 ## Flag Catalog
 
-76 flags across 6 categories. **Default** is the definition default — the value
+77 flags across 6 categories. **Default** is the definition default — the value
 used when neither a DB override nor an environment variable is present.
 
 ### Security (10)
@@ -64,7 +64,7 @@ used when neither a DB override nor an environment variable is present.
 | `AUTH_LOG_INCLUDE_ACCOUNT_ID`           | boolean | `false`  | Include account prefix in AUTH log lines (e.g. "Using <provider> account: abc12345..."). Disabled by default so account identifiers are redacted from shared/multi-tenant process logs. Independent from Debug Mode; flipping Debug Mode does not reveal this. |
 | `OMNIROUTE_OIDC_DISABLE_PASSWORD_LOGIN` | boolean | `false`  | When OIDC is enabled, disable password login so users can only authenticate via OIDC Single Sign-On. When disabled (default), both password login and OIDC are available.                                                                                      |
 
-### Network (18)
+### Network (19)
 
 | Key                                             | Type    | Default | Restart | Description                                                                                                                                                                                                                                                                                                                                                                                                                                             |
 | ----------------------------------------------- | ------- | ------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
@@ -74,6 +74,7 @@ used when neither a DB override nor an environment variable is present.
 | `PROXY_AUTO_SELECT_ENABLED`                     | boolean | `false` |         | When no proxy is assigned to a connection, auto-select the first working proxy from the registry. Off by default (otherwise any registry proxy becomes a global fallback — #3332).                                                                                                                                                                                                                                                                      |
 | `OMNIROUTE_CONTROL_PLANE_PROXY_DIRECT_FALLBACK` | boolean | `false` |         | Allow OAuth and provider validation flows to bypass a pinned proxy and connect directly when proxy reachability pre-checks fail. Off by default because this can change egress IP.                                                                                                                                                                                                                                                                      |
 | `NETWORK_ROTATION_SHARED_EGRESS_GUARD`          | boolean | `true`  |         | On a network exception (timeout, connection refused/reset) for a multi-account rotation executor, when the failing account has no dedicated proxy, apply a short cooldown and skip other proxy-less accounts for the rest of the request instead of retrying each one. On by default (safe: no egress IP change, only reduces latency/cooldown risk on shared-egress accounts). Disable to restore immediate propagation on the first proxy-less throw. |
+| `ROTATION_ATTRIBUTION`                          | boolean | `false` |         | Opencode rotation records which account served or was skipped (masked ids only, never full account ids) and links proxy log entries to their request, so the operator can tell skipped accounts apart from unused ones. Off by default.                                                                                                                                                                                                                 |
 | `PROXY_SKIP_RECENTLY_FAILED`                    | boolean | `false` |         | Proxy pools and the per-account rotation of opencode stop re-serving a proxy that just failed (refused TCP probe, or a 429 received through it) for a per-process period that doubles on each repeat, up to a cap. No proxy status is written; with every candidate set aside the choice is unchanged. Off by default.                                                                                                                                  |
 | `PROXY_POOL_EGRESS_OBSERVATION`                 | boolean | `false` |         | Show, under a proxy pool in the dashboard, how many observed egress IPs served its members over the last 24 h and how many connections used them. Read-only, computed from the proxy log, never used for routing. Off by default.                                                                                                                                                                                                                       |
 | `OPENCODE_RESPONSES_STALL_ROTATION`             | boolean | `false` |         | For the OpenCode executor, watch the first body byte of a streamed Responses reply (window: `RESPONSES_FIRST_BYTE_TIMEOUT_MS`, default `15000`). A 2xx Responses stream that stays silent past the window is treated as stalled: the account is cooled down and the request rotates to the next account once; a second stall fails fast. Off by default: stalled streams keep today's wait until the stream readiness timeout.                          |
@@ -216,7 +217,7 @@ Returns every flag with its effective value, source, and a summary.
       "requiresRestart": false,
       "warningLevel": "caution",
     },
-    // ... all 75 flags
+    // ... all 77 flags
   ],
   "summary": {
     "total": 56,

open-sse/executors/accountRotation.ts

@@ -183,6 +183,31 @@ export function maskAccountId(fingerprint: string): string {
   return `${fingerprint.slice(0, 8)}…`;
 }
 
+/** One per-account rotation entry, keyed by connection id (never a fingerprint). */
+export interface RotationAccountSnapshot {
+  /** Already-masked id (`maskAccountId` output) — never the full fingerprint. */
+  masked: string;
+  ready: boolean;
+  cooldownUntilMs: number | null;
+  consecutiveFails: number;
+}
+
+const rotationSnapshots = new Map<string, RotationAccountSnapshot[]>();
+
+/** Record the current rotation state for a connection (sync, in-memory only). */
+export function recordRotationSnapshot(
+  connectionKey: string,
+  entries: RotationAccountSnapshot[]
+): void {
+  rotationSnapshots.set(connectionKey, [...entries]);
+}
+
+/** Read the last recorded rotation state (no side effects — never clears). */
+export function readRotationSnapshot(connectionKey: string): RotationAccountSnapshot[] | null {
+  const snap = rotationSnapshots.get(connectionKey);
+  return snap ? [...snap] : null;
+}
+
 /**
  * Whether a network exception (timeout, connection refused/reset) on this
  * account should trigger rotation to the next account, vs propagating.