fix(server): rotate Claude on hard usage rejection

[tusharbhardwaj-bk]

What Changed

• detect only Claude SDK rate_limit_event messages whose authoritative rate-limit status is rejected and whose limit type is supported
• call the credential-free host contract claude-autoswitch --hard-limit <type> --json, validate its exit/JSON/type outcome, and stop only the emitting provider session after status == "switched"
• debounce repeated copies of one hard-limit condition while leaving unrelated Claude sessions and non-hard errors untouched
• document the event-driven contract, staging verification, feature-flag rollback, and preserved timer/manual behavior

Why

Claude can authoritatively reject a request while its local utilization cache remains stale. The healthy 30-second fallback timer then leaves the running provider process pinned to an exhausted startup account for minutes. This hook uses the host-owned atomic election contract immediately and never reads, copies, logs, or moves credentials in application code.

Base / Staging Resync Evidence

At PR creation, expbkmain was six commits behind bkmain, so GitHub initially displayed the already-production #123/#124 files beside TEC-961.

• git rev-list --left-right --count origin/expbkmain...origin/bkmain returned 0 6: zero staging-only commits and six production-side commits
• git merge-base --is-ancestor origin/expbkmain origin/bkmain succeeded
• the six commits are already-promoted Bound activity payload bytes, and reclaim what the uncapped ones already wrote #113/Promote expbkmain to bkmain #119 history plus the production fix(server): stop one wedged provider stop from stalling every queued prompt #123/fix(server): bound provider command lane stalls #124 provider-lifecycle fixes
• expbkt3 reported zero running sessions before the reset
• expbkmain was then fast-forwarded, without force, from 9a5596b6 to 966bcc67 per the documented between-experiments reset path

No experimental commit was discarded or silently mixed. After the resync, this PR is the four-file TEC-961 diff based on origin/bkmain.

Verification

• VITEST_MAX_THREADS=2 vp test run apps/server/src/provider/claudeHardLimitRotation.expbkt3.test.ts --maxWorkers=2 — 1 file, 7 tests passed
• node scripts/check-fork-markers.ts — passed
• vp fmt apps/server/src/provider/claudeHardLimitRotation.expbkt3.ts apps/server/src/provider/claudeHardLimitRotation.expbkt3.test.ts apps/server/src/server.ts docs/operations/claude-account-switching.md --check --threads 2 — passed
• git diff --check origin/bkmain...HEAD — passed

Staging Plan

Merge to expbkmain, wait for the SHA-addressed deployment, verify /home/ubuntu/.t3/expbkt3-dev/deployed-sha and t3-expbkt3.service, then exercise only the repository-supported controlled validation for: one hard-limit election/recycle, duplicate suppression, unrelated-session isolation, and non-hard-error ignore behavior. The machine-global election will not be triggered live without a quiescent safe gate.

Rollback

Set T3_CLAUDE_HARD_LIMIT_ROTATION=0 for expbkt3 and restart only t3-expbkt3.service to disable the event listener while keeping claude-autoswitch.timer active. Revert this PR for a complete application rollback. The validated host rollback copy remains /home/ubuntu/.local/bin/claude-autoswitch.bak-20260820-tec961.

TEC-961

https://linear.app/beknown/issue/TEC-961/rotate-claude-immediately-after-a-hard-usage-limit-rejection

Implemented by gpt-5.6-sol via Codex in T3 Code.

[github-actions]

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

ℹ️ No successful main baseline artifact is available yet. This run establishes the initial measurement.

Provider	Metric	Main baseline	This PR	Impact	PR ceiling
Codex	Total thread wire	—	11.4 KiB	—	15.1 KiB	✅
Codex	Thread snapshot wire	—	5.7 KiB	—	7.3 KiB	✅
Codex	Live turn WebSocket wire	—	5.7 KiB	—	7.8 KiB	✅
Codex	Live turn WebSocket decoded	—	50.3 KiB	—	66.4 KiB	✅
Codex	Live turn messages	—	10	—	21	✅
Claude	Total thread wire	—	11.4 KiB	—	15.1 KiB	✅
Claude	Thread snapshot wire	—	5.7 KiB	—	7.3 KiB	✅
Claude	Live turn WebSocket wire	—	5.7 KiB	—	7.8 KiB	✅
Claude	Live turn WebSocket decoded	—	51.2 KiB	—	66.4 KiB	✅
Claude	Live turn messages	—	11	—	21	✅

Baseline: unavailable · PR result: 5d192b2 · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

• Codex decoded thread snapshot: 95.6 KiB
• Claude decoded thread snapshot: 96.3 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.