feat(sse): reserve the Antigravity account for the request's stream lifecycle (re-land of #10011) - #13929
Conversation
…ifecycle Re-land of the account-lease half of #10011 on the current release branch. Its exact-model-scoping half had already shipped in #8050 and its quota half lost to the tip's aggregate-family design (selectAntigravityQuotaWindowNames / antigravityQuotaFamily.ts); none of that is reintroduced here. The lease is a concurrency reservation only and never reads or writes quota state. The Antigravity account selected for a request is reserved for the whole streaming lifecycle of that request, so a concurrent retry — or the credential handoff inside getProviderCredentialsWithQuotaPreflight — cannot re-pick an account already committed to an in-flight upstream stream. The reservation is scoped to (connection, callable upstream model) rather than the whole account, so one account can still serve two different models at once; catalog ids that resolve to the same upstream id (the gemini-3.7-flash tiers, all gemini-3.7-flash-tiered) share one lease. When every eligible account is leased for that model the request returns a structured 503 antigravity_pool_busy with a bounded Retry-After instead of piling onto a busy account. Opt-in behind ANTIGRAVITY_ACCOUNT_LEASE_ENABLED (runtime, default false). With the flag off no reservation is taken, credentials carry no routing descriptor, every release/hold is a no-op on an undefined lease id, and account selection and dispatch behave exactly as before. #10011's original test suite asserted family semantics for a lease that was exact-model scoped and failed deterministically on its own head; the model ids it used (gemini-3.5-flash / gemini-3-flash-agent) no longer exist in the catalog. The contradiction is resolved in favour of one coherent semantic — exact callable upstream model — and the tests assert it against the alias tables as they are on this branch. Co-authored-by: Ardem2025 <openclaw-auto@example.invalid>
…arrows it The discriminated-union form of reserveAntigravityLeaseForSelection's return type did not narrow under tsconfig.typecheck-api.json, so reading `reserved.lease` after the `reserved.busy` early return raised TS2339 in the API Route Typecheck gate. A single optional-property shape carries the same information and type-checks everywhere. Co-authored-by: Ardem2025 <openclaw-auto@example.invalid>
# Conflicts: # config/quality/file-size-baseline.json # src/sse/handlers/chat.ts
|
Validated in local merge-train /tmp/mt-train3b.log on 192.168.0.113 @ train tip 408e2128791696a18966681953136fc96aeb99b0 (32 PRs boarded): static gates green; full test:unit 40694 tests, 17 failing — every one reproduces on the pure release tip (base-red sweep list), zero new reds. Merged --admin per merge-gates §4/§7. |
Resolves the file-size-baseline.json conflict by merging the JSON (base entries + this PR's own rebaseline) and re-deriving the ceilings against the moved tip: auth.ts 3582 (this PR's +29 over a tip at 3552) and base.ts 1754 (tip drift from train 3b, absorbed here — see the baseline note).
|
Validated in local merge-train /tmp/mt-train3b.log on 192.168.0.113 @ 408e2128 (full test:unit 40694, 17 failing all inherited). Re-synced afterwards: the only delta is config/quality/file-size-baseline.json (JSON merge + ceilings re-derived: auth.ts 3582 own growth; base.ts 1754 absorbs the train-3b tip drift). Merged --admin per merge-gates §5 (one-green-run rule, baseline-only re-sync). |
|
Thanks for carrying this forward and for the credit, @diegosouzapw! Glad to see the Antigravity account stream leasing landed in release/v3.8.51. Appreciate the smooth integration. |
Re-sync onto the tip after diegosouzapw#13929 landed: API_REFERENCE.md union (this PR's rerank node notes + the base's provider-node discovery note); feature-flag totals 73 → 74 (this PR's RERANK_REMOTE_PROVIDER_NODES on top of the tip's 73) in FEATURE_FLAGS.md, feature-flags-settings.test.ts and server-owned-tool-loop-flag.test.ts. Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
…ifecycle (re-land of diegosouzapw#10011) (diegosouzapw#13929) * feat(sse): reserve the Antigravity account for the request's stream lifecycle Re-land of the account-lease half of diegosouzapw#10011 on the current release branch. Its exact-model-scoping half had already shipped in diegosouzapw#8050 and its quota half lost to the tip's aggregate-family design (selectAntigravityQuotaWindowNames / antigravityQuotaFamily.ts); none of that is reintroduced here. The lease is a concurrency reservation only and never reads or writes quota state. The Antigravity account selected for a request is reserved for the whole streaming lifecycle of that request, so a concurrent retry — or the credential handoff inside getProviderCredentialsWithQuotaPreflight — cannot re-pick an account already committed to an in-flight upstream stream. The reservation is scoped to (connection, callable upstream model) rather than the whole account, so one account can still serve two different models at once; catalog ids that resolve to the same upstream id (the gemini-3.7-flash tiers, all gemini-3.7-flash-tiered) share one lease. When every eligible account is leased for that model the request returns a structured 503 antigravity_pool_busy with a bounded Retry-After instead of piling onto a busy account. Opt-in behind ANTIGRAVITY_ACCOUNT_LEASE_ENABLED (runtime, default false). With the flag off no reservation is taken, credentials carry no routing descriptor, every release/hold is a no-op on an undefined lease id, and account selection and dispatch behave exactly as before. diegosouzapw#10011's original test suite asserted family semantics for a lease that was exact-model scoped and failed deterministically on its own head; the model ids it used (gemini-3.5-flash / gemini-3-flash-agent) no longer exist in the catalog. The contradiction is resolved in favour of one coherent semantic — exact callable upstream model — and the tests assert it against the alias tables as they are on this branch. Co-authored-by: Ardem2025 <openclaw-auto@example.invalid> * fix(sse): widen the Antigravity lease reservation result so auth.ts narrows it The discriminated-union form of reserveAntigravityLeaseForSelection's return type did not narrow under tsconfig.typecheck-api.json, so reading `reserved.lease` after the `reserved.busy` early return raised TS2339 in the API Route Typecheck gate. A single optional-property shape carries the same information and type-checks everywhere. Co-authored-by: Ardem2025 <openclaw-auto@example.invalid> --------- Co-authored-by: Ardem2025 <openclaw-auto@example.invalid>
Credit
The design and the original implementation are @Ardem2025's, from #10011. This is a
re-land of the piece of that PR that survives on the current release branch — the account
lease — rebuilt on
release/v3.8.51. #10011 itself was closed because its exact-model-scopinghalf had already shipped in #8050 and its quota half lost to the tip's different design.
Problem
Antigravity account selection has no notion of "this account is already busy with a live
stream". A request picks an account; a concurrent retry — or the credential handoff inside
getProviderCredentialsWithQuotaPreflight— can pick the same account again while the firstrequest is still streaming from it. The pool then piles onto one account instead of spreading
across the ones that are actually free.
Design
A process-local account lease, acquired during credential selection and released by the
terminal lifecycle of the response it was reserved for.
(connection, callable upstream model)— not the whole account. An Antigravityaccount can serve two different upstream models concurrently; fencing the whole account would
needlessly serialize unrelated traffic. Catalog ids that resolve to the same callable
upstream id share one lease (the three
gemini-3.7-flash-{high,medium,low}tiers are allgemini-3.7-flash-tiered).pre-dispatch failure, non-streaming response, dispatch throw — each frees it exactly once.
The stored timestamp is a bounded
Retry-Afterhint, never an expiry; nothing compares itagainst the clock to free a lease.
finallycannot free a newer lease that has sinceclaimed the same slot (covered by a test).
that model, the request returns a structured
503withcode: "antigravity_pool_busy"and abounded
Retry-After, instead of being reported as an upstream error.antigravity_pool_busyhad to be added to
SAFE_PUBLIC_ERROR_IDENTIFIERSinopen-sse/utils/error.ts— without itbuildErrorBodyprojects the code onto the status fallback and POOL_BUSY becomesindistinguishable from a generic upstream
service_unavailable.Orthogonal to quota, deliberately. This is a concurrency reservation. It never reads or
writes quota state.
selectAntigravityQuotaWindowNames,antigravityQuotaFamily.ts,quotaPreflight.tsandquotaCache.tsare untouched; #10011's competing exact-model quotalogic is not reintroduced.
Resolving #10011's known defect
#10011's
tests/unit/antigravity-routing-state.test.ts→ "release is fenced and explicitrelease restores availability" failed deterministically on its own head: it asserted family
semantics (a sibling model fenced by another model's lease) against an implementation that was
exact-model scoped. On top of that the model ids it used —
gemini-3.5-flashandgemini-3-flash-agent— no longer exist in this branch's catalog, and the hardcodedgemini-3.5-flash → gemini-3-flash-agentbridge in its canonicalizer is dead code here.Resolved in favour of one coherent semantic: the exact callable upstream model, and the
tests assert that against the alias tables as they actually are on this branch. Canonicalization
consults
ANTIGRAVITY_MODEL_ALIASESbefore the generic catalog alias, becausegemini-3.1-pro-highis callable only asgemini-pro-agentwhileresolveModelAliaswouldcollapse it onto plain
gemini-3.1-pro.Feature flag
ANTIGRAVITY_ACCOUNT_LEASE_ENABLED— categoryruntime,defaultValue: "false",requiresRestart: false. Read throughisAntigravityAccountLeaseEnabled()insrc/shared/utils/featureFlags.ts, which fails closed (an unreadable flag store keeps thepre-flag behaviour), and is short-circuited by a
provider === "antigravity"check so the flagstore is not read for any other provider.
With the flag off: no reservation is taken, credentials carry no
routingdescriptor, everyrelease/holdis a no-op on anundefinedlease id,leaseUnavailablecan never be producedso its branch is dead, and
excludedConnectionIds/ the preselected-credentials fast path startexactly as they do today. The only structural change on the flag-off path is three
.catch()handlers whose callback only releases an
undefinedlease and rethrows — all three wrappedcalls (
applyConnectionReasoningRule,checkAndRefreshToken,safeResolveProxy) areasync functions, so they always return a promise and the rejection value is unchanged.Files
Added
src/sse/services/antigravityRoutingState.ts— the lease registry (acquire / release / query,canonicalization, id-fenced release with an O(1) reverse index)
src/sse/services/antigravityLeaseLifecycle.ts— lifecycle glue: SSE hold-through-response,releasingRethrow, POOL_BUSY builder, per-request statesrc/sse/services/antigravityLeaseSelection.ts— selection glue: reserve, routing descriptor,release-from-credentials
tests/unit/antigravity-routing-state.test.ts,tests/unit/antigravity-lease-lifecycle.test.ts,tests/unit/antigravity-account-lease-flag.test.tschangelog.d/fixes/10011-antigravity-account-lease.mdModified
src/sse/services/auth.ts— acquire on selection, release on every path that abandons theselected connection (+20 lines)
src/sse/handlers/chat.ts— opt-in option, POOL_BUSY handling, hold-through-stream (+35 lines)src/shared/utils/featureFlags.ts,src/shared/constants/featureFlagDefinitions.ts— the flagopen-sse/utils/error.ts—antigravity_pool_busyadded toSAFE_PUBLIC_ERROR_IDENTIFIERSdocs/reference/FEATURE_FLAGS.md— catalog row + counts (72 → 73, Runtime 32 → 33)tests/unit/feature-flags-settings.test.ts,tests/unit/server-owned-tool-loop-flag.test.ts—flag-count assertions 72 → 73
config/quality/file-size-baseline.json— see "Honest reds" belowFail-on-tip proof
Every new test file was copied into a throwaway detached worktree at the pristine tip
(
origin/release/v3.8.51,a2c6f71) and run there.Being explicit about what kind of failure each one is, because it matters:
antigravity_pool_busy survives the public error-identifier projectionis a behaviouralfailure against a module that already exists on the tip:
buildErrorBodyinopen-sse/utils/error.tsreturnsservice_unavailablethere andantigravity_pool_busyhere. This is the one that proves a real behaviour change.ANTIGRAVITY_ACCOUNT_LEASE_ENABLED is a runtime boolean flag that defaults to OFFis alsobehavioural against an existing module — the key is absent from the tip's
FEATURE_FLAG_DEFINITIONS.(
TypeError: … is not a function, and two whole files failing at import). These areimport/missing-export failures, not behavioural ones — the modules under test are the
feature, so no behavioural assertion against them is possible on the tip. Stating that
plainly rather than counting them as behavioural proof.
On this branch all three files pass:
ℹ tests 18 · ℹ pass 18 · ℹ fail 0.Gate results
All run inside the worktree with
DATA_DIR=$(mktemp -d).npm run typecheck:corenode scripts/check/check-api-typecheck.mjsnpm run check:open-sse-typecheckopenSseTypecheckErrors=0, all within frozen baselinetsc(strictNullChecks) over the 6 touchedsrc/sse+src/sharedfilesnpm run lintThere are suppressions left that do not occur anymore.See "Honest reds".node scripts/check/check-file-size.mjsnode scripts/check/check-complexity.mjsnode scripts/check/check-cognitive-complexity.mjsnode scripts/check/check-changelog-integrity.mjsorigin/release/v3.8.51npm run check:cyclesnpm run check:docs-allfeature-flags-settings+feature-flags-doc-sync-static+server-owned-tool-loop-flag+auth-antigravity-account-retry-v2+antigravity-model-aliases+pii-opt-in-defaultrule12-error-sanitization-sweep+error-sanitizer-sk-key-qv45+route-error-sanitization-v382antigravity-quota-skipping+antigravity-byop-account-rotation+combo-antigravity-missing-project-reset-8486+antigravity-missing-project-chat+repro-antigravity-404-family-cooldown-hijacknpx prettier --checkis clean on every file this PR touches.Honest reds and one rebaseline
check-file-size— 3 inherited base-reds, unchanged by this PR:None of those files is touched here.
check-file-size— one rebaseline I did make, deliberately:src/sse/handlers/chat.ts2498 → 2533 (+35) and
src/sse/services/auth.ts3557 → 3577 (+20), with a_rebaseline_2026_09_16_10011_antigravity_account_leasejustification key next to the existing_rebaseline_*entries. The lease registry, its lifecycle glue and its selection glue were allextracted into three new modules precisely to keep the growth down to the call sites; what
remains in
chat.tsandauth.tsis the wiring itself, which cannot be lifted out of theselection loop and the dispatch path. Every added hunk there is inert unless the flag is on.
Flagging this prominently rather than burying it — it is a ratchet loosening and reviewers
should weigh it.
Base-green at the time of writing: no open
🔴 Release branch not green: release/v3.8.51issue,and no open
release-freeze.CI follow-up (first run on this PR)
Two red checks landed on the first CI run. Both investigated; one was mine and is fixed, the
other is an inherited base-red proven on the pristine tip.
1.
API Route Typecheck— one real regression, fixed inf288786src/sse/services/auth.tswas mine. The discriminated-union return type ofreserveAntigravityLeaseForSelectiondid not narrow undertsconfig.typecheck-api.json, soreserved.leaseafter thereserved.busyearly return raisedTS2339: Property 'lease' does not exist on type …. Fixed by flattening the return type to asingle optional-property shape (
{ busy?: …; lease?: … }), which carries the same informationand type-checks everywhere. Verified gone locally.
src/lib/providerModels/modelDiscovery.tsis not mine — and neither is the gate's redoverall. Run on a throwaway detached worktree at the pristine tip:
So
API Route Typecheckis red onrelease/v3.8.51itself. Not fixed here — a base-red fix isits own freeze-gated PR.
2. All four
Unit Tests fast-pathshards +Fast Quality Gates— inherited base-redsThe base branch fails these five jobs on its own, with no PR involved.
release/v3.8.51'smost recent
quality.ymlrun before this PR existed — run35127867599, commit6f26cf27b,2026-09-16 17:23Z — fails on exactly:
The same five jobs, and the last five
quality.ymlruns on that branch (back to 2026-09-15) areall
failure. The 15 distinct failing tests on the base include reasoning-effort clamping, thecontext_management400 retries, the authz pipeline, the raw-NUL-byte scan, the budget card andthe shared-set-size case — none of which this PR touches.
Test-by-test, and the three-test gap
This PR's shards fail 18 distinct tests; the base run failed 15. The three extra are:
They are not from this PR — the base moved on. The base run above is commit
6f26cf27b,while this PR is tested against the newer tip (
5acac80, #13328), which picked these up inbetween. Run on a pristine detached worktree at that current tip:
All three fail on the pristine tip, with no change of mine present. Every one of the 18 is
inherited; this PR adds none.
The one I dug into individually
Nothing to do with the lease (no Antigravity, no lease code on that path). Same file, run on
both trees:
agy-probe2,origin/release/v3.8.51)ℹ tests 67 · ℹ pass 66 · ℹ fail 1— same test, same assertionℹ tests 67 · ℹ pass 66 · ℹ fail 1— identicalByte-identical failure on both. Inherited — and it is one of the 15 already failing on the base.
3.
npm run lint— zero errors, exit 2 on unused suppressionsThe full local run reports no lint violations but exits 2 with:
Targeted
npx eslintover every file this PR touches is clean:src/sse/services/antigravityRoutingState.ts,antigravityLeaseLifecycle.ts,antigravityLeaseSelection.ts, the three new test files,src/shared/utils/featureFlags.tsandsrc/shared/constants/featureFlagDefinitions.tsproduce no output at all;src/sse/handlers/chat.ts,src/sse/services/auth.tsandopen-sse/utils/error.tsproduce onlypre-existing
no-unused-varserrors on imports this PR never touched (lines 17/24/42/96/148/152of chat.ts, etc.) — all frozen in
eslint-suppressions.json, which is whynpm run lintreportszero.
Resolved by CI: the
No new ESLint warningsjob passed (41m26s,actions/runs/35166430414/job/105028667750). The local exit 2 is stale-suppression driftinherited from the base, not anything this diff introduced.
Base-red note
⚠️ base-red inherited—API Route Typecheckand thesse-auth.test.ts#12080case are bothred on
release/v3.8.51itself, verified above. There is no open🔴 Release branch not green: release/v3.8.51issue to reference at the time of writing, so noissue number is cited; the nightly has not caught these yet.