[jthomson04]

Overview

When the Global Frontend's send to a pool frontend fails before the pool has the request, the request now goes to one other pool instead of returning 503. A request is never sent again once a pool may be running it.

On a four-region live run of the Global Frontend, the only two client errors were 503s where the send to one region's pool frontend failed and another pool could have served the request. Those runs logged only "error sending request"; #17524 now logs the cause, so the next one shows whether this retry covers it.

Stacked on #17525 (/metrics). This PR's base is that branch; it will be retargeted once #17525 merges.

Key Changes

About 15 core lines in proxy.rs; no new flags.

• Unsent check: the request body sits in a slot that the upload stream takes on its first poll. If the send then fails and the bytes are still there, the pool never got the body, so the request goes to the selector's choice among the other pools. The body is still freed once written to the pool.
• One retry: a request makes at most two sends. Under affinity, the session's pin moves to wherever the request lands.
• Metrics: each send counts at its own pool in dynamo_global_frontend_requests_total, so a send that failed over shows as no_response at the first pool.

Failure	Before	Now
Connection refused or timed out, failed name lookup	503	One other pool tries it
Connection lost after the body was handed over	503	503; not retried, since the pool may be running the request
Pool answers with any status, or fails mid-stream	Passed through	Unchanged

Per request when nothing fails: one small allocation for the body slot and a copy of the header map. Nothing runs per token.

Validation

Behaviour	Test
A closed pool fails over to the other pool, both pools' in-flight counts return to 0, and each send counts at its own pool in /metrics	a_request_goes_to_another_pool_when_its_pool_cannot_be_reached
A pool that read the whole body and then dropped the connection is not followed by a second send; the other pool sees no request	a_request_the_pool_may_have_is_not_sent_again
A single unreachable pool still gives 503 and counts no_response	unreachable_pools_are_unavailable

The forwarding tests ran 15 times in a row without a failure. No live run: the change acts only when a pool frontend can't be reached.

Issues and Ongoing Work

• Part of feat(global-frontend): forward requests to statically configured pools #17487. Stacked on feat(global-frontend): serve /metrics #17525.
• Not included:
  • Cooldown: with round-robin or least-in-flight and a pool whose frontend black-holes connections, every request pays the 5 s connect timeout again. Leaving a failed frontend out for a while would fix that; it can come as its own PR if that shows up.
  • Retry on pool statuses such as 503 or 429: it would keep each request's body in memory until its pool answers.

Pull Request Checklist

• I used Dynamo's /pr-checklist skill.

🤖 Generated with Claude Code

[jthomson04]

/ok to test c08c52b

[jthomson04]

/ok to test e31cffd

[devin-ai-integration]

✅ Dynamo PR CI passed — run 40931437125 (attempt 1) on 070debde93

Gate checks: ✅ backend-status-check · ✅ deploy-status-check · ✅ dynamo-status-check

_{Posted automatically by Devin for run 40931437125. Updated on every full-CI run of this PR.}

[jthomson04]

/ok to test 98fe133

[github-actions]

🌿 Fern Docs Preview: https://nvidia-preview-01a188f8-3952-702d-8bb7-c00cf82e21e6.docs.buildwithfern.com/dynamo/dev

[jthomson04]

/ok to test c09fdfa

[jthomson04]

/ok to test 298196c

[jthomson04]

/ok to test 070debd