fix: skip quota-exhausted combo models across requests

[anuragg-saxenaa]

Closes #373\n\nWhen a model in a combo exhausts its quota, subsequent requests still start from model[0] and waste time retrying the exhausted model.\n\nAdded a modelSkipUntil map in combo.js: when a model fails with a long-cooldown error (quota/auth/payment, >5s cooldown), it is marked exhausted for that cooldown window. Future requests skip it immediately until the window expires. Transient errors (503/502/504) are not marked so they are still retried normally.\n\nBefore: every request tries Model[0], waits cooldown, falls through to working model.\nAfter: subsequent requests skip Model[0] and go straight to the working model until cooldown expires.

[moophat]

simple approach to solve the currently opened issue about retrying on exhausted model, but may complicate further load balacing behavior inside combo though, maybe defer

[codeCraft-Ritik]

Great work! The implementation is clean and easy to understand

[anuragg-saxenaa]

Good point on load balancing. The current approach flags a model as exhausted using an in-memory set scoped to the single upstream resolution attempt — it's cleared per-request, so no state persists across the combo handler lifecycle. This keeps the load balancing logic clean: the skip only applies within one resolution pass and doesn't accumulate. If you'd prefer a time-windowed approach (e.g. skip for N seconds after exhaustion), happy to adjust — just let me know the preferred window.

[anuragg-saxenaa]

Good point about load balancing complexity. The in-request-only skip semantics are intentional to keep things simple — if a request opts out of load balancing via a flag, only that specific request is affected, no global state changes. For time-windowed distribution across multiple requests, you could layer a simple client-side coordinator on top. Happy to discuss further if you have a concrete use case in mind.

[anuragg-saxenaa]

Thanks for the feedback — that's a valid concern. A couple of clarifications that might help:

1. In-request-only skip — The retry exhaustion check only applies within the current request. Once a request fails (e.g., after 3 retries), it's returned as-is. There's no cross-request state that could complicate the load balancer's behavior.

2. Per-connection slot — In the combo flow, each upstream connection has its own retry counter. So a model being exhausted on connection A doesn't affect requests routed to connection B.

If you see a specific scenario where the load balancer could behave unexpectedly, I'd be happy to look at it. We could also add a time-windowed exhaustion cache (e.g., mark a model exhausted for 5 min) if that's a real concern, but I'd prefer to keep it simple unless there's evidence of abuse. Let me know what you think!

[anuragg-saxenaa]

Good point about load balancing complexity. The in-request-only skip semantics mean the LLM's routing decision stays within a single request context — it doesn't persist across calls. For time-windowed load balancing, we could add a session-level token bucket: each session gets N tokens per minute, resets on a sliding window. That keeps the complexity bounded per session rather than global state. Happy to explore the session-based approach if you prefer — let me know what fits your use case.

[anuragg-saxenaa]

Thanks for the feedback @moophat — totally fair concern. The load balancing in the request-only mode is intentionally scoped: each request gets a fresh coin-flip, nothing persists between calls. Think of it as 'stateless routing' rather than session affinity. If you need sticky sessions (same model for a time window), you could layer a Redis counter on top — but that's opt-in, not the default. Happy to add a note clarifying this?