Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions packages/coding-agent/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,10 @@
### Fixed

- Workflow runs awaiting human input now use the blue `?` indicator in the BACKGROUND panel, `/workflow connect` picker, and `/workflow status` listing, including prompts raised by hidden nested workflow children; the indicator returns to the run's current state when the prompt resolves.
- Fixed main-chat model fallback to classify provider failures consistently with workflows, advance rejected credentials and incompatible or unavailable models to the next candidate, and restore the user-selected model at the next turn without overriding an explicit `/model` choice. A failure that requesting the same model again cannot repair now takes that model out of the chain for the rest of the turn at every reasoning level, so a fallback entry differing only by its `:low`/`:high` suffix is skipped instead of retrying the same dead credential; transient rate-limit and transport failures keep those reasoning variants ([#2170](https://github.com/bastani-inc/atomic/issues/2170)).
- Fixed a context overflow that compaction cannot resolve to advance the configured `fallbackModels` chain instead of ending the turn, so a larger-context candidate can answer. Compaction still runs first, and a compactable overflow spends no fallback candidate ([#2170](https://github.com/bastani-inc/atomic/issues/2170)).
- Fixed a reasoning-level change during a model fallback stranding the session on the fallback model. Changing reasoning effort is not a model choice, so it no longer cancels the pending restore; the next turn returns to the user-selected primary and keeps the reasoning level that was chosen. Only an explicit `/model` selection or model cycle cancels the restore ([#2170](https://github.com/bastani-inc/atomic/issues/2170)).
- Fixed gRPC `ResourceExhausted` provider errors, seen from providers such as NVIDIA NIM, bypassing the same-model auto-retry budget. They are now retried like other transient provider failures, matching upstream pi-ai, and still advance a configured fallback chain when the retries are exhausted ([#2170](https://github.com/bastani-inc/atomic/issues/2170)).

## [0.9.12] - 2026-08-04

Expand Down
8 changes: 7 additions & 1 deletion packages/coding-agent/docs/settings.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,13 @@ Settings and trust JSON files may start with a UTF-8 BOM, as commonly written by

#### fallbackModels

`fallbackModels` gives ordinary main-chat turns an ordered model fallback chain. Atomic starts with the selected/default model. If that model exhausts the normal same-model auto-retry loop for a retryable provider/model failure — including rate limits and quota/usage-limit exhaustion such as a provider reporting `The usage limit has been reached` — Atomic switches to the next configured fallback model and continues the same turn. If `retry.enabled` is `false`, Atomic skips same-model retries and moves directly to the next fallback for retryable failures. Non-retryable task failures and cancellations do not trigger model fallback.
`fallbackModels` gives ordinary main-chat turns an ordered model fallback chain. Atomic starts with the selected/default model. If that model exhausts the normal same-model auto-retry loop for a retryable provider/model failure — including rate limits and quota/usage-limit exhaustion such as a provider reporting `The usage limit has been reached` — Atomic switches to the next configured fallback model and continues the same turn. If `retry.enabled` is `false`, Atomic skips same-model retries and moves directly to the next fallback for retryable failures. Non-retryable task failures and cancellations do not trigger model fallback. After a successful or exhausted fallback turn, Atomic restores the user-selected primary before the next user turn; an explicit `/model` choice during fallback cancels that restore.

A failure that another request to the same model cannot repair — a rejected credential, an unavailable model, a request that model cannot serve — takes that model out of the chain for the rest of the turn at **every** reasoning level, so a candidate that differs only by its `:low`/`:high` suffix is skipped rather than spent. Transient rate-limit and transport failures keep those reasoning variants, because retrying them can succeed.

Context overflow keeps its normal recovery order: compaction runs first, and a compactable overflow costs no fallback candidate. Only once compaction is disabled, fails, or reports the overflow unresolved does Atomic advance to the next configured candidate, which is how a larger-context model gets a chance at the turn.

Changing the reasoning level during a fallback turn is not a model choice, so it does not cancel the restore: the next turn still starts on the user-selected primary, carrying the reasoning level you picked. Only an explicit `/model` selection or model cycle cancels it.

The same list is also **borrowed by compaction**. When the compaction range planner cannot produce a usable plan on the current model — a rate limit, quota exhaustion, provider error, context overflow, or an empty plan — Atomic runs one planner request against the next configured candidate, using that candidate's own credentials. **A configured fallback model may therefore receive the compaction transcript.** Borrowing is planner-only: it never changes the session model, thinking level, or model history, it appends no model-change entry, and it emits no fallback status. See [Compaction](/compaction#planning-rungs-and-failure-behavior).

Expand Down
2 changes: 1 addition & 1 deletion packages/coding-agent/docs/subagents.md
Original file line number Diff line number Diff line change
Expand Up @@ -234,7 +234,7 @@ Dynamic fanout `collect.outputSchema` validates the collected result array after

## Fallback models

Agents can define ordered `fallbackModels` for retryable provider or model failures such as rate limits, quota/usage-limit exhaustion (for example a provider reporting `The usage limit has been reached`, or `usage_limit_reached`/`insufficient_quota` codes), auth problems, unavailable models, network timeouts, or 5xx errors. Atomic tries the requested primary model first, then configured fallbacks, and finally appends the current user-selected model as the last fallback candidate when available. Cancellations, safety refusals, and task/tool failures are never retried on another model.
Agents can define ordered `fallbackModels` for retryable provider or model failures such as rate limits, quota/usage-limit exhaustion (for example a provider reporting `The usage limit has been reached`, or `usage_limit_reached`/`insufficient_quota` codes), auth problems, unavailable models, network timeouts, or 5xx errors. Atomic tries the requested primary model first, then configured fallbacks, and finally appends the current user-selected model as the last fallback candidate when available. The main chat and workflow stages share one failure classifier, so auth, model-availability, request-incompatibility, and transport signals are handled consistently. Cancellations, safety refusals, and task/tool failures are never retried on another model.

A candidate that cannot serve the current request — for example an HTTP 400/413/422 bad/unprocessable/payload-too-large request, an unsupported tool or parameter, a context-length/context-window overflow, or a `too large` / `invalid_request` error — is treated as request/context incompatible and the chain advances to the next candidate rather than stopping. This means that if none of the configured candidates are applicable to the request, Atomic falls back to the currently selected user model instead of failing outright.

Expand Down
10 changes: 9 additions & 1 deletion packages/coding-agent/docs/workflows.md
Original file line number Diff line number Diff line change
Expand Up @@ -2295,10 +2295,18 @@ readonly fallbackModels?: readonly string[];
readonly fallbackThinkingLevels?: readonly string[];
```

`fallbackModels` tries the primary first, each fallback in order, and then the current Atomic-selected model when available. It advances for rate limits and quota or usage-limit exhaustion, including messages such as `The usage limit has been reached` and codes such as `usage_limit_reached` or `insufficient_quota`. Auth/provider outages, unavailable models, network timeouts, generic transport errors such as `Connection error.` or `fetch failed`, and 5xx responses also advance the chain.
`fallbackModels` tries the primary first, each fallback in order, and then the current Atomic-selected model when available. It advances for rate limits and quota or usage-limit exhaustion, including messages such as `The usage limit has been reached` and codes such as `usage_limit_reached` or `insufficient_quota`. Auth/provider outages, unavailable models, network timeouts, generic transport errors such as `Connection error.` or `fetch failed`, and 5xx responses also advance the chain. A thrown failure that another request to the same candidate can plausibly repair — a rate limit, provider outage, network timeout, or transport error — is retried on that candidate with exponential backoff from `settings.retry` before the chain advances; `retry.enabled: false` keeps immediate advancement. A failure the same candidate has already definitively rejected — a rejected credential, an unavailable model, or an incompatible request — skips the same-candidate retry and advances immediately, exactly as in main chat. A same-candidate retry resumes the existing turn when the stage transcript still ends in a message the agent can continue from, and otherwise re-sends the stage prompt; either way the failed provider error is dropped from the live transcript and the prompt is delivered exactly once.

Request/context incompatibility also advances it, including HTTP 400/413/422 bad, unprocessable, or payload-too-large requests; unsupported tools or parameters; context-length or context-window overflow; and `too large`, `invalid_request`, or `bad_request` errors. This lets the chain reach the current selected user model when no configured candidate can serve the request.

A context overflow that the stage session's compaction has already failed to resolve is terminal for its candidate: it skips the same-candidate retry, because re-sending an identical request cannot fit a context compaction could not shrink, and advances straight to the next candidate.

The chain also covers session creation. A stage session created eagerly — by `ctx.__ensureSession()`, an eager stage call, or a control attach — retries transient creation failures on its candidate under `settings.retry` and then walks to the next configured candidate, so a provider that cannot even open a session does not strand the stage. Creation failures that same-candidate retry cannot repair — auth, unavailable model, incompatible request — advance immediately. A creation failure that exhausts the whole chain is not cached: the next call starts a fresh attempt.

That walk runs behind a single creation gate. A concurrent `ctx.__ensureSession()` or a first `ctx.prompt()` joins the creation already in flight rather than starting a second walk, so the stage never has two live sessions competing for the same generation.

Controlled pauses are honored throughout. A pause that starts and finishes while a session is still being created keeps its replacement objective, which the next prompt sends exactly once; a pause during a same-candidate continuation is settled as a pause rather than a model failure, so resuming recovers the stage instead of spending a fallback candidate.

Workflow-code errors, tool failures, validation failures, refusals, content-filter or safety blocks, cancellations, and task failures do not advance the chain. A reattached finished stage starts on the model that last succeeded; if that model fails retryably, the full chain restarts from the primary.

### `thinkingLevel` (deprecated)
Expand Down
72 changes: 59 additions & 13 deletions packages/coding-agent/src/core/agent-session-auto-compaction.ts
Original file line number Diff line number Diff line change
Expand Up @@ -57,19 +57,22 @@ export async function _checkCompaction(
// response — before a fresh user prompt we must not resume the old turn.
const isLiveTurnCompletion = skipAbortedCheck;
const settings = this.settingsManager.getCompactionSettings();
if (!settings.enabled) return;

// Skip if message was aborted (user cancelled) - unless skipAbortedCheck is false
if (skipAbortedCheck && assistantMessage.stopReason === "aborted") return;

const contextWindow = this.model?.contextWindow ?? 0;

// Skip overflow check if the message came from a different model.
// Skip overflow handling if the message came from a different model.
// This handles the case where user switched from a smaller-context model (e.g. opus)
// to a larger-context model (e.g. codex) - the overflow error from the old model
// shouldn't trigger compaction for the new model.
const sameModel =
this.model && assistantMessage.provider === this.model.provider && assistantMessage.model === this.model.id;
if (!settings.enabled) {
// Compaction cannot recover this turn, so a configured fallback chain may
// advance to a larger-context candidate instead of dead-ending.
if (sameModel && isContextOverflow(assistantMessage, contextWindow)) this._contextOverflowUnresolved = true;
return;
}

// Skip if message was aborted (user cancelled) - unless skipAbortedCheck is false
if (skipAbortedCheck && assistantMessage.stopReason === "aborted") return;

// Skip compaction checks if this assistant message is older than the latest
// compaction boundary. This prevents a stale pre-compaction usage/error
Expand All @@ -91,6 +94,9 @@ export async function _checkCompaction(
}

if (this._overflowRecoveryAttempted) {
// One compact-and-retry has already been spent on this turn; a configured
// fallback chain may now advance to a larger-context candidate.
this._contextOverflowUnresolved = true;
this._emit({
type: "compaction_end",
reason: "overflow",
Expand Down Expand Up @@ -266,10 +272,25 @@ export function _schedulePostAutoCompactionContinuationProbe(
): void {
const token = this._postCompactionContinuationToken + 1;
this._postCompactionContinuationToken = token;
const fallbackScopeGeneration = this._fallbackOriginGeneration;
let pending: Promise<void>;
pending = new Promise<void>((resolve) => {
setTimeout(() => {
void (async () => {
const restoreIfOwned = async (): Promise<void> => {
if (
fallbackScopeGeneration === undefined ||
this._fallbackOriginGeneration !== fallbackScopeGeneration ||
typeof this._restoreFallbackModel !== "function"
)
return;
try {
await this._restoreFallbackModel();
} catch {
// A listener must not strand the continuation waiter. The model
// state was already restored before lifecycle notifications ran.
}
};
try {
if (willRetry) {
if (this._postCompactionContinuationToken !== token) return;
Expand All @@ -278,9 +299,24 @@ export function _schedulePostAutoCompactionContinuationProbe(
await this.agent.waitForIdle();
if (this._postCompactionContinuationToken !== token) return;
if (this.isCompacting || this.isStreaming) return;
if (!this.agent.hasQueuedMessages()) return;
if (!this.agent.hasQueuedMessages()) {
await restoreIfOwned();
return;
}
// A queued message starts the next user turn. Restore before
// Agent snapshots the next request's model.
await restoreIfOwned();
}

if (this._pendingPostCompactionContinuation !== pending) return;
// Clear this probe before entering the next run. Its promise is
// still awaited by _awaitPendingPostCompactionContinuation, but
// the nested agent_end must be free to schedule a new probe.
this._pendingPostCompactionContinuation = undefined;
await this._resumeAfterAutoCompaction();
if (willRetry && this._pendingPostCompactionContinuation === undefined) {
await restoreIfOwned();
}
} finally {
if (this._pendingPostCompactionContinuation === pending) {
this._pendingPostCompactionContinuation = undefined;
Expand Down Expand Up @@ -334,8 +370,18 @@ export function _resumeAfterLengthTruncation(this: AgentSession): void {
this._schedulePostAutoCompactionContinuationProbe("threshold", true);
}

function overflowUnresolved(reason: "overflow" | "threshold", aborted = false): boolean | undefined {
return reason === "overflow" && !aborted ? true : undefined;
/**
* Whether an overflow turn is now unrecoverable by compaction, recording it on
* the session so a configured fallback chain may advance to another candidate.
*/
function overflowUnresolved(
this: AgentSession,
reason: "overflow" | "threshold",
aborted = false,
): boolean | undefined {
if (reason !== "overflow" || aborted) return undefined;
this._contextOverflowUnresolved = true;
return true;
}

export async function _runAutoCompaction(
Expand Down Expand Up @@ -367,7 +413,7 @@ export async function _runAutoCompaction(
result: undefined,
aborted: false,
willRetry: false,
unresolvedOverflow: overflowUnresolved(reason),
unresolvedOverflow: overflowUnresolved.call(this, reason),
});
return;
}
Expand Down Expand Up @@ -398,7 +444,7 @@ export async function _runAutoCompaction(
result: undefined,
aborted: false,
willRetry: false,
unresolvedOverflow: overflowUnresolved(reason),
unresolvedOverflow: overflowUnresolved.call(this, reason),
});
return;
}
Expand All @@ -419,7 +465,7 @@ export async function _runAutoCompaction(
result: undefined,
aborted,
willRetry: false,
unresolvedOverflow: overflowUnresolved(reason, aborted),
unresolvedOverflow: overflowUnresolved.call(this, reason, aborted),
errorMessage: aborted
? undefined
: reason === "overflow"
Expand Down
Loading