diff --git a/docs/users/features/goals.md b/docs/users/features/goals.md index 0ac87d54d2f..437d075b525 100644 --- a/docs/users/features/goals.md +++ b/docs/users/features/goals.md @@ -24,6 +24,8 @@ Each turn the session takes on its own reports what the Goal has spent so far, h A long Goal periodically compresses the evidence it has recorded into checkpoint claims with a side model check, so later turns and the verifier still have it to cite. The check is bounded by [`model.goalCheckpointTimeoutSeconds`](../configuration/settings.md), 180 seconds by default. A reply wrapped in a markdown fence is read without the fence. If its claims overrun the aggregate byte budget, include a claim over the per-claim character limit, number more than one checkpoint may hold, cite an id that was not in the request, or change the proof kind of a source they cite, it makes one corrective model call that names what was wrong, and both calls share that ceiling. A reply that is not a JSON object holding a non-empty `claims` array gets no corrective call, and neither does one with any malformed claim, even beside usable ones: an extra key at either level, an unrecognised `proofKind`, an empty claim, or a `sourceRefs` list that is empty, holds a non-string or empty id, repeats an id, or holds more than 32 ids. A check that does not finish in time is abandoned as inconclusive; it counts toward the checkpoint stall limit only when the evidence window has overflowed, while a non-overflowing check preserves the streak and retries on a later turn. The calls are streamed, so the per-request transport timeout bounds only connect and first response, and the ceiling itself stops at the stream guards' 15-minute lifetime cap because past that the guard, not the setting, ends the check. That 15-minute limit on the setting is fixed, and raising the stream guard's own cap does not lift it. +A failing checkpoint shows up before it stops the Goal. While an active Goal's stall streak runs, the footer pill switches to `checkpoint N/3 stalled` on its own; once the Goal pauses or stops, the pill shows that status instead. The web shell's Goal status strip shows the count whatever the status. Whenever a terminal Goal status card is rendered, for example by `/goal` or by a pause, resume or verifier card, it shows how many consecutive checks have stalled out of the three the Goal allows, together with the last failure while one is recorded; the web shell's Goals dialog and headless `/goal` text output show the same line, while the web shell's transcript cards for Goal events show only the stop reason, and the model sees both fields when it reads the Goal. A check that fails while the window still has room is shown too, without spending a stall, but only while the Goal is active, or when that failure is itself what stopped the Goal, as with a checkpoint request too large to send. A checkpoint stop for any other reason clears the failure and keeps the streak, and a completed Goal shows no checkpoint line. The failure is kept as a single line with control characters removed. A Goal stopped by three stalled checkpoints names what the last one ran into. A check that could not fit the window within the checkpoint claim bounds, whether a full claim list that still left evidence behind or claims over the claim count or size budget, means the objective produces more evidence than one window holds, so narrow it. An answer that could not be folded into claims means the checkpoint model is not returning the structured output it is asked for, and narrowing the objective does not fix that. A check that never answered can mean an unreachable or rate-limited provider, a check that did not finish within `model.goalCheckpointTimeoutSeconds`, or an error in the check itself; the recorded failure says which. Resuming after any of the three starts a fresh evidence window. + ## Interrupting a Goal Cancelling a Goal turn pauses the Goal. Press Esc while the model is answering or while its tools are still running, and the turn stops, the Goal moves to `paused`, and the card and `/goal` both say why it stopped. Nothing continues until you run `/goal resume`. diff --git a/packages/cli/src/nonInteractiveCli.test.ts b/packages/cli/src/nonInteractiveCli.test.ts index 3232447aeb5..b8f9377fe47 100644 --- a/packages/cli/src/nonInteractiveCli.test.ts +++ b/packages/cli/src/nonInteractiveCli.test.ts @@ -9348,6 +9348,62 @@ describe('formatGoalState', () => { ); }); + it('writes no control sequence from a stop reason to stdout', () => { + // A pause reason can embed a raw provider error. + const output = formatGoalState( + goalSnapshot({ + status: 'paused', + lastReason: 'paused\r\u001b]52;c;ZXh0cmFjdGVk\u0007 by user', + }), + 'status', + ); + + expect(output).toContain('Reason: paused'); + expect(output).not.toContain('\r'); + expect(output).not.toContain('\u001b'); + expect(output).not.toContain('\u0007'); + }); + + it('names the checkpoint failure below the stop reason', () => { + // The stop reason names the kind of checkpoint failure; only this line + // says which one it was. + expect( + formatGoalState( + goalSnapshot({ + status: 'usage_limited', + lastReason: 'Checkpoints stalled.', + checkpointStalls: 3, + lastCheckpointFailure: + 'Error: Goal checkpoint verifier timed out after 30000ms', + }), + 'status', + ), + ).toBe( + 'Goal usage limited: ship the release notes\nReason: Checkpoints stalled.\nCheckpoint: 3/3 stalled · Error: Goal checkpoint verifier timed out after 30000ms', + ); + }); + + it('shows checkpoint health under the rule the interactive cards use', () => { + expect( + formatGoalState( + goalSnapshot({ lastCheckpointFailure: 'Error: provider failed' }), + 'status', + ), + ).toBe( + 'Goal active: ship the release notes\nCheckpoint: last check failed · Error: provider failed', + ); + expect( + formatGoalState( + goalSnapshot({ + status: 'complete', + checkpointStalls: 1, + lastCheckpointFailure: 'Error: provider failed', + }), + 'status', + ), + ).not.toContain('Checkpoint'); + }); + it('has no usage to report for a cleared Goal', () => { expect( formatGoalState({ v: 2, activity: 'idle', goal: null }, 'clear'), diff --git a/packages/cli/src/nonInteractiveCli.ts b/packages/cli/src/nonInteractiveCli.ts index c116a780ccd..0850c14aa05 100644 --- a/packages/cli/src/nonInteractiveCli.ts +++ b/packages/cli/src/nonInteractiveCli.ts @@ -22,6 +22,7 @@ import type { ServerLlmStreamEvent, } from '@qwen-code/qwen-code-core'; import { isSlashCommand } from './ui/utils/commandUtils.js'; +import { sanitizeTerminalText } from './ui/utils/textUtils.js'; import { isInlineModelOverrideAllowed } from './utils/acpModelUtils.js'; import type { LoadedSettings } from './config/settings.js'; import { @@ -80,6 +81,7 @@ import { getErrorType, getActiveInteractionSpan, buildGoalContinuationParts, + goalCheckpointHealthLine, } from '@qwen-code/qwen-code-core'; import type { Content, Part, PartListUnion } from '@google/genai'; import type { CLIUserMessage, PermissionMode } from './nonInteractive/types.js'; @@ -287,14 +289,21 @@ export function formatGoalState( : `${used} of ${goal.tokenBudget.toLocaleString('en-US')} tokens`, ); } - const withUsage = - usage.length > 0 ? `${summary}\nUsage: ${usage.join(' · ')}` : summary; + const lines = [summary]; + if (usage.length > 0) lines.push(`Usage: ${usage.join(' · ')}`); // Every non-active status now carries a reason, so gating on two of them // drops a paused Goal's reason from TEXT output while STREAM_JSON still // ships it -- and the user doc promises every pause states why. - return goal.status !== 'active' && goal.lastReason - ? `${withUsage}\nReason: ${goal.lastReason}` - : withUsage; + // Both lines are written to stdout as they are, so both are sanitized: a + // pause reason can embed a raw provider error. + if (goal.status !== 'active' && goal.lastReason) { + lines.push(`Reason: ${sanitizeTerminalText(goal.lastReason)}`); + } + // The checkpoint line the interactive cards show, in the same words: a + // checkpoint stop reason names the kind of failure, only this says which. + const checkpoint = goalCheckpointHealthLine(goal, sanitizeTerminalText); + if (checkpoint !== undefined) lines.push(`Checkpoint: ${checkpoint}`); + return lines.join('\n'); } async function claimUserGoalTurn( diff --git a/packages/cli/src/ui/commands/goal-checkpoint-stall-limit-wire-key.test.ts b/packages/cli/src/ui/commands/goal-checkpoint-stall-limit-wire-key.test.ts new file mode 100644 index 00000000000..7d751eedd54 --- /dev/null +++ b/packages/cli/src/ui/commands/goal-checkpoint-stall-limit-wire-key.test.ts @@ -0,0 +1,20 @@ +/** + * @license + * Copyright 2026 Qwen Team + * SPDX-License-Identifier: Apache-2.0 + */ + +import { describe, expect, it } from 'vitest'; +import { GOAL_CHECKPOINT_STALL_LIMIT } from '@qwen-code/qwen-code-core'; +import { GOAL_CHECKPOINT_STALL_LIMIT as SDK_GOAL_CHECKPOINT_STALL_LIMIT } from '@qwen-code/sdk/daemon'; + +// The Web Shell renders the checkpoint stall streak against the SDK's +// hand-duplicated copy of this limit, while the runtime stops a Goal at +// Core's. The SDK has no dependency path to Core, so pin the two copies here, +// where both packages are importable: a drift would show users the wrong N/3 +// rather than fail a build. +describe('goal checkpoint stall limit wire contract', () => { + it('is identical across core and the SDK', () => { + expect(SDK_GOAL_CHECKPOINT_STALL_LIMIT).toBe(GOAL_CHECKPOINT_STALL_LIMIT); + }); +}); diff --git a/packages/cli/src/ui/components/GoalPill.test.tsx b/packages/cli/src/ui/components/GoalPill.test.tsx index 5ac0c3264c6..e1b2aee6f7c 100644 --- a/packages/cli/src/ui/components/GoalPill.test.tsx +++ b/packages/cli/src/ui/components/GoalPill.test.tsx @@ -127,6 +127,40 @@ describe('GoalPill', () => { unmount(); }); + it('warns about stalled checkpoints before the stall breaker stops the Goal', () => { + vi.setSystemTime(NOW); + const { lastFrame, unmount } = renderPill({ + snapshot: snapshot('active', 'running', { + checkpointStalls: 2, + lastCheckpointFailure: 'Error: provider failed', + }), + }); + + expect(lastFrame()).toContain('! /goal checkpoint 2/3 stalled'); + // The footer has no room for the failure itself; the status card has it. + expect(lastFrame()).not.toContain('provider failed'); + unmount(); + }); + + it('keeps the plain labels when no checkpoint has stalled', () => { + vi.setSystemTime(NOW); + // A failure on a window with room spends no stall and is the card's to + // show; the footer stays quiet until the streak starts. + const quiet = renderPill({ + snapshot: snapshot('active', 'running', { + lastCheckpointFailure: 'Error: provider failed', + }), + }); + expect(quiet.lastFrame()).toContain('/goal active'); + quiet.unmount(); + + const checking = renderPill({ + snapshot: snapshot('active', 'verifying', { checkpointStalls: 1 }), + }); + expect(checking.lastFrame()).toContain('/goal checking'); + checking.unmount(); + }); + it('adds the current active span to persisted active time', () => { vi.setSystemTime(NOW); const { lastFrame, unmount } = renderPill({ diff --git a/packages/cli/src/ui/components/GoalPill.tsx b/packages/cli/src/ui/components/GoalPill.tsx index d2de46bd5d3..8864b1db799 100644 --- a/packages/cli/src/ui/components/GoalPill.tsx +++ b/packages/cli/src/ui/components/GoalPill.tsx @@ -9,7 +9,10 @@ import { useEffect, useState } from 'react'; import { Text } from 'ink'; import { elapsedActiveTime } from '@qwen-code/qwen-code-core/goals/goal-reducer.js'; import type { Config } from '@qwen-code/qwen-code-core/config/config.js'; -import type { GoalSnapshotV2 } from '@qwen-code/qwen-code-core/goals/goal-protocol.js'; +import { + GOAL_CHECKPOINT_STALL_LIMIT, + type GoalSnapshotV2, +} from '@qwen-code/qwen-code-core/goals/goal-protocol.js'; import type { GoalRuntime } from '@qwen-code/qwen-code-core/goals/goal-runtime.js'; import { useConfig } from '../contexts/ConfigContext.js'; import { theme } from '../semantic-colors.js'; @@ -98,13 +101,24 @@ function presentation(snapshot: GoalSnapshotV2): { if (!goal || goal.status === 'complete') return null; if (goal.status === 'active') { - return snapshot.activity === 'verifying' - ? { - icon: ICON.CIRCLE_EMPTY, - label: 'checking', - color: theme.text.secondary, - } - : { icon: ICON.BULLSEYE, label: 'active', color: theme.text.accent }; + if (snapshot.activity === 'verifying') { + return { + icon: ICON.CIRCLE_EMPTY, + label: 'checking', + color: theme.text.secondary, + }; + } + // A Goal paying a failed checkpoint every turn otherwise looks exactly + // like one that is working, until the stall breaker stops it. + const stalls = goal.checkpointStalls ?? 0; + if (stalls > 0) { + return { + icon: '!', + label: `checkpoint ${stalls}/${GOAL_CHECKPOINT_STALL_LIMIT} stalled`, + color: theme.status.warning, + }; + } + return { icon: ICON.BULLSEYE, label: 'active', color: theme.text.accent }; } switch (goal.status) { case 'paused': diff --git a/packages/cli/src/ui/components/messages/GoalStatusMessage.test.tsx b/packages/cli/src/ui/components/messages/GoalStatusMessage.test.tsx index 3da8e7c805d..aae88d737e2 100644 --- a/packages/cli/src/ui/components/messages/GoalStatusMessage.test.tsx +++ b/packages/cli/src/ui/components/messages/GoalStatusMessage.test.tsx @@ -187,6 +187,146 @@ describe('', () => { expect(lastFrame()).not.toContain('tokens'); }); + it('shows stalled checkpoints and the last failure on an active card', () => { + const { lastFrame } = render( + , + ); + + expect(lastFrame()).toContain( + 'Checkpoint: 2/3 stalled · Error: provider failed', + ); + }); + + it('shows a checkpoint failure that spent no stall', () => { + const { lastFrame } = render( + , + ); + + expect(lastFrame()).toContain( + 'Checkpoint: last check failed · Error: provider failed', + ); + }); + + it('keeps the failure on the card of a Goal the stall breaker stopped', () => { + // The stop reason names the kind of failure; only this line says which. + const { lastFrame } = render( + , + ); + + expect(lastFrame()).toContain('Reason: checkpoints stalled'); + expect(lastFrame()).toContain( + 'Checkpoint: 3/3 stalled · Error: provider failed', + ); + }); + + it('hides checkpoint health on a completed Goal that still carries it', () => { + // The terminal snapshot spreads the record and overrides only `status`, + // so a Goal that completed after a failed check journals both fields. + const { lastFrame } = render( + , + ); + + expect(lastFrame()).toContain('Goal complete'); + expect(lastFrame()).not.toContain('Checkpoint'); + }); + + it('hides a stall-free failure once the Goal stops for another reason', () => { + const paused = render( + , + ); + expect(paused.lastFrame()).not.toContain('Checkpoint'); + + // A running streak is still the truth about the window a resume re-enters. + const streak = render( + , + ); + expect(streak.lastFrame()).toContain('Checkpoint: 2/3 stalled'); + }); + + it('shows a bare stall streak without a trailing separator', () => { + // A stop for another reason clears the diagnostic and keeps the streak. + const { lastFrame } = render( + , + ); + + expect(lastFrame()).toContain('Checkpoint: 2/3 stalled'); + expect(lastFrame()).not.toContain('stalled ·'); + }); + + it('never writes control or bidi characters from the diagnostic to the terminal', () => { + const { lastFrame } = render( + , + ); + + const frame = lastFrame() ?? ''; + expect(frame).toContain('Checkpoint: 1/3 stalled'); + expect(frame).not.toContain('\r'); + expect(frame).not.toContain('\u202e'); + }); + + it('never writes control characters from a stop reason to the terminal', () => { + // A pause reason can embed a raw provider error. + const { lastFrame } = render( + , + ); + + const frame = lastFrame() ?? ''; + expect(frame).toContain('Reason: paused'); + expect(frame).not.toContain('\r'); + expect(frame).not.toContain('\u0007'); + }); + + it('says nothing about checkpoints on a healthy card', () => { + const { lastFrame } = render( + , + ); + + expect(lastFrame()).not.toContain('Checkpoint'); + }); + it('leaves the legacy card without spend it cannot know', () => { // The legacy props carry an iteration count and nothing else; there is no // record behind them to read a spend off. diff --git a/packages/cli/src/ui/components/messages/GoalStatusMessage.tsx b/packages/cli/src/ui/components/messages/GoalStatusMessage.tsx index fe0862f4ffc..ca4525d847c 100644 --- a/packages/cli/src/ui/components/messages/GoalStatusMessage.tsx +++ b/packages/cli/src/ui/components/messages/GoalStatusMessage.tsx @@ -6,8 +6,13 @@ import React from 'react'; import { Box, Text } from 'ink'; -import type { GoalSnapshotV2, GoalStateCause } from '@qwen-code/qwen-code-core'; +import { + goalCheckpointHealthLine, + type GoalSnapshotV2, + type GoalStateCause, +} from '@qwen-code/qwen-code-core'; import { theme } from '../../semantic-colors.js'; +import { sanitizeTerminalText } from '../../utils/textUtils.js'; import { ICON } from '../../constants.js'; import { formatDuration } from '../../utils/formatters.js'; import { formatTokenCount } from '../../statusLinePresets.js'; @@ -123,10 +128,20 @@ const GoalStateCard: React.FC = ({ ); } const subtitle = stats.length > 0 ? stats.join(' · ') : null; + // This renderer writes straight to the terminal, so both lines below are + // sanitized here: a pause reason can embed a raw provider error, and the + // checkpoint diagnostic, though cleaned where it is written, can come back + // from a journal record verbatim. const reason = goal.status !== 'active' || snapshot.activity === 'verifying' - ? goal.lastReason?.trim() + ? sanitizeTerminalText(goal.lastReason ?? '').trim() : undefined; + // Checkpoint health, shown before the stall breaker has to stop the Goal: + // a Goal paying a failed checkpoint every turn otherwise looks like one + // that is working. A Goal the breaker stopped keeps the line, since its stop + // reason names the kind of failure but not the failure itself; which + // records show it, and in what words, is decided once in core. + const checkpoint = goalCheckpointHealthLine(goal, sanitizeTerminalText); return ( @@ -153,6 +168,11 @@ const GoalStateCard: React.FC = ({ Reason: {reason} ) : null} + {checkpoint ? ( + + Checkpoint: {checkpoint} + + ) : null} ); diff --git a/packages/cli/src/ui/opentui/event-adapter.ts b/packages/cli/src/ui/opentui/event-adapter.ts index 7f1614ad90d..ac639a759f5 100644 --- a/packages/cli/src/ui/opentui/event-adapter.ts +++ b/packages/cli/src/ui/opentui/event-adapter.ts @@ -804,6 +804,8 @@ export type GoalSnapshotLike = { activeTimeMs?: number; tokensUsed?: number; tokenBudget?: number; + checkpointStalls?: number; + lastCheckpointFailure?: string; lastReason?: string; } | null; activity?: string; diff --git a/packages/cli/src/ui/opentui/live-session-model.test.ts b/packages/cli/src/ui/opentui/live-session-model.test.ts index 8c21b101b44..6208c551f15 100644 --- a/packages/cli/src/ui/opentui/live-session-model.test.ts +++ b/packages/cli/src/ui/opentui/live-session-model.test.ts @@ -762,6 +762,66 @@ describe('describeGoalCard (ink GoalStateCard)', () => { }); }); + it('shows checkpoint health, matching the ink card', () => { + expect( + describeGoalCard( + snap({ + objective: 'o', + status: 'active', + checkpointStalls: 2, + lastCheckpointFailure: 'Error: provider failed', + }), + ), + ).toMatchObject({ + checkpoint: 'Checkpoint: 2/3 stalled · Error: provider failed', + }); + expect( + describeGoalCard( + snap({ + objective: 'o', + status: 'active', + lastCheckpointFailure: 'Error: provider failed', + }), + ), + ).toMatchObject({ + checkpoint: 'Checkpoint: last check failed · Error: provider failed', + }); + // A stop for another reason clears the diagnostic and keeps the streak: + // the line is the count alone, with no trailing separator. + expect( + describeGoalCard( + snap({ objective: 'o', status: 'paused', checkpointStalls: 2 }), + ), + ).toMatchObject({ checkpoint: 'Checkpoint: 2/3 stalled' }); + const healthy = describeGoalCard( + snap({ objective: 'o', status: 'active' }), + ); + expect(healthy).toMatchObject({ state: 'card' }); + expect(healthy).not.toHaveProperty('checkpoint'); + + // Same visibility rule as the ink card: never on a completed Goal, and a + // stall-free failure only while the Goal is active. + expect( + describeGoalCard( + snap({ + objective: 'o', + status: 'complete', + checkpointStalls: 1, + lastCheckpointFailure: 'Error: provider failed', + }), + ), + ).not.toHaveProperty('checkpoint'); + expect( + describeGoalCard( + snap({ + objective: 'o', + status: 'paused', + lastCheckpointFailure: 'Error: provider failed', + }), + ), + ).not.toHaveProperty('checkpoint'); + }); + it('builds the subtitle from turns and active time', () => { expect( describeGoalCard( diff --git a/packages/cli/src/ui/opentui/live-session-model.ts b/packages/cli/src/ui/opentui/live-session-model.ts index 11eb1af5a09..f2633012c99 100644 --- a/packages/cli/src/ui/opentui/live-session-model.ts +++ b/packages/cli/src/ui/opentui/live-session-model.ts @@ -16,6 +16,7 @@ import type { HistoryItem } from '../model/streaming-model.js'; import type { GoalSnapshotLike, OpenTuiStreamEvent } from './event-adapter.js'; import type { TodoItem } from '../components/TodoDisplay.js'; import type { AnsiToken } from '@qwen-code/qwen-code-core'; +import { goalCheckpointHealthLine } from '@qwen-code/qwen-code-core/goals/goal-protocol.js'; import type { ArenaAgentCardData, CompressionProps } from '../types.js'; import { ICON } from '../constants.js'; import { formatDuration } from '../utils/formatters.js'; @@ -656,6 +657,8 @@ export type GoalCardView = subtitle: string | null; objective: string; reason?: string; + /** Checkpoint health, when goalCheckpointHealthVisible shows it. */ + checkpoint?: string; }; /** Computes the GoalStateCard view (icon/title/subtitle/objective/reason) @@ -730,6 +733,11 @@ export function describeGoalCard( (goal.status ?? 'active') !== 'active' || activity === 'verifying' ? goal.lastReason?.trim() : undefined; + // Checkpoint health, worded by core like the ink card's; transcript-view + // sanitizes the line when it renders it, so no cleaner is passed here. + const checkpointLine = goalCheckpointHealthLine(goal); + const checkpoint = + checkpointLine === undefined ? undefined : `Checkpoint: ${checkpointLine}`; return { state: 'card', icon: lifecycle.icon, @@ -738,6 +746,7 @@ export function describeGoalCard( subtitle: stats.length > 0 ? stats.join(' · ') : null, objective: goal.objective ?? '', reason, + ...(checkpoint ? { checkpoint } : {}), }; } diff --git a/packages/cli/src/ui/opentui/transcript-view.tsx b/packages/cli/src/ui/opentui/transcript-view.tsx index 3880ee0fda0..27f1f935dbd 100644 --- a/packages/cli/src/ui/opentui/transcript-view.tsx +++ b/packages/cli/src/ui/opentui/transcript-view.tsx @@ -580,6 +580,11 @@ function GoalCard({ {` ${sanitizeTerminalText(view.reason)}`} ) : null} + {view.checkpoint ? ( + + {` ${sanitizeTerminalText(view.checkpoint)}`} + + ) : null} ); } diff --git a/packages/core/src/goals/goal-checkpoint-verifier.test.ts b/packages/core/src/goals/goal-checkpoint-verifier.test.ts index 9d497873c42..051b94ca240 100644 --- a/packages/core/src/goals/goal-checkpoint-verifier.test.ts +++ b/packages/core/src/goals/goal-checkpoint-verifier.test.ts @@ -1068,6 +1068,34 @@ describe('createGoalCheckpointVerifier', () => { expect(caller.signal.aborted).toBe(false); }); + it('reports the timeout when the provider answers the abort with its own error', async () => { + // A real provider SDK rejects an aborted request with its own error and + // drops the reason the signal carried, so the Goal record used to say + // "Request was aborted." without saying the check had timed out. + const generateText = vi.fn().mockImplementation( + (request: { abortSignal?: AbortSignal }) => + new Promise((_resolve, reject) => { + request.abortSignal?.addEventListener('abort', () => { + reject(new Error('Request was aborted.')); + }); + }), + ); + const { config } = finishConfig(generateText); + + await expect( + createGoalCheckpointVerifier(config, { timeoutMs: 1 })(input()), + ).rejects.toThrow('Goal checkpoint verifier timed out after 1ms'); + + // The caller's own abort is an interrupt, not a timeout: it keeps the + // error the provider raised. + const caller = new AbortController(); + const interrupted = createGoalCheckpointVerifier(config, { + timeoutMs: 60_000, + })(input(), caller.signal); + caller.abort(new Error('user interrupt')); + await expect(interrupted).rejects.toThrow('Request was aborted.'); + }); + it.each([ ['the configured ceiling', 45_000, { timeoutMs: 45_000 }], ['the built-in default', GOAL_CHECKPOINT_VERIFIER_DEFAULT_TIMEOUT_MS, {}], diff --git a/packages/core/src/goals/goal-checkpoint-verifier.ts b/packages/core/src/goals/goal-checkpoint-verifier.ts index 11151a250fc..45e3f222d93 100644 --- a/packages/core/src/goals/goal-checkpoint-verifier.ts +++ b/packages/core/src/goals/goal-checkpoint-verifier.ts @@ -106,8 +106,9 @@ export interface CreateGoalCheckpointVerifierOptions { * and one the emitted schema cannot prevent -- JSON Schema has no aggregate * byte bound, and the per-claim and per-item bounds it does carry are * stripped before the request goes out. It stays an - * `InvalidGoalCheckpointError`, so a retry that overruns again reaches the - * runtime as the unusable result it is. + * `InvalidGoalCheckpointError`, but `describeCheckpointFailure` reads it as a + * capacity failure: a retry that overruns again stops a stalled Goal with the + * narrow-the-objective advice, not the unusable-output one. */ export class GoalCheckpointClaimBudgetError extends InvalidGoalCheckpointError { constructor(readonly byteLength: number) { @@ -216,7 +217,7 @@ function proofKindErrorMessage( export class GoalCheckpointVerifierInputTooLargeError extends Error { constructor(readonly byteLength: number) { super( - `Goal checkpoint verifier request exceeds the ${GOAL_CHECKPOINT_VERIFIER_REQUEST_BYTE_LIMIT}-byte limit`, + `Goal checkpoint verifier request of ${byteLength} bytes exceeds the ${GOAL_CHECKPOINT_VERIFIER_REQUEST_BYTE_LIMIT}-byte limit`, ); this.name = 'GoalCheckpointVerifierInputTooLargeError'; } @@ -678,6 +679,16 @@ export function createGoalCheckpointVerifier( retryCause = error; } } + } catch (error) { + // A provider SDK rejects its own aborted request with its own error + // ("Request was aborted.") and drops the reason the signal carried, so + // a check that ran past this ceiling would never say it timed out. The + // caller's abort is left alone: the runtime treats that as an + // interrupt, not a failed check. + if (timeoutController.signal.aborted && !attemptSignal?.aborted) { + throw timeoutController.signal.reason; + } + throw error; } finally { clearTimeout(timer); } diff --git a/packages/core/src/goals/goal-checkpoint.ts b/packages/core/src/goals/goal-checkpoint.ts index 792fe9dd3c8..83e0bf2f71c 100644 --- a/packages/core/src/goals/goal-checkpoint.ts +++ b/packages/core/src/goals/goal-checkpoint.ts @@ -71,9 +71,16 @@ export function isGoalCheckpointStalled( } /** - * An unusable checkpoint verifier result. Nothing in production branches on - * the class -- the stall breaker counts by window state, not error class -- - * so it is a diagnostic carrier: its name and message are what an + * An unusable checkpoint verifier result. + * + * Production branches on this class. The stall breaker still counts by window + * state, but `describeCheckpointFailure` (goal-runtime.ts) picks the stall + * stop's advice from it: this class reads as "the verifier answered, but not + * with usable claims", its claim-count, claim-budget and claim-length + * subclasses read as capacity failures, and anything outside the hierarchy reads as "no answer + * arrived". Keep provider-side failures -- transport errors, rate limits, + * rejected requests -- out of this hierarchy, or a provider outage would be + * reported as malformed output. Its name and message are also what an * investigation into a stalled Goal gets to see. */ export class InvalidGoalCheckpointError extends Error { diff --git a/packages/core/src/goals/goal-protocol.test.ts b/packages/core/src/goals/goal-protocol.test.ts index 9b254d4b32d..c85ebb2ab40 100644 --- a/packages/core/src/goals/goal-protocol.test.ts +++ b/packages/core/src/goals/goal-protocol.test.ts @@ -6,6 +6,15 @@ import { describe, expect, it } from 'vitest'; import { + capGoalCheckpointFailure, + GOAL_CHECKPOINT_FAILURE_MAX_CHARACTERS, + GOAL_CHECKPOINT_STALLED_REASON, + GOAL_CHECKPOINT_UNREACHABLE_REASON, + GOAL_CHECKPOINT_UNUSABLE_REASON, + goalCheckpointHealthLine, + goalCheckpointHealthVisible, + goalCheckpointStalledReason, + goalLimitKindForReason, GOAL_PAUSE_REASON_COMMAND, GOAL_PAUSE_REASON_HEADLESS_RUN_ENDED, GOAL_PAUSE_REASON_MAX_CHARACTERS, @@ -118,6 +127,211 @@ describe('goal pause reasons', () => { }); }); +describe('goal checkpoint stall reasons', () => { + it('advises by what the check that spent the last stall ran into', () => { + expect(goalCheckpointStalledReason('capacity')).toBe( + GOAL_CHECKPOINT_STALLED_REASON, + ); + expect(goalCheckpointStalledReason('unusable')).toBe( + GOAL_CHECKPOINT_UNUSABLE_REASON, + ); + expect(goalCheckpointStalledReason('unreachable')).toBe( + GOAL_CHECKPOINT_UNREACHABLE_REASON, + ); + // Capacity is fixed by a narrower objective; malformed output is not, + // and telling that user to rewrite their Goal is the bug. + expect(GOAL_CHECKPOINT_STALLED_REASON).toContain('narrower objective'); + expect(GOAL_CHECKPOINT_UNUSABLE_REASON).toContain( + 'Narrowing the objective does not fix this', + ); + // A check that never answered may have run past its own ceiling on a + // window too large to verify in time, so its advice keeps every remedy + // that can apply instead of blaming the provider. + expect(GOAL_CHECKPOINT_UNREACHABLE_REASON).toContain( + 'model.goalCheckpointTimeoutSeconds', + ); + expect(GOAL_CHECKPOINT_UNREACHABLE_REASON).toContain( + 'narrow the objective', + ); + expect(GOAL_CHECKPOINT_UNREACHABLE_REASON).not.toContain( + 'does not fix this', + ); + }); + + it('keeps resumability on limitKind rather than on the stop prose', () => { + // The stall stop writes `limitKind: 'evidence_catalog'` beside every one + // of these reasons; none may start denoting a kind of its own, or the + // prose and the field could disagree about how a resume behaves. + for (const reason of [ + GOAL_CHECKPOINT_STALLED_REASON, + GOAL_CHECKPOINT_UNUSABLE_REASON, + GOAL_CHECKPOINT_UNREACHABLE_REASON, + ]) { + expect(goalLimitKindForReason(reason)).toBeUndefined(); + } + }); + + it('caps a failure diagnostic on a code point boundary', () => { + expect(capGoalCheckpointFailure(' Error: provider failed ')).toBe( + 'Error: provider failed', + ); + const exact = 'x'.repeat(GOAL_CHECKPOINT_FAILURE_MAX_CHARACTERS); + expect(capGoalCheckpointFailure(exact)).toBe(exact); + + // Astral characters are two UTF-16 units: a slice by `.length` would + // split one and leave a lone surrogate in the journaled record. + const capped = capGoalCheckpointFailure( + '😀'.repeat(GOAL_CHECKPOINT_FAILURE_MAX_CHARACTERS + 10), + ); + expect(codePoints(capped)).toBe(GOAL_CHECKPOINT_FAILURE_MAX_CHARACTERS); + expect(capped.endsWith('…')).toBe(true); + expect([...capped].slice(0, -1).every((char) => char === '😀')).toBe(true); + }); + + it('keeps a failure diagnostic on one display-safe line', () => { + expect(capGoalCheckpointFailure('Error: a\nb c')).toBe('Error: a b c'); + expect(capGoalCheckpointFailure('Error: a\r\n b\tc')).toBe( + 'Error: a b c', + ); + // Escape sequences go whole and bidi overrides go entirely, so no + // surface can be repainted or reordered by a provider's error text. + expect( + capGoalCheckpointFailure( + 'Error: \u001b[31mred\u001b[0m rate\u202e limit\u0007', + ), + ).toBe('Error: red rate limit'); + + // Collapsing runs before the cap, so a response body's indentation + // cannot spend the bound. + const capped = capGoalCheckpointFailure( + `Error:${'\n word'.repeat(200)}`, + ); + expect(capped).not.toMatch(/\s{2,}|\n/); + expect(codePoints(capped)).toBe(GOAL_CHECKPOINT_FAILURE_MAX_CHARACTERS); + }); +}); + +describe('goal checkpoint health visibility', () => { + const failure = 'Error: provider failed'; + + it.each([ + [ + 'an active Goal whose failure spent no stall', + { status: 'active', lastCheckpointFailure: failure }, + true, + ], + [ + 'an active Goal mid-streak', + { status: 'active', checkpointStalls: 2, lastCheckpointFailure: failure }, + true, + ], + [ + 'a Goal the stall breaker stopped', + { + status: 'usage_limited', + checkpointStalls: 3, + lastCheckpointFailure: failure, + }, + true, + ], + [ + 'a paused Goal that keeps its streak', + { status: 'paused', checkpointStalls: 1, lastCheckpointFailure: failure }, + true, + ], + [ + 'a Goal paused for another reason after a stall-free failure', + { status: 'paused', lastCheckpointFailure: failure }, + false, + ], + [ + 'a Goal stopped by another bound after a stall-free failure', + { status: 'usage_limited', lastCheckpointFailure: failure }, + false, + ], + [ + 'a Goal stopped because its checkpoint request was too large', + { + status: 'usage_limited', + limitKind: 'checkpoint_request', + lastCheckpointFailure: failure, + }, + true, + ], + [ + 'a Goal stopped by its checkpoint request with a blank diagnostic', + { + status: 'usage_limited', + limitKind: 'checkpoint_request', + lastCheckpointFailure: ' ', + }, + false, + ], + [ + 'a completed Goal that still carries both fields', + { + status: 'complete', + checkpointStalls: 1, + lastCheckpointFailure: failure, + }, + false, + ], + ['a healthy active Goal', { status: 'active' }, false], + ] as const)('%s', (_label, goal, expected) => { + expect(goalCheckpointHealthVisible(goal)).toBe(expected); + }); +}); + +describe('goal checkpoint health line', () => { + const failure = 'Error: provider failed'; + + it('words the streak, or the stall-free failure, then the diagnostic', () => { + expect( + goalCheckpointHealthLine({ + status: 'active', + checkpointStalls: 2, + lastCheckpointFailure: failure, + }), + ).toBe('2/3 stalled · Error: provider failed'); + expect( + goalCheckpointHealthLine({ + status: 'active', + lastCheckpointFailure: ` ${failure} `, + }), + ).toBe('last check failed · Error: provider failed'); + // A bare streak carries no trailing separator. + expect( + goalCheckpointHealthLine({ status: 'paused', checkpointStalls: 2 }), + ).toBe('2/3 stalled'); + }); + + it('shows nothing the visibility rule hides', () => { + expect( + goalCheckpointHealthLine({ + status: 'complete', + checkpointStalls: 1, + lastCheckpointFailure: failure, + }), + ).toBeUndefined(); + expect( + goalCheckpointHealthLine({ + status: 'paused', + lastCheckpointFailure: failure, + }), + ).toBeUndefined(); + }); + + it('cleans the diagnostic before trimming and joining it', () => { + // A diagnostic the cleaner empties leaves no dangling separator. + expect( + goalCheckpointHealthLine( + { status: 'active', checkpointStalls: 1, lastCheckpointFailure: 'x' }, + () => ' ', + ), + ).toBe('1/3 stalled'); + }); +}); + describe('Goal cadence budget reasons', () => { it('formats singular and plural turn budgets', () => { expect(goalTurnBudgetReason(1)).toContain('(1 turn)'); diff --git a/packages/core/src/goals/goal-protocol.ts b/packages/core/src/goals/goal-protocol.ts index c318e74f6eb..7dfb86dcf2e 100644 --- a/packages/core/src/goals/goal-protocol.ts +++ b/packages/core/src/goals/goal-protocol.ts @@ -4,6 +4,11 @@ * SPDX-License-Identifier: Apache-2.0 */ +import { + stripDisplayControlChars, + stripTerminalControlSequences, +} from '../utils/terminalSafe.js'; + export const GOAL_STATE_VERSION = 2 as const; export const GOAL_PROPOSAL_REASON_MAX_CHARACTERS = 8_000; export const GOAL_PROPOSAL_REASON_MAX_BYTES = 16_000; @@ -29,8 +34,141 @@ export const GOAL_CHECKPOINT_STALL_LIMIT = 3; * the loop. */ export const GOAL_NO_PROGRESS_TURN_LIMIT = 3; +/** + * The stall stop for a Goal whose last stalled check could not fit the window + * inside the checkpoint's claim bounds: a full claim list that still left + * evidence behind, or well-formed claims over the claim count, the byte budget + * or the per-claim length. Compaction itself cannot keep up -- the objective produces more + * evidence than one window holds -- so narrowing it is the remedy. + */ export const GOAL_CHECKPOINT_STALLED_REASON = - 'The current Goal revision ran three consecutive evidence checkpoints without relief: the evidence window overflowed every time, and each check either came back with a full claim list, came back with a result that could not be folded into claims, or did not come back at all, so every turn paid a checkpoint call and lost uncatalogued evidence. Automatic retries cannot recover. Edit or replace the Goal with a narrower objective before resuming it.'; + 'The current Goal revision ran three consecutive evidence checkpoints without relief: the evidence window overflowed every time, and the last check could not fit it within the checkpoint claim bounds, so every turn paid a checkpoint call and lost uncatalogued evidence. Automatic retries cannot recover. Edit or replace the Goal with a narrower objective before resuming it.'; + +/** + * The stall stop for a Goal whose last stalled check answered with output that + * is not usable claims at all. The objective may not be too wide -- the + * checkpoint model is returning output the runtime cannot accept -- so the + * capacity advice to narrow it would send the user to rewrite a Goal that was + * never the problem. + */ +export const GOAL_CHECKPOINT_UNUSABLE_REASON = + 'The current Goal revision ran three consecutive evidence checkpoints without relief: the evidence window overflowed every time, and the last check answered with output that could not be folded into claims. Narrowing the objective does not fix this. Check that the checkpoint model returns the structured JSON it is asked for, or switch models, then resume the Goal; resuming starts a fresh evidence window.'; + +/** + * The stall stop for a Goal whose last stalled check produced no answer to + * judge. The runtime cannot tell why from here: the provider may be + * unreachable or rate-limited, the check may have run past its own ceiling on + * a window too large to verify in time, or the check itself may have failed. + * The recorded failure says which, so this names every remedy that can apply + * rather than blaming the provider. + */ +export const GOAL_CHECKPOINT_UNREACHABLE_REASON = + 'The current Goal revision ran three consecutive evidence checkpoints without relief: the evidence window overflowed every time, and the last check failed before the checkpoint verifier returned an answer. The recorded checkpoint failure says why: an unreachable or rate-limited provider, a check that did not finish within model.goalCheckpointTimeoutSeconds, or an error in the check itself. Fix the provider, raise that timeout, or narrow the objective so the window checkpoints in time, then resume the Goal; resuming starts a fresh evidence window.'; + +/** + * What the last stalled checkpoint check ran into, which decides the advice + * the stop carries. `capacity`: the check could not fit the window within the + * claim bounds (a full claim list that left evidence behind, or claims over + * the count, byte or length bound). `unusable`: it answered with output that is not + * usable claims. `unreachable`: no answer arrived to judge. + */ +export type GoalCheckpointFailureShape = + | 'capacity' + | 'unusable' + | 'unreachable'; + +/** The `lastReason` a checkpoint stall stop records for its last failure. */ +export function goalCheckpointStalledReason( + shape: GoalCheckpointFailureShape, +): string { + switch (shape) { + case 'capacity': + return GOAL_CHECKPOINT_STALLED_REASON; + case 'unusable': + return GOAL_CHECKPOINT_UNUSABLE_REASON; + case 'unreachable': + return GOAL_CHECKPOINT_UNREACHABLE_REASON; + default: { + const exhaustive: never = shape; + return exhaustive; + } + } +} + +/** + * Longest `lastCheckpointFailure` a record keeps. A provider error can carry a + * whole response body, and the record is journaled on every checkpoint check. + */ +export const GOAL_CHECKPOINT_FAILURE_MAX_CHARACTERS = 500; + +/** + * Makes a checkpoint failure safe to keep as a one-line diagnostic: terminal + * control sequences and bidi overrides removed, every run of whitespace (line + * breaks included) collapsed to one space, and the result capped by code + * point. Collapsing runs before the cap, so the bound spends its code points + * on the message rather than on a response body's indentation. The value is + * journaled, handed to the model, and rendered on every Goal surface, so it is + * cleaned once where it is written rather than trusted to each reader. + */ +export function capGoalCheckpointFailure(text: string): string { + const oneLine = stripDisplayControlChars(stripTerminalControlSequences(text)) + .replace(/\s+/g, ' ') + .trim(); + const codePoints = [...oneLine]; + return codePoints.length <= GOAL_CHECKPOINT_FAILURE_MAX_CHARACTERS + ? oneLine + : `${codePoints.slice(0, GOAL_CHECKPOINT_FAILURE_MAX_CHARACTERS - 1).join('')}…`; +} + +/** + * Whether a Goal surface should show checkpoint health, decided once so every + * card and summary agrees. A completed Goal never does: its checkpoints no + * longer matter, and the terminal snapshot keeps whatever the record carried. + * A running stall streak always does, whatever the status, because it is + * still the truth about the evidence window a resume re-enters. A Goal stopped + * because its checkpoint request was too large to send shows the failure that + * stopped it, since that failure is the stop. Any other failure that spent no + * stall shows only while the Goal is active: once the Goal stops or pauses for + * another reason, that diagnostic explains nothing about the stop and would + * read as though it did. + */ +export function goalCheckpointHealthVisible(goal: { + status?: string; + checkpointStalls?: number; + lastCheckpointFailure?: string; + limitKind?: string; +}): boolean { + if (goal.status === 'complete') return false; + if ((goal.checkpointStalls ?? 0) > 0) return true; + const failed = Boolean(goal.lastCheckpointFailure?.trim()); + if (goal.limitKind === 'checkpoint_request') return failed; + return (goal.status ?? 'active') === 'active' && failed; +} + +/** + * The checkpoint health a text surface prints -- the stall count, or the + * stall-free label, then the diagnostic -- or undefined when + * `goalCheckpointHealthVisible` hides it. Worded once so every terminal + * surface says the same thing. `clean` runs on the diagnostic before it is + * trimmed and joined, for a caller that writes straight to a terminal; a + * caller that sanitizes the rendered line itself passes nothing, so the text + * is not escaped twice. + */ +export function goalCheckpointHealthLine( + goal: Parameters[0], + clean: (text: string) => string = (text) => text, +): string | undefined { + if (!goalCheckpointHealthVisible(goal)) return undefined; + const stalls = goal.checkpointStalls ?? 0; + return [ + stalls > 0 + ? `${stalls}/${GOAL_CHECKPOINT_STALL_LIMIT} stalled` + : 'last check failed', + clean(goal.lastCheckpointFailure ?? '').trim(), + ] + .filter(Boolean) + .join(' · '); +} /** * Default autonomous spend window armed on a newly created Goal, in model @@ -298,6 +436,22 @@ export interface GoalRecord { * an evidence-limited Goal. */ checkpointStalls?: number; + /** + * What the most recent checkpoint check that gave no relief ran into, as a + * one-line diagnostic: `ErrorName: message` for a check that failed, or the + * runtime's own phrase for one that answered with a full claim list while + * the window overflowed -- so it does not always mean the check threw. + * Capped at GOAL_CHECKPOINT_FAILURE_MAX_CHARACTERS. Set by every such check, + * whether or not it spends a stall, and kept on the record the stall breaker + * stops, so the stop can be diagnosed from the record alone. Cleared by a + * check that finds room or writes a checkpoint without stalling, by every + * control action that clears `checkpointStalls`, and by a checkpoint stop + * whose cause is not itself a check (missing recovery dependencies, an + * exhausted catalog, an unreadable transcript), so it can be absent while + * `checkpointStalls` is still non-zero. A check that proves nothing either + * way (a turn that recorded no evidence) leaves it as it was. + */ + lastCheckpointFailure?: string; /** * Consecutive autonomous turns that recorded neither a tool result nor a * terminal proposal. A model that only restates status never reaches the diff --git a/packages/core/src/goals/goal-reducer.test.ts b/packages/core/src/goals/goal-reducer.test.ts index e4b2c877e35..1f1eb5643bf 100644 --- a/packages/core/src/goals/goal-reducer.test.ts +++ b/packages/core/src/goals/goal-reducer.test.ts @@ -868,6 +868,96 @@ describe('goal reducer', () => { expect(edited?.checkpointStalls).toBeUndefined(); }); + it('restores a persisted checkpoint failure and rejects a malformed one', () => { + const stalled = snapshot( + goalRecord({ + checkpointStalls: 1, + lastCheckpointFailure: 'Error: provider failed', + }), + ); + expect(parseGoalSnapshotV2(stalled)).toEqual(stalled); + // A check that failed while the window had room reports its failure + // without spending a stall, so the diagnostic stands on its own. + const unstalled = snapshot( + goalRecord({ lastCheckpointFailure: 'Error: provider failed' }), + ); + expect(parseGoalSnapshotV2(unstalled)).toEqual(unstalled); + expect( + parseGoalSnapshotV2(snapshot(goalRecord({ lastCheckpointFailure: '' }))), + ).toBeUndefined(); + expect( + parseGoalSnapshotV2( + snapshot({ + ...goalRecord(), + lastCheckpointFailure: 42, + } as unknown as GoalRecord), + ), + ).toBeUndefined(); + }); + + it('clears the checkpoint failure wherever it clears the stall streak', () => { + const failure = 'Error: provider failed'; + const resume = { + action: 'resume' as const, + expectedGoalId: 'g-1', + expectedRevision: 1, + }; + const edited = reduceGoalControl( + goalRecord({ checkpointStalls: 2, lastCheckpointFailure: failure }), + { + request: { + action: 'edit', + objective: 'deliver the rest', + expectedGoalId: 'g-1', + expectedRevision: 1, + }, + now: 200, + nextGoalId: 'g-next', + cursor: { recordId: 'r-200' }, + }, + ); + expect(edited?.lastCheckpointFailure).toBeUndefined(); + + const restarted = reduceGoalControl( + goalRecord({ + status: 'usage_limited', + limitKind: 'evidence_catalog', + checkpointStalls: 3, + lastCheckpointFailure: failure, + }), + { + request: resume, + now: 200, + nextGoalId: 'unused', + cursor: { recordId: 'r-200' }, + }, + ); + expect(restarted).toMatchObject({ status: 'active' }); + expect(restarted?.checkpointStalls).toBeUndefined(); + expect(restarted?.lastCheckpointFailure).toBeUndefined(); + + // A paused Goal resumes into the window it left: like the streak, the + // diagnostic is still the truth about that window. + const unpaused = reduceGoalControl( + goalRecord({ + status: 'paused', + checkpointStalls: 2, + lastCheckpointFailure: failure, + }), + { + request: resume, + now: 200, + nextGoalId: 'unused', + cursor: { recordId: 'r-200' }, + }, + ); + expect(unpaused).toMatchObject({ + status: 'active', + checkpointStalls: 2, + lastCheckpointFailure: failure, + }); + }); + it('rejects a snapshot carrying negative spend', () => { expect( parseGoalSnapshotV2(snapshot(goalRecord({ tokensUsed: -1 }))), diff --git a/packages/core/src/goals/goal-reducer.ts b/packages/core/src/goals/goal-reducer.ts index d6cf0ae9d77..5b04efe3507 100644 --- a/packages/core/src/goals/goal-reducer.ts +++ b/packages/core/src/goals/goal-reducer.ts @@ -151,6 +151,7 @@ export function reduceGoalControl( evidenceCursor: copyCursor(transition.cursor), evidenceCheckpoint: undefined, checkpointStalls: undefined, + lastCheckpointFailure: undefined, noProgressTurns: undefined, ...rearmedBudgets(current, transition.now, transition), lastReason: undefined, @@ -228,6 +229,7 @@ export function reduceGoalControl( // a different one, so carrying it over would spend the new window's // allowance on the old window's failures. checkpointStalls: undefined, + lastCheckpointFailure: undefined, noProgressTurns: undefined, ...rearmedBudgets(current, transition.now, transition), lastReason: undefined, @@ -694,6 +696,7 @@ function parseGoalRecord(value: unknown): GoalRecord | undefined { 'updatedAt', 'evidenceCheckpoint', 'checkpointStalls', + 'lastCheckpointFailure', 'noProgressTurns', 'lastReason', 'limitKind', @@ -724,6 +727,9 @@ function parseGoalRecord(value: unknown): GoalRecord | undefined { !isGoalEvidenceCheckpoint(value['evidenceCheckpoint']) || (value['checkpointStalls'] !== undefined && !isNonNegativeInteger(value['checkpointStalls'])) || + (value['lastCheckpointFailure'] !== undefined && + (typeof value['lastCheckpointFailure'] !== 'string' || + !value['lastCheckpointFailure'])) || (value['noProgressTurns'] !== undefined && !isNonNegativeInteger(value['noProgressTurns'])) || (value['lastReason'] !== undefined && @@ -775,6 +781,9 @@ function parseGoalRecord(value: unknown): GoalRecord | undefined { ...(value['checkpointStalls'] ? { checkpointStalls: value['checkpointStalls'] } : {}), + ...(value['lastCheckpointFailure'] === undefined + ? {} + : { lastCheckpointFailure: value['lastCheckpointFailure'] }), ...(value['noProgressTurns'] ? { noProgressTurns: value['noProgressTurns'] } : {}), diff --git a/packages/core/src/goals/goal-runtime.test.ts b/packages/core/src/goals/goal-runtime.test.ts index f392d9fb390..5d49c928917 100644 --- a/packages/core/src/goals/goal-runtime.test.ts +++ b/packages/core/src/goals/goal-runtime.test.ts @@ -10,14 +10,18 @@ import type { GoalRecoveryRecord } from './goal-persistence.js'; import { GOAL_INFEASIBLE_NEXT_STEP, GOAL_CHECKPOINT_CLAIM_LIMIT, + GOAL_CHECKPOINT_FAILURE_MAX_CHARACTERS, GOAL_CHECKPOINT_REQUEST_TOO_LARGE_REASON, GOAL_CHECKPOINT_STALL_LIMIT, GOAL_CHECKPOINT_STALLED_REASON, + GOAL_CHECKPOINT_UNREACHABLE_REASON, + GOAL_CHECKPOINT_UNUSABLE_REASON, GOAL_DEFAULT_TOKEN_BUDGET, GOAL_NO_PROGRESS_TURN_LIMIT, GOAL_PAUSE_REASON_NO_PROGRESS, GOAL_PROPOSAL_REASON_MAX_BYTES, goalActiveTimeBudgetReason, + goalCheckpointHealthVisible, goalTurnBudgetReason, type GoalSnapshotV2, type GoalStateCause, @@ -33,12 +37,18 @@ import { type GoalTurnHost, } from './goal-runtime.js'; import { GoalConflictError } from './goal-reducer.js'; -import type { - GoalCheckpointVerificationResult, - GoalCheckpointVerifier, - GoalCheckpointVerifierInput, +import { + InvalidGoalCheckpointError, + type GoalCheckpointVerificationResult, + type GoalCheckpointVerifier, + type GoalCheckpointVerifierInput, } from './goal-checkpoint.js'; -import { GoalCheckpointVerifierInputTooLargeError } from './goal-checkpoint-verifier.js'; +import { + GoalCheckpointClaimBudgetError, + GoalCheckpointClaimCountError, + GoalCheckpointClaimLengthError, + GoalCheckpointVerifierInputTooLargeError, +} from './goal-checkpoint-verifier.js'; import type { GoalVerifier } from './goal-verifier.js'; // Records the GOAL_RUNTIME debug-log calls so tests can assert that a failed @@ -2013,6 +2023,7 @@ describe('goal runtime', () => { host, runtime, checkpointVerifier, + evidenceSource, setRecords: (next: readonly RuntimeRecord[]) => { records = next; }, @@ -2035,10 +2046,11 @@ describe('goal runtime', () => { `window-${stall}`, ); // Each stalled checkpoint is still written -- the streak is counted on - // the record, not held back in memory. + // the record, not held back in memory, and so is what it ran into. expect(runtime.getSnapshot().goal).toMatchObject({ status: 'active', checkpointStalls: stall, + lastCheckpointFailure: expect.stringContaining('full claim list'), }); } expect(host.started).toHaveLength(GOAL_CHECKPOINT_STALL_LIMIT); @@ -2105,6 +2117,10 @@ describe('goal runtime', () => { expect(checkpointVerifier).toHaveBeenCalledTimes(3); expect(runtime.getSnapshot().goal?.status).toBe('active'); expect(runtime.getSnapshot().goal).not.toHaveProperty('checkpointStalls'); + // A check that succeeded is the only thing that retires the diagnostic. + expect(runtime.getSnapshot().goal).not.toHaveProperty( + 'lastCheckpointFailure', + ); // The streak restarts from zero rather than continuing from two. await runCheckpointTurn(runtime, host, setRecords, records, 101, 'd'); @@ -2141,6 +2157,11 @@ describe('goal runtime', () => { expect(checkpointVerifier).toHaveBeenCalledTimes(2); expect(runtime.getSnapshot().goal?.status).toBe('active'); expect(runtime.getSnapshot().goal).not.toHaveProperty('checkpointStalls'); + // The check that found room is the one that proved the window healthy, + // so it also retires the diagnostic the two stalls left behind. + expect(runtime.getSnapshot().goal).not.toHaveProperty( + 'lastCheckpointFailure', + ); }); it('counts a verifier failure on an overflowing window as a stall', async () => { @@ -2177,13 +2198,17 @@ describe('goal runtime', () => { expect(checkpointVerifier).toHaveBeenCalledTimes( GOAL_CHECKPOINT_STALL_LIMIT, ); + // The stop follows the check that spent the last stall. The first two + // came back full, but the last never answered, so the advice is not to + // narrow the objective -- and the record says what the failure was. expect(runtime.getSnapshot()).toMatchObject({ activity: 'idle', goal: { status: 'usage_limited', limitKind: 'evidence_catalog', - lastReason: GOAL_CHECKPOINT_STALLED_REASON, + lastReason: GOAL_CHECKPOINT_UNREACHABLE_REASON, checkpointStalls: GOAL_CHECKPOINT_STALL_LIMIT, + lastCheckpointFailure: 'Error: provider failed', }, }); expect(journal.appended.at(-1)?.cause).toBe('usage_limited'); @@ -2213,9 +2238,13 @@ describe('goal runtime', () => { ]; setRecords(records); await runtime.finishTurn(permit); + // The failure is on the record from the first stall on, so every + // surface can show it long before the breaker has to stop the Goal. expect(runtime.getSnapshot().goal).toMatchObject({ status: 'active', checkpointStalls: turn, + lastCheckpointFailure: + 'Error: Goal checkpoint verifier timed out after 30000ms', evidenceCursor: { recordId: cursor }, }); expect(runtime.getSnapshot().goal).not.toHaveProperty( @@ -2239,8 +2268,10 @@ describe('goal runtime', () => { goal: { status: 'usage_limited', limitKind: 'evidence_catalog', - lastReason: GOAL_CHECKPOINT_STALLED_REASON, + lastReason: GOAL_CHECKPOINT_UNREACHABLE_REASON, checkpointStalls: GOAL_CHECKPOINT_STALL_LIMIT, + lastCheckpointFailure: + 'Error: Goal checkpoint verifier timed out after 30000ms', }, }); expect(journal.appended.at(-1)?.cause).toBe('usage_limited'); @@ -2284,6 +2315,9 @@ describe('goal runtime', () => { expect(runtime.getSnapshot().goal).toMatchObject({ status: 'active', checkpointStalls: turn, + lastCheckpointFailure: expect.stringMatching( + /^InvalidGoalCheckpointError: /, + ), }); } } @@ -2291,13 +2325,18 @@ describe('goal runtime', () => { expect(checkpointVerifier).toHaveBeenCalledTimes( GOAL_CHECKPOINT_STALL_LIMIT, ); + // The verifier answered every time, just not with claims: the stop says + // so instead of sending the user to narrow an objective that was fine. expect(runtime.getSnapshot()).toMatchObject({ activity: 'idle', goal: { status: 'usage_limited', limitKind: 'evidence_catalog', - lastReason: GOAL_CHECKPOINT_STALLED_REASON, + lastReason: GOAL_CHECKPOINT_UNUSABLE_REASON, checkpointStalls: GOAL_CHECKPOINT_STALL_LIMIT, + lastCheckpointFailure: expect.stringMatching( + /^InvalidGoalCheckpointError: /, + ), }, }); expect(journal.appended.at(-1)?.cause).toBe('usage_limited'); @@ -2305,6 +2344,188 @@ describe('goal runtime', () => { expect(host.started).toHaveLength(GOAL_CHECKPOINT_STALL_LIMIT); }); + it.each([ + ['claim-count', new GoalCheckpointClaimCountError(33)], + ['claim-budget', new GoalCheckpointClaimBudgetError(20_000)], + ['claim-length', new GoalCheckpointClaimLengthError(0, 9_000)], + ] as const)( + 'reads a %s overrun as capacity, not as unusable output', + async (_label, overrun) => { + // An overrun of a claim bound is well-formed JSON that could not fit the + // window within the checkpoint's bounds -- the capacity failure a + // narrower objective fixes -- so its stop keeps that advice rather than + // telling the user to debug JSON that was valid. + const { host, runtime, checkpointVerifier, setRecords } = stallHarness(); + checkpointVerifier.mockRejectedValue(overrun); + await runtime.dispatch({ action: 'create', objective: 'deliver result' }); + + let records: RuntimeRecord[] = []; + for (let turn = 1; turn <= GOAL_CHECKPOINT_STALL_LIMIT; turn++) { + records = await runCheckpointTurn( + runtime, + host, + setRecords, + records, + 101, + `overrun-${turn}`, + ); + } + + expect(runtime.getSnapshot().goal).toMatchObject({ + status: 'usage_limited', + limitKind: 'evidence_catalog', + lastReason: GOAL_CHECKPOINT_STALLED_REASON, + checkpointStalls: GOAL_CHECKPOINT_STALL_LIMIT, + lastCheckpointFailure: `${overrun.name}: ${overrun.message}`, + }); + }, + ); + + it('reads any other subclass of the unusable-result error as unusable output', async () => { + // Pins `instanceof` rather than an exact-constructor check: the hierarchy + // is open, and a new unusable shape must keep the unusable advice. + class ProbeUnusableCheckpointError extends InvalidGoalCheckpointError {} + const { host, runtime, checkpointVerifier, setRecords } = stallHarness(); + checkpointVerifier.mockRejectedValue( + new ProbeUnusableCheckpointError('probe'), + ); + await runtime.dispatch({ action: 'create', objective: 'deliver result' }); + + let records: RuntimeRecord[] = []; + for (let turn = 1; turn <= GOAL_CHECKPOINT_STALL_LIMIT; turn++) { + records = await runCheckpointTurn( + runtime, + host, + setRecords, + records, + 101, + `probe-${turn}`, + ); + } + + expect(runtime.getSnapshot().goal).toMatchObject({ + status: 'usage_limited', + lastReason: GOAL_CHECKPOINT_UNUSABLE_REASON, + }); + }); + + it('records a bounded, one-line, display-safe failure', async () => { + // The only production call site of the record's bound: a provider error + // carrying a whole response body must not reach the journal, the model + // or a card intact. + const { host, runtime, checkpointVerifier, setRecords } = stallHarness(); + checkpointVerifier.mockRejectedValueOnce( + new Error( + `upstream said:\n ${'x'.repeat(GOAL_CHECKPOINT_FAILURE_MAX_CHARACTERS + 100)}\r\u202e`, + ), + ); + await runtime.dispatch({ action: 'create', objective: 'deliver result' }); + + await runCheckpointTurn(runtime, host, setRecords, [], 101, 'body'); + + const value = runtime.getSnapshot().goal!.lastCheckpointFailure!; + expect([...value]).toHaveLength(GOAL_CHECKPOINT_FAILURE_MAX_CHARACTERS); + expect(value.endsWith('…')).toBe(true); + expect(value.startsWith('Error: upstream said: xxx')).toBe(true); + expect(value).not.toMatch(/[\n\r\u202e]/); + }); + + it('records the failure that made the checkpoint request too large on the stop it caused', async () => { + // The stop's diagnostic belongs to this stop, not to whatever an earlier + // check left: the too-large request is itself the failure to report. + const { host, runtime, checkpointVerifier, setRecords } = stallHarness(); + checkpointVerifier.mockRejectedValueOnce(new Error('provider failed')); + await runtime.dispatch({ action: 'create', objective: 'deliver result' }); + + const records = await runCheckpointTurn( + runtime, + host, + setRecords, + [], + 101, + 'first', + ); + expect(runtime.getSnapshot().goal).toMatchObject({ + checkpointStalls: 1, + lastCheckpointFailure: 'Error: provider failed', + }); + + checkpointVerifier.mockRejectedValueOnce( + new GoalCheckpointVerifierInputTooLargeError(300_000), + ); + await runCheckpointTurn(runtime, host, setRecords, records, 101, 'second'); + + expect(runtime.getSnapshot().goal).toMatchObject({ + status: 'usage_limited', + limitKind: 'checkpoint_request', + lastReason: GOAL_CHECKPOINT_REQUEST_TOO_LARGE_REASON, + checkpointStalls: 1, + // The measured size, so a reader can tell how far over the limit it is. + lastCheckpointFailure: expect.stringMatching( + /^GoalCheckpointVerifierInputTooLargeError: .*\b300000 bytes\b/, + ), + }); + }); + + it('shows the failure of a checkpoint request too large to send, with no streak behind it', async () => { + // That stop spends no stall, so only its own limit kind can make the + // failure it recorded visible. + const { host, runtime, checkpointVerifier, setRecords } = stallHarness(); + checkpointVerifier.mockRejectedValueOnce( + new GoalCheckpointVerifierInputTooLargeError(300_000), + ); + await runtime.dispatch({ action: 'create', objective: 'deliver result' }); + + await runCheckpointTurn(runtime, host, setRecords, [], 101, 'only'); + + const goal = runtime.getSnapshot().goal!; + expect(goal).toMatchObject({ + status: 'usage_limited', + limitKind: 'checkpoint_request', + lastCheckpointFailure: expect.stringContaining('300000 bytes'), + }); + expect(goal).not.toHaveProperty('checkpointStalls'); + expect(goalCheckpointHealthVisible(goal)).toBe(true); + }); + + it('clears the diagnostic but keeps the streak when a checkpoint stop has a cause of its own', async () => { + // Its own lastReason says what happened; an earlier provider failure left + // on the record would read as the cause of this stop. + const { host, runtime, checkpointVerifier, evidenceSource, setRecords } = + stallHarness(); + checkpointVerifier.mockRejectedValue(new Error('provider failed')); + await runtime.dispatch({ action: 'create', objective: 'deliver result' }); + + let records: RuntimeRecord[] = []; + for (let turn = 1; turn < GOAL_CHECKPOINT_STALL_LIMIT; turn++) { + records = await runCheckpointTurn( + runtime, + host, + setRecords, + records, + 101, + `stall-${turn}`, + ); + } + expect(runtime.getSnapshot().goal).toMatchObject({ + checkpointStalls: GOAL_CHECKPOINT_STALL_LIMIT - 1, + lastCheckpointFailure: 'Error: provider failed', + }); + + evidenceSource.readActiveTranscriptChain.mockRejectedValueOnce( + new Error('evidence source unavailable'), + ); + await runCheckpointTurn(runtime, host, setRecords, records, 101, 'lost'); + + const goal = runtime.getSnapshot().goal!; + expect(goal).toMatchObject({ + status: 'usage_limited', + lastReason: 'evidence source unavailable', + checkpointStalls: GOAL_CHECKPOINT_STALL_LIMIT - 1, + }); + expect(goal).not.toHaveProperty('lastCheckpointFailure'); + }); + it('does not count an unusable result while the window has room', async () => { const { host, runtime, checkpointVerifier, setRecords } = stallHarness(); checkpointVerifier.mockResolvedValue({ claims: [] }); @@ -2355,9 +2576,15 @@ describe('goal runtime', () => { checkpointVerifier.mockRejectedValueOnce(new Error('provider failed')); await runCheckpointTurn(runtime, host, setRecords, records, 60, 'b'); + // The failure spends no stall, but it is still a failure the surfaces + // should show: the record carries it beside the unchanged streak. expect(runtime.getSnapshot()).toMatchObject({ activity: 'running', - goal: { status: 'active', checkpointStalls: 1 }, + goal: { + status: 'active', + checkpointStalls: 1, + lastCheckpointFailure: 'Error: provider failed', + }, }); // The room arm leaves the same trace the truncated arm does: the // discarded error is diagnosable from the first failure, not only once @@ -2392,9 +2619,12 @@ describe('goal runtime', () => { // only. That close proved nothing about room, so it keeps the streak. await runCheckpointTurn(runtime, host, setRecords, records, 0, 'quiet'); + // The same close proved nothing about the failure either, so the + // diagnostic the stall left stays until a check actually succeeds. expect(runtime.getSnapshot().goal).toMatchObject({ status: 'active', checkpointStalls: 1, + lastCheckpointFailure: expect.stringContaining('full claim list'), }); expect(host.started).toHaveLength(3); }); @@ -5044,11 +5274,14 @@ describe('goal runtime', () => { // stop: the restored Goal keeps the streak it crashed with, and the // continuation the replay mints re-earns any stall as a live turn. expect(checkpointVerifier).toHaveBeenCalledOnce(); + // The exemption covers the streak, not the diagnostic: the replay did + // fail, and the record says so without spending a stall on it. expect(runtime.getSnapshot()).toMatchObject({ activity: 'running', goal: { status: 'active', checkpointStalls: GOAL_CHECKPOINT_STALL_LIMIT - 1, + lastCheckpointFailure: 'Error: provider failed', }, }); expect(journal.appended.map((payload) => payload.cause)).toEqual([ @@ -5073,8 +5306,9 @@ describe('goal runtime', () => { goal: { status: 'usage_limited', limitKind: 'evidence_catalog', - lastReason: GOAL_CHECKPOINT_STALLED_REASON, + lastReason: GOAL_CHECKPOINT_UNREACHABLE_REASON, checkpointStalls: GOAL_CHECKPOINT_STALL_LIMIT, + lastCheckpointFailure: 'Error: provider failed', }, }); expect(journal.appended.map((payload) => payload.cause)).toEqual([ diff --git a/packages/core/src/goals/goal-runtime.ts b/packages/core/src/goals/goal-runtime.ts index 35266d70830..ae2333d8842 100644 --- a/packages/core/src/goals/goal-runtime.ts +++ b/packages/core/src/goals/goal-runtime.ts @@ -16,15 +16,23 @@ import { type GoalEvidenceRecord, } from './goal-evidence.js'; import { + InvalidGoalCheckpointError, isGoalCheckpointStalled, materializeGoalEvidenceCheckpoint, type GoalCheckpointVerifier, } from './goal-checkpoint.js'; -import { GoalCheckpointVerifierInputTooLargeError } from './goal-checkpoint-verifier.js'; import { + GoalCheckpointClaimBudgetError, + GoalCheckpointClaimCountError, + GoalCheckpointClaimLengthError, + GoalCheckpointVerifierInputTooLargeError, +} from './goal-checkpoint-verifier.js'; +import { + capGoalCheckpointFailure, + GOAL_CHECKPOINT_CLAIM_LIMIT, GOAL_CHECKPOINT_REQUEST_TOO_LARGE_REASON, GOAL_CHECKPOINT_STALL_LIMIT, - GOAL_CHECKPOINT_STALLED_REASON, + goalCheckpointStalledReason, GOAL_DEFAULT_TOKEN_BUDGET, GOAL_EVIDENCE_CATALOG_EXHAUSTED_REASON, GOAL_INFEASIBLE_NEXT_STEP, @@ -38,6 +46,7 @@ import { isGoalTokenBudgetSpent, isGoalTurnBudgetSpent, isRepeatedBlockerProposal, + type GoalCheckpointFailureShape, type GoalControlRequest, type GoalEvidenceCheckpoint, type GoalLimitKind, @@ -74,6 +83,56 @@ import { createDebugLogger } from '../utils/debugLogger.js'; const debugLogger = createDebugLogger('GOAL_RUNTIME'); +/** + * What a failed checkpoint check ran into. The shape decides the advice a + * stall stop carries; the detail is the one-line diagnostic the record keeps. + */ +interface CheckpointFailure { + shape: GoalCheckpointFailureShape; + detail: string; +} + +/** + * Classifies a checkpoint check that threw. A claim-count, claim-budget or + * claim-length overrun is well-formed output that could not fit the window + * within the checkpoint's bounds -- the same capacity failure as a full claim + * list, with the same remedy -- so those three subclasses are tested before + * their base class. Any other `InvalidGoalCheckpointError` means the verifier answered + * with output that is not usable claims. Anything else means no answer + * arrived to judge: a provider error, the check's own timeout, or a failure + * inside the check. + */ +function describeCheckpointFailure(error: unknown): CheckpointFailure { + return { + shape: + error instanceof GoalCheckpointClaimCountError || + error instanceof GoalCheckpointClaimBudgetError || + error instanceof GoalCheckpointClaimLengthError + ? 'capacity' + : error instanceof InvalidGoalCheckpointError + ? 'unusable' + : 'unreachable', + detail: capGoalCheckpointFailure( + error instanceof Error + ? `${error.name}: ${error.message}` + : String(error), + ), + }; +} + +/** The stall a checkpoint that came back at the claim ceiling spends. */ +const FULL_CLAIM_LIST_FAILURE: CheckpointFailure = { + shape: 'capacity', + detail: `checkpoint came back with a full claim list (${GOAL_CHECKPOINT_CLAIM_LIMIT} claims) while the evidence window overflowed`, +}; + +/** + * What a checkpoint check tells the record about checkpoint health: a failure + * to report, `'clear'` after a check that succeeded, or `undefined` when the + * check proved nothing either way and the previous diagnostic stays. + */ +type CheckpointHealthUpdate = CheckpointFailure | 'clear' | undefined; + export const GOAL_RUNTIME_DISPOSED_MESSAGE = 'Goal runtime has been disposed'; export const STALE_GOAL_TURN_MESSAGE = 'Goal turn permit is no longer valid'; @@ -559,12 +618,32 @@ export function createGoalRuntime( }).catch(() => undefined); }; - const withCheckpointStalls = ( + /** + * The Goal record with its checkpoint health brought up to date: the stall + * streak (zero is spelled as no field) and the last failure diagnostic, + * which moves only when the check actually learned something. + */ + const withCheckpointHealth = ( goal: NonNullable, checkpointStalls: number, + update: CheckpointHealthUpdate, ): NonNullable => { - const { checkpointStalls: _previous, ...rest } = goal; - return checkpointStalls > 0 ? { ...rest, checkpointStalls } : rest; + const { + checkpointStalls: _previousStalls, + lastCheckpointFailure: previousFailure, + ...rest + } = goal; + const lastCheckpointFailure = + update === 'clear' + ? undefined + : update === undefined + ? previousFailure + : update.detail; + return { + ...rest, + ...(checkpointStalls > 0 ? { checkpointStalls } : {}), + ...(lastCheckpointFailure ? { lastCheckpointFailure } : {}), + }; }; const assertAvailable = () => { @@ -1094,6 +1173,7 @@ export function createGoalRuntime( const finishCheckpointCheck = async ( attempt: CheckpointAttempt, outcome: 'room' | 'stalled' | 'inconclusive' = 'inconclusive', + failure?: CheckpointFailure, ): Promise => { await enqueue(async () => { if (!isCurrentCheckpointAttempt(attempt) || !snapshot.goal) return; @@ -1109,11 +1189,18 @@ export function createGoalRuntime( : outcome === 'stalled' ? (snapshot.goal.checkpointStalls ?? 0) + 1 : (snapshot.goal.checkpointStalls ?? 0); + // The diagnostic follows what the check learned, not the streak: a + // check that found room clears it, a check that failed reports why + // whether or not it spent a stall, and a check that never ran leaves + // the previous one standing. + const health: CheckpointHealthUpdate = + outcome === 'room' ? 'clear' : failure; if ( await settleIfCheckpointStalled( attempt, snapshot.goal, checkpointStalls, + health, ) ) { return; @@ -1124,7 +1211,7 @@ export function createGoalRuntime( const checkedSnapshot: GoalSnapshotV2 = { v: GOAL_STATE_VERSION, goal: { - ...withCheckpointStalls(snapshot.goal, checkpointStalls), + ...withCheckpointHealth(snapshot.goal, checkpointStalls, health), activeTimeMs: elapsedActiveTime(snapshot.goal, now), updatedAt: now, }, @@ -1170,32 +1257,56 @@ export function createGoalRuntime( /** * Stops the Goal once its stall streak reaches the limit, persisting the - * streak with the stop so the record explains itself. Returns whether the - * attempt was settled. + * streak and the last failure with the stop so the record explains itself. + * The stop reason follows the check that spent the last stall: narrowing + * the objective is the remedy for a full claim list, not for a verifier + * that answered unusably or never answered. Every arm that spends a stall + * reports its failure; a streak reaching the limit without one falls back + * to the compaction reading. Returns whether the attempt was settled. */ const settleIfCheckpointStalled = async ( attempt: CheckpointAttempt, goal: NonNullable, checkpointStalls: number, + health: CheckpointHealthUpdate, ): Promise => { if (checkpointStalls < GOAL_CHECKPOINT_STALL_LIMIT) return false; + const shape: GoalCheckpointFailureShape = + health !== undefined && health !== 'clear' ? health.shape : 'capacity'; await settleCheckpointFailure( attempt, - withCheckpointStalls(goal, checkpointStalls), - GOAL_CHECKPOINT_STALLED_REASON, + withCheckpointHealth(goal, checkpointStalls, health), + goalCheckpointStalledReason(shape), 'evidence_catalog', ); return true; }; + /** + * Stops the Goal at a checkpoint bound other than the stall breaker. The + * record's diagnostic is scoped to this stop: an arm whose cause is itself a + * failed check (the request that was too large) records that failure, and + * every other arm clears whatever an earlier, unrelated check left, since + * its own `lastReason` already says what happened. + */ const recordCheckpointFailure = async ( attempt: CheckpointAttempt, reason: string, limitKind?: GoalLimitKind, + health: CheckpointHealthUpdate = 'clear', ): Promise => { await enqueue(async () => { if (!isCurrentCheckpointAttempt(attempt) || !snapshot.goal) return; - await settleCheckpointFailure(attempt, snapshot.goal, reason, limitKind); + await settleCheckpointFailure( + attempt, + withCheckpointHealth( + snapshot.goal, + snapshot.goal.checkpointStalls ?? 0, + health, + ), + reason, + limitKind, + ); }); }; @@ -1210,6 +1321,9 @@ export function createGoalRuntime( const checkpointStalls = stalled ? (snapshot.goal.checkpointStalls ?? 0) + 1 : 0; + const health: CheckpointHealthUpdate = stalled + ? FULL_CLAIM_LIST_FAILURE + : 'clear'; // A stopped Goal discards the checkpoint it would have written: a // resumed window restarts from a fresh cursor anyway. if ( @@ -1217,6 +1331,7 @@ export function createGoalRuntime( attempt, snapshot.goal, checkpointStalls, + health, ) ) { return; @@ -1227,7 +1342,7 @@ export function createGoalRuntime( const checkpointSnapshot: GoalSnapshotV2 = { v: GOAL_STATE_VERSION, goal: { - ...withCheckpointStalls(snapshot.goal, checkpointStalls), + ...withCheckpointHealth(snapshot.goal, checkpointStalls, health), evidenceCursor: { recordId: attempt.recordUuid }, evidenceCheckpoint: checkpoint, activeTimeMs: elapsedActiveTime(snapshot.goal, now), @@ -1327,6 +1442,7 @@ export function createGoalRuntime( attempt, GOAL_CHECKPOINT_REQUEST_TOO_LARGE_REASON, 'checkpoint_request', + describeCheckpointFailure(error), ); return; } @@ -1336,6 +1452,10 @@ export function createGoalRuntime( error, `replay=${replay}`, ); + // The trace above is for an investigation; this is what the record + // keeps, on every arm, so the failure is visible before the stall + // breaker stops the Goal and still readable after it has. + const failure = describeCheckpointFailure(error); // A restore replay is exempt: it runs no turn of its own, so a // transient failure at startup must not spend a streak the restored // session never re-earned. The replay mints a continuation whose own @@ -1350,13 +1470,13 @@ export function createGoalRuntime( // overflowing window run a Goal in circles: each turn paid the // call, kept the same cursor, and was told to retry, with nothing // but the token budget left to stop it. - await finishCheckpointCheck(attempt, 'stalled'); + await finishCheckpointCheck(attempt, 'stalled', failure); return; } // A failure while the window still has room must not abort a // healthy Goal: settle the attempt as bookkeeping so the evidence // stays citable and a later turn retries the checkpoint. - await finishCheckpointCheck(attempt); + await finishCheckpointCheck(attempt, 'inconclusive', failure); return; } await recordCheckpoint( diff --git a/packages/core/src/goals/goal-tools.test.ts b/packages/core/src/goals/goal-tools.test.ts index b79ac93523d..b8baded830f 100644 --- a/packages/core/src/goals/goal-tools.test.ts +++ b/packages/core/src/goals/goal-tools.test.ts @@ -248,6 +248,192 @@ describe('GetGoalTool', () => { ); }); + it('reports checkpoint health in the last Goal summary', async () => { + // A Goal the stall breaker stopped names the kind of failure in + // lastReason; only these two fields say how often and what it was. + const failure = + 'InvalidGoalCheckpointError: Goal checkpoint verifier returned invalid JSON'; + const config = makeConfig({ + getGoalForWorker: vi.fn(), + getSnapshot: () => ({ + v: 2 as const, + activity: 'idle' as const, + goal: { + goalId: 'goal-1', + revision: 3, + objective: 'Ship Goal v3', + status: 'usage_limited' as const, + evidenceCursor: { recordId: 'record-1' }, + turnCount: 5, + activeTimeMs: 10, + tokensUsed: 0, + createdAt: 1, + updatedAt: 2, + checkpointStalls: 3, + lastCheckpointFailure: failure, + lastReason: 'checkpoints stalled', + limitKind: 'evidence_catalog' as const, + }, + }), + }); + + const result = await execute(new GetGoalTool(config)); + + expect(JSON.parse(String(result.llmContent))).toEqual({ + active: false, + lastGoal: { + goalId: 'goal-1', + revision: 3, + status: 'usage_limited', + turnCount: 5, + activeTimeMs: 10, + tokensUsed: 0, + checkpointStalls: 3, + lastCheckpointFailure: failure, + lastReason: 'checkpoints stalled', + }, + }); + }); + + it('leaves checkpoint health out of the summary of a completed Goal', async () => { + // The terminal snapshot keeps whatever the record carried; a Goal that + // completed cleanly must not be reported with a stale failure. + const config = makeConfig({ + getGoalForWorker: vi.fn(), + getSnapshot: () => ({ + v: 2 as const, + activity: 'idle' as const, + goal: { + goalId: 'goal-1', + revision: 3, + objective: 'Ship Goal v3', + status: 'complete' as const, + evidenceCursor: { recordId: 'record-1' }, + turnCount: 5, + activeTimeMs: 10, + tokensUsed: 0, + createdAt: 1, + updatedAt: 2, + checkpointStalls: 1, + lastCheckpointFailure: 'Error: provider failed', + lastReason: 'Evidence satisfies the objective', + }, + }), + }); + + const { lastGoal } = JSON.parse( + String((await execute(new GetGoalTool(config))).llmContent), + ); + + expect(lastGoal).toMatchObject({ + status: 'complete', + lastReason: 'Evidence satisfies the objective', + }); + expect(lastGoal).not.toHaveProperty('checkpointStalls'); + expect(lastGoal).not.toHaveProperty('lastCheckpointFailure'); + }); + + it('leaves a stall-free failure out of the summary of a Goal paused for another reason', async () => { + const config = makeConfig({ + getGoalForWorker: vi.fn(), + getSnapshot: () => ({ + v: 2 as const, + activity: 'idle' as const, + goal: { + goalId: 'goal-1', + revision: 3, + objective: 'Ship Goal v3', + status: 'paused' as const, + evidenceCursor: { recordId: 'record-1' }, + turnCount: 5, + activeTimeMs: 10, + tokensUsed: 0, + createdAt: 1, + updatedAt: 2, + lastCheckpointFailure: 'Error: provider failed', + lastReason: 'no progress in three turns', + }, + }), + }); + + const { lastGoal } = JSON.parse( + String((await execute(new GetGoalTool(config))).llmContent), + ); + + expect(lastGoal).toMatchObject({ status: 'paused' }); + expect(lastGoal).not.toHaveProperty('lastCheckpointFailure'); + }); + + it('reports a stall-free checkpoint failure while the Goal is still active', async () => { + // A turn without a Goal permit can find the Goal still active; the failure + // is what its later checks keep running into before any stall stops it. + const config = makeConfig({ + getGoalForWorker: vi.fn(), + getSnapshot: () => ({ + v: 2 as const, + activity: 'idle' as const, + goal: { + goalId: 'goal-1', + revision: 3, + objective: 'Ship Goal v3', + status: 'active' as const, + evidenceCursor: { recordId: 'record-1' }, + turnCount: 5, + activeTimeMs: 10, + tokensUsed: 0, + createdAt: 1, + updatedAt: 2, + lastCheckpointFailure: 'Error: provider failed', + }, + }), + }); + + const { lastGoal } = JSON.parse( + String((await execute(new GetGoalTool(config))).llmContent), + ); + + expect(lastGoal.lastCheckpointFailure).toBe('Error: provider failed'); + expect(lastGoal).not.toHaveProperty('checkpointStalls'); + }); + + it('reports the failure that stopped a Goal whose checkpoint request was too large', async () => { + // That stop spends no stall, and its failure is the whole explanation. + const failure = + 'GoalCheckpointVerifierInputTooLargeError: Goal checkpoint verifier request of 300000 bytes exceeds the 256000-byte limit'; + const config = makeConfig({ + getGoalForWorker: vi.fn(), + getSnapshot: () => ({ + v: 2 as const, + activity: 'idle' as const, + goal: { + goalId: 'goal-1', + revision: 3, + objective: 'Ship Goal v3', + status: 'usage_limited' as const, + evidenceCursor: { recordId: 'record-1' }, + turnCount: 5, + activeTimeMs: 10, + tokensUsed: 0, + createdAt: 1, + updatedAt: 2, + lastCheckpointFailure: failure, + lastReason: 'checkpoint request too large', + limitKind: 'checkpoint_request' as const, + }, + }), + }); + + const { lastGoal } = JSON.parse( + String((await execute(new GetGoalTool(config))).llmContent), + ); + + expect(lastGoal).toMatchObject({ + status: 'usage_limited', + lastCheckpointFailure: failure, + }); + expect(lastGoal).not.toHaveProperty('checkpointStalls'); + }); + it('keeps the objective and the evidence checkpoint behind the permit', async () => { const config = makeConfig({ getGoalForWorker: vi.fn(), @@ -992,7 +1178,11 @@ describe('UpdateGoalTool', () => { readyForVerification: false, goalLifecycleChanged: false, checkpointRequired: true, - nextAction: expect.stringContaining('checkpoint the evidence catalog'), + // A retry that runs another tool first can push a cited entry out of + // the catalog, so the hint says to retry before anything else. + nextAction: expect.stringMatching( + /checkpoint the evidence catalog[\s\S]*before running any other tool/, + ), }); expect(result.terminateTurn).toBe(true); expect(recordTerminalProposal).not.toHaveBeenCalled(); diff --git a/packages/core/src/goals/goal-tools.ts b/packages/core/src/goals/goal-tools.ts index ebe8900c9a1..62a911f3d55 100644 --- a/packages/core/src/goals/goal-tools.ts +++ b/packages/core/src/goals/goal-tools.ts @@ -40,6 +40,7 @@ import { goalTurnContext } from './goal-turn-context.js'; import { type GoalBlockerKind, type GoalControlRequest, + goalCheckpointHealthVisible, GOAL_PROPOSAL_REASON_MAX_CHARACTERS, type GoalRecord, type GoalSnapshotV2, @@ -92,6 +93,8 @@ type LastGoalSummary = Pick< | 'tokenBudget' | 'turnBudget' | 'activeTimeBudgetMs' + | 'checkpointStalls' + | 'lastCheckpointFailure' | 'lastReason' >; @@ -160,7 +163,7 @@ export class GetGoalTool extends BaseDeclarativeTool< super( GetGoalTool.Name, ToolDisplayNames.GET_GOAL, - `Read the current Goal identity, objective, evidence cursor, and bounded evidence-reference catalog for this permitted Goal turn. The default "summary" view keeps every read small: checkpoint claims are reported as a count (each claim is already an evidenceCatalog entry with its own preview), entries from this turn and checkpoint entries keep full previews, and entries from earlier turns carry previews shortened to ${SUMMARY_PREVIEW_BYTE_LIMIT} bytes. Every entry uuid is present in both views and is valid for update_goal; request view "full" only when a shortened preview is not enough to decide what to cite. Outside a permitted Goal turn it reports "active": false together with "lastGoal", a scalar summary (goalId, revision, status, turnCount, activeTimeMs, tokensUsed, plus tokenBudget, turnBudget, activeTimeBudgetMs and lastReason when recorded) of the session's most recent Goal, so a Goal that has already stopped can still be inspected. It never returns uncited transcript history or changes Goal state. Use the result silently; do not narrate or acknowledge the retrieval to the user.`, + `Read the current Goal identity, objective, evidence cursor, and bounded evidence-reference catalog for this permitted Goal turn. The default "summary" view keeps every read small: checkpoint claims are reported as a count (each claim is already an evidenceCatalog entry with its own preview), entries from this turn and checkpoint entries keep full previews, and entries from earlier turns carry previews shortened to ${SUMMARY_PREVIEW_BYTE_LIMIT} bytes. Every entry uuid is present in both views and is valid for update_goal; request view "full" only when a shortened preview is not enough to decide what to cite. Outside a permitted Goal turn it reports "active": false together with "lastGoal", a scalar summary (goalId, revision, status, turnCount, activeTimeMs, tokensUsed, plus tokenBudget, turnBudget, activeTimeBudgetMs and lastReason when recorded, and checkpointStalls and lastCheckpointFailure while a stall streak stands, while the Goal is active, or when an oversized checkpoint request stopped it, never for a completed Goal) of the session's most recent Goal, so a Goal that has already stopped can still be inspected. The Goal record's checkpointStalls counts consecutive evidence checkpoints that failed to relieve an overflowing catalog (the Goal stops at three), and lastCheckpointFailure says what the most recent check that gave no relief ran into (a thrown error, or a full claim list on an overflowing window). It never returns uncited transcript history or changes Goal state. Use the result silently; do not narrate or acknowledge the retrieval to the user.`, Kind.Read, { type: 'object', @@ -225,6 +228,20 @@ export class GetGoalTool extends BaseDeclarativeTool< ...(goal.activeTimeBudgetMs === undefined ? {} : { activeTimeBudgetMs: goal.activeTimeBudgetMs }), + // A Goal the stall breaker stopped names the kind of failure in + // `lastReason`; these two say how often and what exactly it was. They + // follow the visibility rule every rendered card uses, so a Goal that + // completed cleanly is not reported with a stale failure. + ...(goalCheckpointHealthVisible(goal) + ? { + ...(goal.checkpointStalls + ? { checkpointStalls: goal.checkpointStalls } + : {}), + ...(goal.lastCheckpointFailure === undefined + ? {} + : { lastCheckpointFailure: goal.lastCheckpointFailure }), + } + : {}), ...(goal.lastReason === undefined ? {} : { lastReason: goal.lastReason }), }; } @@ -350,7 +367,7 @@ class UpdateGoalInvocation extends BaseToolInvocation< goalLifecycleChanged: false, checkpointRequired: true, nextAction: - 'End this turn without user-facing text so the runtime can checkpoint the evidence catalog. In the next Goal turn, call get_goal and retry the terminal proposal with the new evidence UUIDs.', + 'End this turn without user-facing text so the runtime can checkpoint the evidence catalog. In the next Goal turn, call get_goal first and retry the terminal proposal with the UUIDs it returns before running any other tool: every new tool result can push an older entry out of the bounded catalog and invalidate a UUID you meant to cite.', }), returnDisplay: 'Goal evidence reached its bounded catalog; ending the turn to checkpoint before terminal verification.', @@ -412,7 +429,7 @@ export class UpdateGoalTool extends BaseDeclarativeTool< super( UpdateGoalTool.Name, ToolDisplayNames.UPDATE_GOAL, - 'Propose that the current Goal is complete or blocked. Before calling, call get_goal in the current turn and cite only values from evidenceCatalog.entries[].uuid, never goalId, turnId, or lineageTurnIds. If completion depends on user-facing content delivered in the current turn, emit only the content required by the objective, then call get_goal, wait for its result, and call update_goal in a later model step with the returned delivered_output UUID. Do not add progress or completion commentary when the objective requires an exact output format. For blocked proposals, use authority when a user or maintainer decision or permission is required, external when an unavailable external resource or capability is evidenced, repeated for the same evidenced blocker with the exact same reason text across three consecutive Goal turns, and infeasible when a cited external_fact (a tool result, not your own text) shows the objective cannot be satisfied as written -- it contradicts itself, names a target that verifiably does not exist, or needs an action no tool can perform; infeasible is not for difficulty, uncertainty, information you could still obtain, or wanting to ask, and its reason must state what was checked and why no in-scope work could satisfy the objective. Omitting blockerKind follows the repeated-blocker audit. Core records at most one proposal for the exact permitted turn and queues eligible proposals for independent verification. This tool never changes the Goal lifecycle or claims a terminal result. Do not tell the user the Goal is complete or blocked. If this tool reports readyForVerification or checkpointRequired, end the turn without additional user-facing text; after checkpointRequired, call get_goal and retry in the next Goal turn. Otherwise continue the turn without claiming a terminal result. The Goal status card reports the independent verification result.', + 'Propose that the current Goal is complete or blocked. Before calling, call get_goal in the current turn and cite only values from evidenceCatalog.entries[].uuid, never goalId, turnId, or lineageTurnIds. If completion depends on user-facing content delivered in the current turn, emit only the content required by the objective, then call get_goal, wait for its result, and call update_goal in a later model step with the returned delivered_output UUID. Do not add progress or completion commentary when the objective requires an exact output format. For blocked proposals, use authority when a user or maintainer decision or permission is required, external when an unavailable external resource or capability is evidenced, repeated for the same evidenced blocker with the exact same reason text across three consecutive Goal turns, and infeasible when a cited external_fact (a tool result, not your own text) shows the objective cannot be satisfied as written -- it contradicts itself, names a target that verifiably does not exist, or needs an action no tool can perform; infeasible is not for difficulty, uncertainty, information you could still obtain, or wanting to ask, and its reason must state what was checked and why no in-scope work could satisfy the objective. Omitting blockerKind follows the repeated-blocker audit. Core records at most one proposal for the exact permitted turn and queues eligible proposals for independent verification. This tool never changes the Goal lifecycle or claims a terminal result. Do not tell the user the Goal is complete or blocked. If this tool reports readyForVerification or checkpointRequired, end the turn without additional user-facing text; after checkpointRequired, call get_goal first in the next Goal turn and retry before running any other tool, because every new tool result can push a cited entry out of the bounded catalog. Otherwise continue the turn without claiming a terminal result. The Goal status card reports the independent verification result.', Kind.Think, { type: 'object', diff --git a/packages/sdk-typescript/src/daemon/index.ts b/packages/sdk-typescript/src/daemon/index.ts index 91c74cbe789..f8337bd7a1c 100644 --- a/packages/sdk-typescript/src/daemon/index.ts +++ b/packages/sdk-typescript/src/daemon/index.ts @@ -246,6 +246,7 @@ export { DAEMON_APPROVAL_MODES, DAEMON_ERROR_KINDS, DaemonCapabilityMissingError, + GOAL_CHECKPOINT_STALL_LIMIT, GOAL_PAUSE_REASON_COMMAND, isDaemonContentHash, requireWorkspaceCwd, diff --git a/packages/sdk-typescript/src/daemon/types.ts b/packages/sdk-typescript/src/daemon/types.ts index 44ca5b4e515..af463934b97 100644 --- a/packages/sdk-typescript/src/daemon/types.ts +++ b/packages/sdk-typescript/src/daemon/types.ts @@ -86,6 +86,23 @@ export interface GoalRecord { activeTimeBudgetMs?: number; createdAt: number; updatedAt: number; + /** + * Consecutive evidence checkpoints that failed to relieve an overflowing + * window; the Goal stops when this reaches three. Absent means zero, which + * is also what an older daemon's snapshot looks like. + */ + checkpointStalls?: number; + /** + * A one-line diagnostic for the most recent checkpoint check that gave no + * relief: `ErrorName: message` for a check that failed, or the runtime's own + * phrase for one that answered with a full claim list while the window + * overflowed, so it does not always mean the check threw. Cleared by a check + * that finds room or writes a checkpoint without stalling, by every control + * action that clears `checkpointStalls`, and by a checkpoint stop whose cause + * is not itself a check, so it can be absent while `checkpointStalls` is + * still non-zero. Also absent when the daemon predates the field. + */ + lastCheckpointFailure?: string; lastReason?: string; limitKind?: GoalLimitKind; } @@ -110,6 +127,13 @@ export interface GoalSnapshotV2 { */ export const GOAL_PAUSE_REASON_COMMAND = 'Paused with /goal pause.'; +/** + * How many consecutive stalled evidence checkpoints stop a Goal, duplicated so + * a client can show `checkpointStalls` against it. It must match + * `GOAL_CHECKPOINT_STALL_LIMIT` in `packages/core/src/goals/goal-protocol.ts`. + */ +export const GOAL_CHECKPOINT_STALL_LIMIT = 3; + export type GoalControlRequest = | { action: 'create'; objective: string } | { diff --git a/packages/web-shell/client/components/GoalStatusStrip.module.css b/packages/web-shell/client/components/GoalStatusStrip.module.css index a5923875381..2e17209e937 100644 --- a/packages/web-shell/client/components/GoalStatusStrip.module.css +++ b/packages/web-shell/client/components/GoalStatusStrip.module.css @@ -60,6 +60,16 @@ color: color-mix(in srgb, var(--muted-foreground) 78%, transparent); } +/* Gives way on a narrow pane rather than pushing into the action buttons. */ +.checkpoint { + flex: 0 1 auto; + min-width: 0; + overflow: hidden; + text-overflow: ellipsis; + white-space: nowrap; + color: var(--warning-color); +} + .actions { display: inline-flex; align-items: center; diff --git a/packages/web-shell/client/components/GoalStatusStrip.test.tsx b/packages/web-shell/client/components/GoalStatusStrip.test.tsx index 9313729a8b7..20e35a84092 100644 --- a/packages/web-shell/client/components/GoalStatusStrip.test.tsx +++ b/packages/web-shell/client/components/GoalStatusStrip.test.tsx @@ -244,4 +244,45 @@ describe('GoalStatusStrip', () => { ?.textContent, ).toBe('2.5M / 30.0M tokens'); }); + + it('shows a running checkpoint stall streak, like the terminal footer pill', () => { + render('active', { + checkpointStalls: 2, + lastCheckpointFailure: 'Error: provider failed', + }); + + expect( + container.querySelector('[data-testid="goal-checkpoint-stalls"]') + ?.textContent, + ).toBe('2/3 checks stalled'); + // The label is ellipsized on a narrow pane, so it is also the tooltip. + expect( + container + .querySelector('[data-testid="goal-checkpoint-stalls"]') + ?.getAttribute('title'), + ).toBe('2/3 checks stalled'); + // The failure text belongs to the Goals dialog; the strip has no room. + expect(container.textContent).not.toContain('provider failed'); + }); + + it('keeps the streak on a Goal the stall breaker stopped', () => { + render('usage_limited', { + checkpointStalls: 3, + lastCheckpointFailure: 'Error: provider failed', + }); + + expect( + container.querySelector('[data-testid="goal-checkpoint-stalls"]') + ?.textContent, + ).toBe('3/3 checks stalled'); + expect(container.textContent).not.toContain('provider failed'); + }); + + it('shows no streak when no checkpoint has stalled', () => { + render('active', { lastCheckpointFailure: 'Error: provider failed' }); + + expect( + container.querySelector('[data-testid="goal-checkpoint-stalls"]'), + ).toBeNull(); + }); }); diff --git a/packages/web-shell/client/components/GoalStatusStrip.tsx b/packages/web-shell/client/components/GoalStatusStrip.tsx index d65e6ae80d3..0d414017ae4 100644 --- a/packages/web-shell/client/components/GoalStatusStrip.tsx +++ b/packages/web-shell/client/components/GoalStatusStrip.tsx @@ -1,5 +1,8 @@ import { useEffect, useState } from 'react'; -import type { GoalSnapshotV2 } from '@qwen-code/sdk/daemon'; +import { + GOAL_CHECKPOINT_STALL_LIMIT, + type GoalSnapshotV2, +} from '@qwen-code/sdk/daemon'; import { Pause, Pencil, Play, Target, Trash2 } from 'lucide-react'; import { useI18n } from '../i18n'; import { formatRuntime } from '../utils/formatRuntime'; @@ -67,6 +70,16 @@ export function GoalStatusStrip({ const canPause = goal.status === 'active'; const canResume = canResumeGoal(goal); const tokenLabel = getGoalTokenLabel(goal, t); + // A stall streak shows here whatever the status, where a daemon-session user + // is already looking -- including on a Goal the breaker stopped, which the + // terminal footer pill labels by its status instead. The failure text itself + // is left to the Goals dialog, which has room for it. + const checkpointStalls = goal.checkpointStalls ?? 0; + // Kept as the tooltip too: on a narrow pane the label is ellipsized. + const checkpointLabel = t('goal.checkpointStalled', { + count: checkpointStalls, + limit: GOAL_CHECKPOINT_STALL_LIMIT, + }); return (
) : null} + {checkpointStalls > 0 ? ( + <> + + + {checkpointLabel} + + + ) : null}