feat(agent): CostBudget - hard per-run USD spend ceiling

[aman-a-shah]

Why

2026-07-24 cost incident: OpenRouter burn peaked at $455/hr, with one chat turn burning ~$35 producing nothing. Every retry/iteration cap in the loop is bounded individually (API retries 3, compression 3, length-continues 3, empty-content 3, invalid-JSON 6, iterations via IterationBudget) — but their product had no dollar bound.

What

• CostBudget — thread-safe shared USD ceiling, mirroring IterationBudget: caller creates one, the parent agent and every delegate_task subagent share it, so delegation can't bypass it.
• Spend accumulates from estimate_usage_cost after each successful API call (unpriced calls add nothing — best-effort bound).
• Ceiling crossed → the loop stops making tool calls and produces a final no-tools summary of the work already paid for (same graceful exit as the iteration cap, never a hard error). A run that starts over-budget (recovery re-prompt on the same agent) returns a static message with zero API calls.
• Result dict gains cost_limited so callers can log/surface the stop.
• Opt-in: cost_budget=None default → zero behavior change for every existing caller.

Tests

tests/test_cost_budget.py — budget semantics (incl. thread-safety), normal completion under budget, mid-run breach → summary + cost_limited, pre-exhausted budget → zero API calls, unpriced calls don't trip, delegate child inherits the parent's budget. Also ran test_run_agent.py, test_agent_guardrails.py, test_1630_context_overflow_loop.py, test_413_compression.py, test_cli_interrupt_subagent.py — 250 passed, 12 skipped (baseline).

Consumed by the hermes PR that sets CHAT_MAX_COST_USD (default $5/turn) for interactive chat. Please use a merge commit (not squash) so the hermes submodule pin can point at a reachable SHA.

🤖 Generated with Claude Code

[aman-a-shah]

Self-review (adversarial pass, requested by Aman)

F1 (fixed in 6a5b8a3) — shared IterationBudget was silently broken. run_conversation unconditionally replaced the budget at turn start, and delegate children call run_conversation — so every child granted itself a fresh full budget and the documented shared cap never constrained delegation (pre-existing cost amplifier, found while verifying this PR's 'mirrors IterationBudget' claim). Now ownership-guarded: self-created budgets keep the per-turn refresh (CLI/gateway unchanged); externally-provided shared pools survive and children genuinely drain the parent's cap. Locked by TestIterationBudgetOwnership (3 tests).

F3 (fixed) — salvage fallback copy said 'iteration limit' on the cost path. Now reason-aware.

F4 (fixed) — the salvage summary call's own usage was invisible to session counters and the CostBudget. Now metered across all API-mode variants, best-effort, never breaks the summary path. Locked by TestCostLimitedSummaryPath (2 tests).

Suite: 16 cost-budget tests + 239 adjacent (run_agent / guardrails / 1630 / 413 / subagent-interrupt) all passing.

[aman-a-shah]

Code review — approve ✅

Verified the PR head (6a5b8a3) is byte-identical to the pinned submodule already inspected during the hermes NousResearch#100 review, so all four core mechanics are confirmed in place: loop-top check-before-every-call (run_agent.py:5760, before api_call_count/iteration consume, so a breach costs zero iterations), delegate inheritance (tools/delegate_tool.py:198,240), external-budget-not-reset (:5574), and the cost-reason summary exit (:5314+). Fresh scrutiny of the rest also came back clean: the main-loop metering piggybacks on the single normalized-usage block all three API modes funnel through; _meter_summary_usage covers all six salvage call sites; thread-safety holds (single lock, immutable max_usd, worst-case overshoot is one in-flight call — inherent to check-before-call); add() input hardening and the >= exceeded check are right. Branch is CLEAN/MERGEABLE and the PR body correctly carries the merge-commit (not squash) instruction.

Findings (all low severity)

1. Early-return result dicts omit cost_limited (run_agent.py:6137-6173 and ~15 similar sites) — only the main exit (:7350) carries it. The sole current consumer uses .get(), so nothing breaks today, but the result shape is inconsistent.
2. Side-channel LLM calls are unmetered (compression summaries context_compressor.py:342, memory flush run_agent.py:4551-4589) — pre-existing gap, individually bounded, consistent with the documented best-effort framing; "every spend path metered" holds for main loop + salvage + children only.
3. completed=True for cost-limited runs is deliberate but undocumented (:7300) — it correctly suppresses client-side whole-turn retries, and hermes gates recovery on cost_limited explicitly, but a caller inferring truncation from completed alone would be misled. Worth a comment.

Cross-PR note (from the #11 review): _meter_summary_usage adds the salvage call's spend to session counters and the CostBudget, but once #11's usage_listener merges, that call's delta never reaches the listener — the flusher's primary source. A forward-compatible fix (guarded getattr(self, "usage_listener", None)) is being pushed to this branch.

Tests: tests/test_cost_budget.py 16/16 pass locally; assertions are behavioral (exact call counts, injected summary prompt, salvage metering into both counters and the shared budget, 8-thread accumulation with exact-sum check, _owns_iteration_budget regression trio).

🤖 Generated with Claude Code

[aman-a-shah]

Review follow-up pushed (eb5ed0e)

Cross-PR composition fix (#10 × #11): _meter_summary_usage now fires the per-call usage_listener after metering the salvage summary call, so once #11 merges, that call's spend reaches the delta stream (hermes's UsageFlusher) — not just the session counters and CostBudget. The payload mirrors #11's main-loop delta field-for-field (input/output/total/reasoning/cache_read/cache_write_tokens, cost_usd, cost_status, cost_source, model, provider), and the call uses the same swallow-and-debug-log protection: a raising listener can never break the salvage flow. Since this branch has no usage_listener attribute/kwarg (that arrives with #11), it's fetched via getattr(self, "usage_listener", None) — a guaranteed no-op until both PRs land together, with no merge-order dependency.

Finding 3: added a comment at the completed computation documenting that cost-limited runs deliberately report completed=True (suppresses client-side whole-turn retries; consumers must check cost_limited), unlike iteration-capped runs.

Finding 1 (early-return dicts missing cost_limited): skipped as agreed — low value, high churn, sole consumer uses .get().

Tests: new TestSummaryUsageListener (2 tests) injects a mock listener directly on the agent object (works without #11's kwarg), drives the cost-limited salvage path, and asserts exactly one delta with the exact #11 key contract and the metered summary cost; plus a raising-listener test proving the salvage flow and metering survive. tests/test_cost_budget.py: 18/18 passing.

🤖 Generated with Claude Code

[GurneeshBudhiraja]

Code Review — PR #10: feat(agent): CostBudget — hard per-run USD spend ceiling

Overview

Adds an opt-in, thread-safe CostBudget that mirrors IterationBudget: the caller creates one USD ceiling, the parent agent and every delegate_task subagent share it, and spend accrues from estimate_usage_cost after each priced API call. When the ceiling is crossed, the loop stops issuing tool calls and falls into the existing salvage-summary path (_handle_max_iterations), returning cost_limited=True. Motivated by the 2026-07-24 incident where individually-bounded caps multiplied into ~$35/turn of unbounded burn.

I read the diff plus the head versions of run_agent.py, the summary/metering path, and delegate_tool.py. The change is careful, narrowly scoped, and genuinely well-tested. No correctness or security blockers found. Notes below are mostly design/coordination.

---

Findings

🟢 The salvage-summary call(s) are not themselves gated by the budget (run_agent.py _handle_max_iterations, ~L5311+)
Once the ceiling trips, the exit path makes one summary call — and up to a second on the retry branch — that are metered after the fact but never blocked, even when already far over budget. On a large context these are real input-token spend, so the effective ceiling can be overshot by ~1–2 large calls. This is documented ("one bounded no-tools call") and is the same graceful-exit contract as the iteration cap, so it's acceptable; flagging only because the whole feature exists to bound spend. Metering them (F4) is the right call.

🟢 completed=True for cost-limited runs is a consumer footgun (run_agent.py ~L7321–7358)
Deliberate and clearly documented (suppresses client-side whole-turn retries that would re-spend the budget), but it diverges from the iteration-cap path, which reports completed=False. Any caller that keys off completed alone — not cost_limited — will treat a truncated salvage summary (or the zero-call "stopped before it started" message) as a fully successful turn. The consuming hermes PR (CHAT_MAX_COST_USD) must check cost_limited; worth calling out to any other consumer of the result dict too.

🟢 No validation on max_usd (CostBudget.__init__, ~L35)
A 0 or negative ceiling makes exceeded true immediately (_spent_usd >= max_usd), silently forcing every run to zero API calls and a static "stopped before it started" reply. A misconfigured/negative CHAT_MAX_COST_USD would disable the agent rather than fail loudly. Consider a max_usd > 0 guard (or at least a warning).

🟢 Best-effort overshoot within a single iteration (design, documented)
The ceiling is only checked at the top of the loop, so a single iteration that fans out to many parallel subagent API calls can overshoot before the next check. Mitigated because each subagent re-checks the shared budget in its own loop, and it's explicitly documented as best-effort. No action needed.

🟢 Minor: logging.debug vs logger.debug mix inside _meter_summary_usage — the outer swallow uses module-level logging.debug, the listener guard uses logger.debug. Harmless, just inconsistent.

---

Positives worth noting

• F1 is a real pre-existing bug fix, not just scaffolding. run_conversation unconditionally rebuilt iteration_budget every turn, so delegate children each got a fresh full budget and the "shared cap" never constrained them. The ownership guard (_owns_iteration_budget) fixes that and is covered by TestIterationBudgetOwnership.
• Opt-in default (cost_budget=None) means zero behavior change for existing callers, and the top-of-loop check correctly catches both mid-run breach (summarize) and a pre-exhausted budget (zero API calls).
• Placement of the check — before api_call_count += 1 / iteration_budget.consume(), and re-evaluated after every continue (compression/length/empty-content retries) — is exactly right.
• Thread-safety is real (single lock, monotonic accumulator) and smoke-tested; unpriced calls correctly add nothing.
• Test coverage is strong: class semantics, mid-run breach → summary + cost_limited, pre-exhausted → zero calls, unpriced no-trip, salvage metering into session counters + budget, reason-aware fallback copy, delegate inheritance, and the #11 usage_listener cross-PR contract behind a guarded getattr.
• No secrets logged; the warning line logs only spend/max/api_calls/model. api_key reaches estimate_usage_cost via the same getattr(..., "") pattern as the existing main-loop call.

---

Verdict

Solid, defensive, well-tested fix that closes a genuine cost-safety gap and fixes a real pre-existing shared-budget bug along the way. No blockers. Suggest (1) a max_usd > 0 guard and (2) ensuring the hermes consumer keys off cost_limited rather than completed. The requested merge commit (not squash) is appropriate given the hermes submodule pin.