fix(agent): reword SKILLS_GUIDANCE trigger and stop mislabelling its 400 as billing

[jackulau]

What does this PR do?

On an Anthropic subscription OAuth credential, every request failed with HTTP 400 "You're out of extra usage. Add more at claude.ai/settings/usage and keep going." That is not a billing condition — Anthropic's server-side content filter rejects the first sentence of Hermes' own built-in SKILLS_GUIDANCE prompt, and the rejection is surfaced with a billing-shaped message. Because the message points at the usage settings page, it reliably sends people to buy quota they don't need; the reporter lost three debugging sessions to it before capturing the actual request.

The issue's bisect (live API, real 71,721-char assembled prompt, claude-opus-5, stable across repeats): the first SKILLS_GUIDANCE sentence alone reproduces the 400 and removing it alone clears it. Size was ruled out (20 KB of unrelated filler → 200) and so was the system[0] identity gate (that returns 429, a different failure).

Three changes, all serving one outcome — a subscription user can no longer be misdirected by this 400:

1. Reword the trigger (agent/prompt_builder.py) — to the phrasing the reporter verified returns 200. Meaning, the skill_manage reference, and the ## Skill Safety Rule block are preserved. The reword is empirically validated rather than understood, so a comment records the bisect and warns that any rewrite must be re-verified against an OAuth token, not an sk-ant-api… key (which doesn't hit the filter).

2. Stop asserting exhaustion as fact (agent/conversation_loop.py) — this is the durable half. The reword chases a moving server-side target; the misleading message is what cost the debugging sessions, and it will keep costing them the next time some other prose trips the classifier. The Anthropic branch of _billing_or_entitlement_message now hedges its opening line, names the content-filter alternative, and gives the operator a discriminator (if the usage page still shows quota, suspect a content rejection). It also points at hermes auth reset anthropic, because of the reporter's observation 1 below.

3. Document the setup-token confusion (hermes_cli/auth.py) — the issue's observation 2.

On the issue's two related observations

Observation 1 (cached exhaustion latch replays errors). Addressed as guidance, not as a mechanism change. After a failure the credential is latched exhausted for ~60 min and the stored error is replayed without issuing a request, so a real fix looks like it didn't work — exactly the trap that made this bug so expensive to diagnose. Actually suppressing or labelling the replay at its source means touching the credential-pool cooldown machinery (_exhausted_until / STATUS_EXHAUSTED in agent/credential_pool.py), which is shared across every provider and well outside this fix's blast radius. I deliberately left that alone and instead surfaced the reset command at the moment the user reads the confusing error. If maintainers want the replayed error tagged as cached at the source, that's a clean follow-up and I'm happy to open it.

Observation 2 (claude setup-token is not an API key). The issue suggests "a docstring note, or dropping that var from the tuple." Dropping it would be a regression — api_key_env_vars doubles as the credential-discovery list (agent/credential_pool.py builds its env scan from pconfig.api_key_env_vars), so removing CLAUDE_CODE_OAUTH_TOKEN would stop Hermes finding a setup-token credential at all. Took the docstring option and wrote down why the entry has to stay.

Relationship to open PR #78025 — please read before merging

#78025 (open, needs-decision) already contains this same SKILLS_GUIDANCE reword as its "layer 2", alongside a much larger wire-level tool-name aliasing change (session_search → chat_history_lookup). It targets a different issue (#65365), whose bisect found a three-sentence AND-condition rather than the single-sentence trigger reported here.

I am not trying to race it. This PR is deliberately the small, surgical version of #82154: the prompt reword plus the diagnosability fixes that no open PR covers. If #78025 lands first, the prompt_builder.py hunk here becomes redundant and I'll happily rebase it out — items 2 and 3 stand on their own either way. If maintainers would rather fold everything into #78025, close this and I'll port items 2 and 3 across.

Related open PRs in the same area, none of which touch the message or the registry note: #76807, #80854, #72173, #80618.

Related Issue

Fixes #82154

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)
• ✨ New feature (non-breaking change that adds functionality)
• 🔒 Security fix
• 📝 Documentation update
• ✅ Tests (adding or improving test coverage)
• ♻️ Refactor (no behavior change)
• 🎯 New skill (bundled or hub)

Changes Made

• agent/prompt_builder.py — reword the SKILLS_GUIDANCE first sentence; comment records the bisect and the re-verification requirement.
• agent/conversation_loop.py — _billing_or_entitlement_message Anthropic branch: hedge is exhausted → may be exhausted, add the content-filter alternative with a discriminator, add the hermes auth reset anthropic note for the cached-replay trap. Additive — every prior line is retained.
• hermes_cli/auth.py — comment on the anthropic ProviderConfig explaining that CLAUDE_CODE_OAUTH_TOKEN is an OAuth token (sk-ant-oat01…: 401s as x-api-key, 429s as bare Bearer), that only ANTHROPIC_API_KEY/ANTHROPIC_TOKEN are literal API keys, and why the entry must stay for discovery.
• tests/agent/test_skills_guidance_content_filter.py — new, 11 tests: rejected fragments absent (parametrized), the verified reword pinned, behaviour preserved, Safety Rule intact, newline count held above the bound test_ghost_skill_pruning.py asserts, and the constant still wired into the system prompt.
• tests/agent/test_anthropic_billing_guidance.py — 5 tests: content-filter alternative named, exhaustion not asserted as fact, cached-replay warning present, original billing remedies retained, and the Anthropic-specific lore not leaking to other providers.
• website/docs/user-guide/features/skills.md — "When the Agent Creates Skills" quoted the old prompt verbatim (After completing a complex task (5+ tool calls)); updated so the docs don't drift from the reworded prompt.

The tests deliberately assert on the trigger substrings rather than only on an exact replacement string, so a future reword is free to change the prose as long as it doesn't reintroduce the rejected phrasing.

How to Test

The root cause only reproduces against the live API with a subscription OAuth credential:

# With ~/.claude/.credentials.json present (or CLAUDE_CODE_OAUTH_TOKEN set to a
# `claude setup-token` value) and provider: anthropic configured:
hermes auth reset anthropic     # clear any cached exhaustion latch first
hermes chat -q "Hello"
# Before: HTTP 400 "You're out of extra usage..." on every request.
# After:  the request succeeds.

Automated:

pytest tests/agent/test_skills_guidance_content_filter.py -q
pytest tests/agent/test_anthropic_billing_guidance.py tests/agent/test_ghost_skill_pruning.py \
       tests/agent/test_error_classifier.py -q
python scripts/check-windows-footguns.py agent/prompt_builder.py agent/conversation_loop.py hermes_cli/auth.py

To see the new guidance text without an Anthropic credential:

from agent.conversation_loop import _billing_or_entitlement_message
print(_billing_or_entitlement_message(capability="model access", provider="anthropic",
                                      base_url="https://api.anthropic.com", model="claude-opus-5"))

Checklist

Code

• I've read the Contributing Guide
• My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
• I searched for existing PRs to make sure this isn't a duplicate — see the fix(anthropic): alias session_search/memory OAuth billing-classifier triggers (fixes #65365, #82154) #78025 section above, which I'd ask a maintainer to read rather than treat this as an independent PR
• My PR contains only changes related to this fix/feature (no unrelated commits)
• I've run pytest tests/ -q and all tests pass — see note below
• I've added tests for my changes (required for bug fixes, strongly encouraged for features)
• I've tested on my platform: Windows 11 (Python 3.13)

Note on the test checkbox, stated rather than ticked. I developed this on Windows and cannot reproduce the root cause here — it needs a live Anthropic subscription OAuth token, so the end-to-end fix is verified by the issue reporter's bisect, not by me. What I did verify locally:

• The 16 new tests pass.
• test_anthropic_billing_guidance.py, test_ghost_skill_pruning.py, test_error_classifier.py pass: 87 tests on upstream/main in a clean worktree, and 103 with this branch — exactly 87 + the 16 new ones. No pre-existing test changed outcome.
• tests/agent/ as a whole has ~143 Windows-environment failures on both this branch and upstream/main (test_title_generator, test_model_metadata, test_models_dev, test_verification_*, path-separator assertions in test_system_prompt / test_prompt_builder, …). None are in the subsystems this PR touches. I could not get a full clean before/after diff of that suite inside my time budget, which is why I scoped the comparison to the three suites that actually cover the changed code and ran those against a pristine upstream/main checkout.

I also grepped the tree for any other assertion on the old sentence or the old billing string — the only hits were the docs file (updated here) and a user-story quote in website/src/data/userStories.json, which is quoted testimony and deliberately left alone.

CI on Linux/macOS is the real check here, and a reviewer with a subscription token running the reproduction above would be worth more than any of my local runs.

Documentation & Housekeeping

• I've updated relevant documentation — website/docs/user-guide/features/skills.md quoted the old prompt verbatim and would otherwise have drifted
• N/A — no config keys added or changed
• N/A — no architecture or workflow change
• I've considered cross-platform impact — prompt/message/comment text only, no OS-touching code; scripts/check-windows-footguns.py clean on the diff
• N/A — no tool behavior or schema changed

Screenshots / Logs

The guidance a subscription user now sees on that 400:

anthropic reported that your Claude subscription usage may be exhausted for claude-opus-5 (included quota + extra-usage credits).
Options: wait for the billing cycle to reset, or add extra usage at https://claude.ai/settings/usage
You can also switch to an Anthropic API key or another provider with /model <model> --provider <provider>.
If https://claude.ai/settings/usage still shows quota remaining, this is probably NOT a billing problem: on a Claude subscription (OAuth) token Anthropic returns this same message when its content filter rejects part of the request — typically a phrase in the system prompt.
Retry with a fresh credential state: `hermes auth reset anthropic`. Until that cooldown clears, this error can be replayed from cache without contacting the API.

The first three lines are unchanged apart from is → may be; the last two are new.

[alt-glitch]

This was generated by AI during triage.

Related: #78025 addresses the same Anthropic OAuth content-filter/billing-classifier failure with a materially broader alternate repair; this PR directly implements the isolated trigger reword and diagnosis fix for #82154.

[jackulau]

Not a duplicate of #78025 — complementary, different failure mode:

	This PR (#82177 / #82154)	#78025 / #65365
Trigger	First sentence of built-in SKILLS_GUIDANCE content-filter 400	session_search / memory tool-schema names misrouted by OAuth billing classifier
Fix	Reword the verified trigger phrase; stop mislabelling the 400 as billing exhaustion	Alias tool schema names/descriptions for OAuth
Surfaces	prompt_builder.SKILLS_GUIDANCE, conversation-loop billing guidance, auth messaging	anthropic_adapter, transport aliasing

Both can land independently. Rebased onto current main; targeted tests green (18 passed).

[egilewski]

suggesting changes

The exact Anthropic 400 is still classified, persisted, and surfaced as billing; this patch only adds a caveat after that decision. agent/error_classifier.py:118 still puts out of extra usage in _BILLING_PATTERNS, and :1437-1442 therefore returns FailoverReason.billing with credential rotation. That keeps the full billing cooldown and still drives Billing or credits exhausted in the fallback and terminal paths at agent/conversation_loop.py:4856-4859, :5685-5697, :5740-5742, and :5860-5866, including failure_reason=billing and billing_block.

I reproduced the exact 400 on current main and on this head replayed onto current main: both returned reason=billing, retryable=False, rotation enabled, and a 3,600-second sole-credential cooldown. The new guidance caveat appeared on the PR tree, but the unhedged terminal label and structured billing result remained. The focused tests pass because test_400_anthropic_extra_usage_exhausted still explicitly requires FailoverReason.billing.

Please carry the ambiguity through the classifier, persisted credential state, fallback/terminal labels, and structured result, and add a regression for the returned terminal response. Otherwise a content-filter rejection still tells every downstream surface that billing is exhausted and benches the credential as such.

Security evidence:

• trust boundary: Anthropic's ambiguous HTTP 400 enters Hermes error classification, credential-pool state, and user-visible result surfaces.
• source/sink/invariant: out of extra usage is the source; billing classification/cooldown/status/result are the sinks; a content-filter rejection must not be asserted as billing exhaustion.
• current-main reproduction: the exact body classified as billing, non-retryable, rotating, and subject to a 3,600-second billing cooldown.
• PR-head or patch-replay validation: the head merged cleanly onto current main; only the explanatory guidance changed while classification, cooldown, and terminal labels remained billing-specific.
• positive/negative cases: the new content-filter caveat is present, but the exact ambiguous response still yields failure_reason=billing and an unhedged terminal prefix.
• residual bypass search: both terminal-result paths and the configured-fallback status retain the billing assertion.
• reviewer validation: 19 focused tests and the changed-file Windows footgun check pass; the retained exact-400 classifier test confirms the unchanged billing verdict.

Not checked:

• Live Anthropic OAuth request
• Full test suite
• CodeRabbit review

Signed: GPT-5.6-sol-xhigh in Codex

[jackulau]

Addressed in 42ca5eb (rebased onto current main). The ambiguity is now carried through every layer the review named, not just the guidance caveat:

Classifier (agent/error_classifier.py): an 'out of extra usage' match on the 400 path and the status-less path attaches error_context={billing_unverified: true, possible_content_filter: true}, exposed as ClassifiedError.billing_unverified. Reason stays FailoverReason.billing because rotation + fallback remain the right recovery for both readings of the body; every surface that asserts anything now checks the flag. An unambiguous billing body ('credit balance is too low') keeps a confirmed verdict; there is a negative test pinning that.

Persisted credential state (agent/credential_pool.py, agent_runtime_helpers.py): the pool now receives and persists failure_reason='billing_unverified' for this error. The exhausted entry gets the short transient cooldown (60s) instead of the 3,600-second bench, regardless of pool size, and the marker survives reload so a restart cannot upgrade it back to a bench. A true 402 keeps the full bench even if mislabelled. This also removes the hour-long sole-credential latch that replayed the stored error without contacting the API.

Fallback and terminal labels (agent/conversation_loop.py): the fallback-switch status, max-retries status, and both terminal returns use a hedged label ('Provider reported usage/credit exhaustion (unverified: may be a content-filter rejection, not billing)') when the flag is set. 'Billing or credits exhausted' is now only emitted for confirmed verdicts, and the confirmed Anthropic guidance regained its assertive wording so the caveat no longer dilutes a genuine depletion.

Structured result: both terminal result dicts now carry billing_unverified: true/false and the billing_block gains unverified: true. failure_reason stays 'billing' deliberately so existing exit-code consumers (kanban worker path in cli.py) keep matching; the ambiguity rides in the dedicated field.

Regression for the returned terminal response: terminal-response construction was factored into _billing_failure_result (used by both the non-retryable abort and max-retries paths) and tests/agent/test_billing_unverified_carrythrough.py pins that the exact ambiguous 400 produces a hedged final_response, billing_unverified: true, and an annotated billing_block, while a confirmed body keeps the original label with no flag. Also updated test_400_anthropic_extra_usage_exhausted to require the unverified marking, plus cooldown TTL / persistence round-trip / pool failure_reason plumbing tests.

Verification: 110 targeted tests green (classifier, guidance, carry-through, sole-cooldown, skills-guidance), and the changed-file Windows footgun check passes. One note: test_credential_pool_routing.py::TestFailureAttribution::test_unmatched_key_does_not_retry_only_pool_entry fails identically on current main with this head's changes stashed, so it is pre-existing and unrelated (Windows run).

[teknium1]

Merged via PR #86677. Both commits were cherry-picked onto current main with your authorship preserved in git log — thank you, especially for carrying the billing-ambiguity flag through every surface rather than just the guidance text.