fix(anthropic): prepended leading user turn must be non-whitespace (#70909)

[SHL0MS]

Summary

Fixes #70909. A session wedges behind HTTP 400: messages: text content blocks must contain non-whitespace text — endless "thinking", no reply, every retry replaying the same 400.

Root cause (proven, self-inflicted): _ensure_leading_user_turn prepends a user turn when messages[0] is not a user message — which is the normal shape after context compaction emits an assistant summary first. The placeholder it inserted was a bare single space:

result.insert(0, {"role": "user", "content": [{"type": "text", "text": " "}]})

A space-only text block is exactly what the Messages API rejects, so the guard added for #52160 trips this 400 itself. Bedrock's equivalent prepend already uses the shared non-whitespace placeholder (bedrock_adapter.py), and this function's docstring claims to mirror it, but it diverged on the one detail that matters.

Verified against a real wedged history: the assembled request carried exactly one blank text block — at index 0, the prepended turn. Stored rows were clean (no blank content/api_content/reasoning*; every stored assistant row run through _convert_assistant_message produced zero blanks), which is what pointed at assembly-time synthesis. With this change the same history assembles with zero blank blocks.

Changes

1. Root-cause fix — _ensure_leading_user_turn uses _EMPTY_TEXT_PLACEHOLDER instead of " ", matching Bedrock.
2. Backstop — _coerce_blank_text_blocks() at the end of convert_messages_to_anthropic walks the fully-assembled request (system prompt + every message) and coerces any remaining blank text block. Independently justified: with the backstop disabled the root-cause fix handles the leading-turn case, but two other real paths still leak blanks — content that arrives pre-shaped as blocks with a blank text part, and a blank cache_control system block (neither is walked by the per-message converters). Only text blocks are touched; thinking/tool_use/image obey different schema rules.
3. Test invariants — the two bug(adapter): HTTP 400 after double context compression — first message is assistant, not user #52160 tests asserted the frozen buggy literal (assert content == [{"type": "text", "text": " "}]), which would have blocked this fix. Rewritten to assert the invariant: one text block whose text is non-whitespace.

Testing

pytest tests/agent/test_anthropic_request_blank_block_guard.py -q      # 5 passed
pytest tests/agent/test_anthropic_whitespace_text_blocks.py \
       tests/agent/test_anthropic_thinking_block_order.py \
       tests/agent/test_anthropic_adapter.py -q                        # 204 passed

Each layer verified independently: disabling the backstop leaves the leading-turn test passing (root cause genuinely fixed) while the pre-shaped-block and system-block tests fail (backstop not redundant). The 3 TestRunOauthSetupToken failures in that file are pre-existing on clean main (MagicMock/JSON in credential mocking), unrelated to this change.

---

Root cause identified and both layers verified against a real wedged history.

[AIalliAI]

Ran this against current main (0157180) — the diff still applies cleanly despite the stale base. Three of the five new guard tests fail on clean main (the leading-turn block, the pre-shaped blank block, and the blank cache_control system block) and all five pass with the patch; the surrounding Anthropic suites came back identical to a clean-main baseline, so no regression. For reviewers comparing the overlapping PRs (#70081, #70159, #71935): those cover the root-cause hunk only — the two request-assembly cases above fail on main today and are only handled here.

[GottZ]

This was generated by AI during triage.

Summary

Four PRs address the same assembly-time Anthropic whitespace failure. #70081, #70159, and #71935 only replace the invalid synthetic leading-user placeholder, while #70910 uses the canonical sentinel at that source and additionally guards pre-shaped message blocks and cache-controlled system blocks in the fully assembled request.

Related pull requests

• fix(anthropic): synthesized leading user turn must be non-whitespace #70081 fixes — (+46/-3) — partial fix: Replaces the whitespace-only synthesized leading-user block with printable text and tests the non-whitespace invariant, but hard-codes "(empty)" instead of the requested _EMPTY_TEXT_PLACEHOLDER. Despite the keep_open review on fix(anthropic): synthesized leading user turn must be non-whitespace #70081, fix(anthropic): prepended leading user turn must be non-whitespace (#70909) #70910 contains the same source fix using the canonical constant and covers additional assembly-time blank-block paths, leaving fix(anthropic): synthesized leading user turn must be non-whitespace #70081 as a duplicate.
• fix(anthropic): non-whitespace placeholder for synthetic leading user turn #70159 fixes — (+8/-3) — partial fix: Replaces the same invalid leading-user placeholder with "." and updates existing tests to assert non-whitespace behavior, but bypasses the repository's canonical placeholder. Despite the keep_open review on fix(anthropic): non-whitespace placeholder for synthetic leading user turn #70159, fix(anthropic): prepended leading user turn must be non-whitespace (#70909) #70910 fixes the identical site with _EMPTY_TEXT_PLACEHOLDER and adds tested protection for blank pre-shaped and system blocks, so fix(anthropic): non-whitespace placeholder for synthetic leading user turn #70159 is a duplicate.
• fix(anthropic): prepended leading user turn must be non-whitespace (#70909) #70910 best fix — (+165/-3) — best fix: Uses _EMPTY_TEXT_PLACEHOLDER at the contributor-confirmed synthesis site and uniquely adds a final text-block-only guard covering the reproduced pre-shaped-content and cache-controlled-system cases. Keep it open with a salvage path that preserves both the root-cause hunk and focused request-level regressions while reconciling the tests with current main.
• fix(anthropic): non-whitespace placeholder for synthesized leading user turn #71935 fixes — (+48/-3) — partial fix: Correctly replaces the synthesized leading-user space with _EMPTY_TEXT_PLACEHOLDER and adds direct regression coverage, but does not protect the fully assembled request from the additional blank-block paths covered by fix(anthropic): prepended leading user turn must be non-whitespace (#70909) #70910. Despite the MAINTAINER-BOT keep_open verdict on fix(anthropic): non-whitespace placeholder for synthesized leading user turn #71935, fix(anthropic): prepended leading user turn must be non-whitespace (#70909) #70910 contains the same source fix plus tested request-level protection, leaving no distinct implementation in fix(anthropic): non-whitespace placeholder for synthesized leading user turn #71935 to salvage.

Duplicates

#70081, #70159, and #71935 duplicate the synthesized-leading-user fix contained in #70910; #70081 and #71935 are closest at the source level, while #70159 differs primarily by introducing a new "." literal.

Suggested consolidation

Keep #70910 open with a salvage path: retain its canonical root-cause fix, final text-block-only request guard, and focused regressions, rebasing or reconciling the test additions against current main as needed. Close #70081, #70159, and #71935 as duplicates of #70910 because their diffs cover only the root-cause subset and add no independent behavior beyond the recorded best fix.

Complex graph

flowchart LR
    classDef open fill:#dbeafe,stroke:#1d4ed8,color:#1e3a8a
    classDef merged fill:#dcfce7,stroke:#15803d,color:#14532d
    classDef closed fill:#e5e7eb,stroke:#6b7280,color:#1f2937
    classDef unverified fill:#f3f4f6,stroke:#9ca3af,color:#374151
    classDef best stroke-width:3px,stroke:#b45309
    classDef target stroke-width:3px,stroke:#4338ca
    I70909(["issue #70909 (open)"])
    P70910["PR #70910 (open)"]
    P70910 -->|best fix| I70909
    class I70909 open
    class P70910 open
    class P70910 best
    class P70910 target
    click I70909 "https://github.com/NousResearch/hermes-agent/issues/70909"
    click P70910 "https://github.com/NousResearch/hermes-agent/pull/70910"

Loading

Graph: solid arrow = fixes / best fix, dashed arrow = partial or unverified (see edge label); boxed group = PRs duplicating each other; amber border = best fix; indigo border = target; gray node = closed (state tag in the node label).

Cross-PR triage: Reviewed 4 pull requests and 1 issue in this complex. Each diff was read against this issue; Assessment working set: 21 kB of PR diffs, 13 kB of issue/PR text, 4 kB of discussion (8 comments), 8 verify verdicts. verdicts reflect diff content, not PR titles. Part of an automated triage batch.

[Benny-Gil]

Independent confirmation of this root cause from a production deployment, in case it helps move this along — I hit it in the wild before finding this PR, diagnosed it from the other end, and landed on exactly the block you identify here.

Deployment: Hermes Agent v0.19.1 (2026.7.30) · upstream 470cf66b, docker, Debian 13, Python 3.13.5. ~8 uid-isolated tenants on one host, Telegram adapter, Anthropic-compatible upstream, claude-sonnet-5.

What matched your diagnosis exactly. From sessions/request_dump_<session>_<ts>.json on a failing turn:

• 3 messages in the request body
• messages[0]: role user, content a list with one {"type": "text"} block whose text is a single whitespace character

That is _ensure_leading_user_turn's placeholder, verbatim.

Two details that support "synthesized, not stored", if you need more evidence for review:

1. The block exists nowhere in state.db. I checked every messages row for that session — ordinary short user messages, no empty content, no whitespace-only content, nothing malformed. I initially assumed corrupted history and went looking there first; there is nothing to find, which is what makes this expensive to diagnose from the error alone.

2. The session's stored system_prompt was NULL while sibling sessions on the same host held 15–19 KB, with this logged every turn:

   WARNING agent.conversation_loop: Stored system prompt for session <id> is null;
     rebuilding from scratch this turn. Prefix cache will miss until the rebuild persists.
     Investigate the previous turn's update.
   

   Consistent with your "normal shape after context compaction emits an assistant summary first" — the affected session had been compacted, the siblings had not.

Blast radius, since it may affect priority. It is permanent, not transient. 28 identical failures over ~4 hours; the agent never answered again on its own. Seven sibling tenants on the same image, host, upstream and model were unaffected throughout, so it is per-session rather than environmental.

Two things make it worse for end users:

• The turn is retried (API call failed (attempt 1/3)) on a non-retryable 400. Retrying replays the identical synthesized block, so it cannot succeed — it just spends the user's time before failing.
• All the end user sees is The model provider failed after retries., which is indistinguishable from the provider being down. The operator-visible cause is only in gateway.log.

Workaround for anyone stuck before this merges: delete the session's entry from sessions/sessions.json (the session_key → session_id map). The next inbound message starts a fresh session; state.db is untouched so history survives and stays searchable. Back the file up first. This recovered the affected agent for me.

I filed #83037 before searching properly — closing it as a duplicate of #70909 and pointing at this PR. Happy to share the sanitized request dump privately with a maintainer if a second reproduction is useful for review; it contains end-user conversation content so I have not attached it.

[egilewski]

Seems to have been fixed by #77509

[teknium1]

Merged via #86643 with your authorship preserved (rebase merge). On current main, #77509 had already landed the request-level scrub, so 4 of your 5 regression cases passed — the salvage carried the remaining residual: a whitespace-only system block with cache_control still reached the wire; it is now coerced to the shared non-whitespace placeholder at system extraction (the block can't be dropped without losing the cache breakpoint). Your full regression suite from this PR landed as tests/agent/test_anthropic_request_blank_block_guard.py. Thanks @SHL0MS for the forensic diagnosis, and thanks for the detailed production evidence in the thread — the wedged-session workaround documented there helped scope blast radius.