Skip to content

fix(bedrock): subagent transport routing + retry transient streaming faults - #43916

Closed
gutosantos82 wants to merge 2 commits into
NousResearch:mainfrom
gutosantos82:fix/bedrock-stream-transient-retryable
Closed

fix(bedrock): subagent transport routing + retry transient streaming faults#43916
gutosantos82 wants to merge 2 commits into
NousResearch:mainfrom
gutosantos82:fix/bedrock-stream-transient-retryable

Conversation

@gutosantos82

@gutosantos82 gutosantos82 commented Jun 11, 2026

Copy link
Copy Markdown

Closes #43915.

Two related Bedrock fixes surfaced while running global.anthropic.claude-fable-5 with delegated subagents. The main agent worked; every delegate_task subagent died with a Bedrock 400.


Fix 1 — Subagent transport routing (the actual cause of the failure)

A delegated subagent whose runtime resolves to a Bedrock base_url (bedrock-runtime.<region>.amazonaws.com) was routed through the chat_completions transport, because _detect_api_mode_for_url() recognized OpenAI/xAI/Anthropic endpoints but not Bedrock — so it fell through to the default.

The subagent then POSTed an OpenAI-shaped body to bedrock-runtime/.../chat/completions with Authorization: Bearer None (no SigV4). Bedrock can't process this and returns a persistent internalServerException 400 — failing identically across all retries.

url: https://bedrock-runtime.us-east-1.amazonaws.com/chat/completions
Authorization: Bearer None      ← wrong protocol + no auth

Fix:

  • runtime_provider._detect_api_mode_for_url() returns bedrock_converse for Bedrock runtime URLs, mirroring the main agent's auto-detection in agent/agent_init.py. Includes a suffix-spoof guard (…amazonaws.com.evil.test → not matched).
  • delegate_tool: add bedrock_converse to the explicit api_mode override allowlist.

Fix 2 — Retry transient streaming faults

The Anthropic SDK's Bedrock event-stream decoder (anthropic/lib/bedrock/_stream_decoder.py:58) raises a bare ValueError("Bad response code, expected 200: {...}") for non-200 event frames, dropping the HTTP status (agent sees HTTP None). When the embedded :exception-type is a transient fault (internalServerException, modelStreamErrorException, throttlingException, serviceUnavailableException, modelTimeoutException), the is_local_validation_error predicate misclassified it as a non-retryable local bug.

Fix: narrow carve-out so those transient types are retryable, while a genuine validationException still aborts. Mirrors the existing JSONDecodeError (#14782) and NoneType-not-iterable (#33136) carve-outs.


Tests

$ pytest tests/hermes_cli/test_detect_api_mode_for_url.py \
         tests/run_agent/test_bedrock_stream_transient_retryable.py -q
29 passed
  • 5 new detection tests (region variants, uppercase, lookalike host, path-segment spoof)
  • 8 retry-classification tests (transient types retryable; validationException + bare ValueError still abort)
  • sibling test_jsondecodeerror_retryable.py unchanged (9 passed)

… bugs

The Anthropic SDK's Bedrock event-stream decoder
(anthropic/lib/bedrock/_stream_decoder.py) raises a bare
ValueError("Bad response code, expected 200: {...}") when a streamed
event frame carries a non-200 status. The embedded :exception-type is
frequently a TRANSIENT server fault (internalServerException,
modelStreamErrorException, throttlingException, serviceUnavailableException,
modelTimeoutException) that AWS explicitly tells callers to retry.

Because the SDK raises a bare ValueError, the HTTP status is lost (the agent
sees "HTTP None") and the conversation loop's is_local_validation_error
predicate treats every ValueError/TypeError as a local programming bug,
aborting the turn non-retryably. Observed in practice on
global.anthropic.claude-fable-5 where a single transient internalServerException
killed a delegated subagent task; the identical request succeeded on retry.

Add a narrow carve-out: a ValueError matching the decoder's message AND
carrying a known transient exception-type is excluded from the local-bug
classification so the existing retry/fallback path runs. A genuine
validationException (real client bug) is intentionally NOT excluded and still
aborts. Mirrors the existing JSONDecodeError (NousResearch#14782) and NoneType-not-iterable
(NousResearch#33136) carve-outs.

Adds a regression test covering both the transient types (retryable) and
validationException + bare ValueError (still non-retryable).
@liuhao1024

Copy link
Copy Markdown
Contributor

Verification review — Bedrock transient streaming fault carve-out is correct and well-tested.

Reviewed the diff in agent/conversation_loop.py and the new test file. Key observations:

  1. The carve-out is surgically narrow: it only exempts ValueError instances whose message contains "bad response code, expected 200" AND one of the 5 known transient AWS exception types. A bare ValueError (programming bug) or a validationException (real client error) still classifies as non-retryable — correct behavior.

  2. The predicate structure mirrors existing SSL/JSONDecodeError carve-outs — consistent pattern in the same if chain.

  3. TestAgentLoopSourceStillHasCarveOut is a belt-and-suspenders guard: it reads the production source via inspect.getsource() and asserts the carve-out string is present. Protects against accidental reverts that leave the test file intact.

  4. Test _mirror_agent_predicate duplicates the exact predicate shape from conversation_loop.py. The docstring explicitly warns "If you change one, change both" — good maintenance signal.

  5. 5 transient types covered: internalServerException, modelStreamErrorException, throttlingException, serviceUnavailableException, modelTimeoutException. These match the AWS Bedrock documented retryable exceptions.

Clean merge candidate. No CI checks yet (just submitted), but the logic is straightforward and the test coverage is thorough.

@alt-glitch alt-glitch added type/bug Something isn't working comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint provider/bedrock AWS Bedrock (boto3, IAM) P2 Medium — degraded but workaround exists labels Jun 11, 2026
…ions

A delegated subagent whose runtime resolves to a Bedrock base_url
(bedrock-runtime.<region>.amazonaws.com) was sent through the
chat_completions transport: _detect_api_mode_for_url() recognized OpenAI,
xAI, and Anthropic endpoints but not Bedrock, so it fell through to the
default. The subagent then POSTed an OpenAI-shaped body to
bedrock-runtime/.../chat/completions with 'Bearer None' (no SigV4), which
Bedrock cannot process — surfacing as a PERSISTENT internalServerException
400 that failed identically across all retries (unlike the transient faults
addressed in the first commit).

Observed on global.anthropic.claude-fable-5: the main agent worked (it uses
native Converse + boto3 credentials) but every delegate_task subagent died.

Fix:
- runtime_provider._detect_api_mode_for_url() now returns 'bedrock_converse'
  for bedrock-runtime.<region>.amazonaws.com URLs, mirroring the main agent's
  auto-detection in agent/agent_init.py. Includes a suffix-spoof guard so a
  lookalike host (…amazonaws.com.evil.test) is NOT matched.
- delegate_tool: add 'bedrock_converse' to the explicit api_mode override
  allowlist so users can force it for non-standard endpoints too.

Adds 5 detection tests (region variants, uppercase, lookalike host, path-segment
spoof).
@gutosantos82 gutosantos82 changed the title fix(bedrock): retry transient streaming faults misclassified as local bugs fix(bedrock): subagent transport routing + retry transient streaming faults Jun 11, 2026
@gutosantos82

Copy link
Copy Markdown
Author

Closing this PR. After deeper investigation, the root cause of the original symptom turned out to be AWS Bedrock-side streaming instability for newly-launched models, not a hermes bug.

Findings:

  • The identical request succeeds ~100% via non-streaming messages.create() but fails ~50% via messages.stream() on global.anthropic.claude-fable-5 and global.anthropic.claude-opus-4-8 (both launched within the last ~2 weeks).
  • The failure is non-deterministic and the "break point" in the system prompt shifted between runs — i.e. not payload-specific.
  • Established models on the same account/region/transport stream reliably: Opus 4.7 (18/18), Opus 4.5 (5/5), Sonnet 4.6 (5/5).
  • The bedrock-runtime/.../chat/completions + Bearer None seen in the request debug dump was just the dumper's generic rendering of an AnthropicBedrock SDK call — Claude-on-Bedrock correctly routes through anthropic_messages (AnthropicBedrock SDK) via runtime_provider.py, not the OpenAI chat-completions transport. So commit 2 (subagent routing) addressed a non-issue.

Re the two commits:

  • Commit 1 (retry transient streaming ValueErrors) is arguably still a reasonable hardening — it correctly retries internalServerException/throttlingException/etc. instead of aborting as a local bug. If a maintainer thinks it's worth having on its own, happy to re-open a focused PR for just that.
  • Commit 2 (subagent Converse routing) was based on a misdiagnosis and isn't needed.

Workaround for users hitting this: route subagents (or the session) to an established model until AWS scales streaming capacity for the newest releases — e.g. delegation.model: global.anthropic.claude-opus-4-7.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P2 Medium — degraded but workaround exists provider/bedrock AWS Bedrock (boto3, IAM) type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Bedrock streaming transient faults (internalServerException) abort turn non-retryably

3 participants