Skip to content

fix(moa): route bedrock MoA slots through signed bedrock branch - #54912

Closed
iizotov wants to merge 1 commit into
NousResearch:mainfrom
iizotov:fix/moa-bedrock-sigv4-aggregator
Closed

fix(moa): route bedrock MoA slots through signed bedrock branch#54912
iizotov wants to merge 1 commit into
NousResearch:mainfrom
iizotov:fix/moa-bedrock-sigv4-aggregator

Conversation

@iizotov

@iizotov iizotov commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Summary

MoA presets that use a Bedrock model as the aggregator (or as a reference) fail: the aggregator turn returns an empty/malformed ChatCompletion (choices=None), which trips the auxiliary response validator and aborts the turn.

Root cause

_slot_runtime() in agent/moa_loop.py resolves each MoA slot to its real runtime and, for most providers, forwards the resolved base_url + api_key to call_llm. A small name-preserve set (nous, openai-codex, xai-oauth) is exempted because their auth isn't a forwardable bearer token.

Bedrock was missing from that set. resolve_runtime_provider returns:

  • api_mode = anthropic_messages
  • base_url = https://bedrock-runtime.<region>.amazonaws.com
  • api_key = "aws-sdk" ← a placeholder, not a real token (auth is AWS SigV4 / IAM-role signing)

So _slot_runtime forwarded the bedrock-runtime URL plus the fake "aws-sdk" key. call_llm then treated it as a plain OpenAI-compatible endpoint and issued an unsigned bearer POST to the Bedrock runtime. Bedrock answered with an empty/malformed ChatCompletion, failing .choices[0].message validation.

References using real bearer tokens (e.g. openai-codex, custom xAI) survived because their auth works through the forwarded path; only Bedrock's SigV4 path broke.

Fix

Add bedrock to the name-preserve set so bedrock slots are passed by provider name only, routing through call_llm's dedicated SigV4-signed bedrock branch (the same branch the main agent already uses successfully).

if resolved_provider in {"nous", "openai-codex", "xai-oauth", "bedrock"}:
    return out

Verification

  • _slot_runtime({provider:"bedrock", ...}) now returns only ['provider','model'] — no leaked base_url / placeholder key.
  • Direct call_llm(provider="bedrock", ...) aggregator call returns a valid completion.
  • Full aggregate_moa_context() end-to-end with a bedrock aggregator now produces a complete synthesis instead of an empty ChatCompletion.

Impact

Affects any MoA preset with a bedrock aggregator or bedrock reference. No effect on non-bedrock providers.

_slot_runtime() resolved a bedrock slot to its bedrock-runtime base_url
plus the placeholder api_key "aws-sdk" and forwarded both to call_llm.
call_llm then treated it as a plain OpenAI-compatible endpoint and issued
an UNSIGNED bearer POST (no AWS SigV4 / IAM signing), so Bedrock returned
an empty/malformed ChatCompletion (choices=None) and the MoA aggregator
turn failed validation.

Add 'bedrock' to the name-preserve set alongside nous/openai-codex/
xai-oauth so bedrock slots are passed by provider name only, routing
through call_llm's dedicated SigV4-signed bedrock branch.

Affects any MoA preset using a bedrock aggregator or bedrock reference.
@alt-glitch alt-glitch added type/bug Something isn't working P3 Low — cosmetic, nice to have comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint provider/bedrock AWS Bedrock (boto3, IAM) labels Jun 29, 2026

@tonydwb tonydwb left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review Summary

Verdict: LGTM

Clean, well-scoped change with no concerns.

--- Reviewed by Hermes Agent

@teknium1

teknium1 commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Merged via #55897 — your commit was cherry-picked onto current main with your authorship preserved in git log.

The bedrock case is the same class as the OAuth providers: the provider branch builds an AWS-SigV4-signed client, and resolve_runtime_provider returns a placeholder api_key ("aws-sdk") with no real bearer — so forwarding base_url/api_key produced an unsigned POST and an empty/malformed ChatCompletion. #55897 adds bedrock to the _slot_runtime name-preservation set as you did, bundled with the anthropic fix (#54609). Thanks!

@teknium1 teknium1 closed this Jul 1, 2026
teknium1 added a commit that referenced this pull request Jul 1, 2026
…okepoint (#55991)

_slot_runtime maintained a hand-listed name-preservation set
({nous, anthropic, openai-codex, xai-oauth, bedrock}) that returned bare
provider+model to avoid call_llm collapsing an explicit base_url to the generic
'custom' route. That duplicated _resolve_task_provider_model's
_preserve_provider_with_base_url guard (a provider-catalog capability check)
and had to be extended by hand for every provider with custom auth/signing —
the exact drift that produced the anthropic (#54609) and bedrock (#54912) 429/
empty-response bugs.

Removes the whitelist: _slot_runtime now forwards the resolved base_url/api_key/
api_mode for every slot, and the single chokepoint
(_resolve_task_provider_model -> _preserve_provider_with_base_url) decides
identity preservation. Behavior is unchanged for the five providers — their
provider branches (codex Responses+Cloudflare, xai-oauth, bedrock SigV4,
anthropic OAuth Bearer+anthropic-beta, nous Portal tags) re-resolve their own
credentials by name and ignore a forwarded base_url/api_key, so forwarding is
safe even for bedrock's placeholder 'aws-sdk' key.

Verified via real-import E2E: _slot_runtime -> _resolve_task_provider_model
preserves openai-codex/xai-oauth/bedrock/anthropic/nous (+openrouter control) —
none collapse to custom. Tests updated to assert the pipeline invariant against
the real resolver instead of the removed whitelist's bare-return shape.
dtera pushed a commit to dtera/hermes-agent that referenced this pull request Jul 1, 2026
dtera pushed a commit to dtera/hermes-agent that referenced this pull request Jul 1, 2026
…okepoint (NousResearch#55991)

_slot_runtime maintained a hand-listed name-preservation set
({nous, anthropic, openai-codex, xai-oauth, bedrock}) that returned bare
provider+model to avoid call_llm collapsing an explicit base_url to the generic
'custom' route. That duplicated _resolve_task_provider_model's
_preserve_provider_with_base_url guard (a provider-catalog capability check)
and had to be extended by hand for every provider with custom auth/signing —
the exact drift that produced the anthropic (NousResearch#54609) and bedrock (NousResearch#54912) 429/
empty-response bugs.

Removes the whitelist: _slot_runtime now forwards the resolved base_url/api_key/
api_mode for every slot, and the single chokepoint
(_resolve_task_provider_model -> _preserve_provider_with_base_url) decides
identity preservation. Behavior is unchanged for the five providers — their
provider branches (codex Responses+Cloudflare, xai-oauth, bedrock SigV4,
anthropic OAuth Bearer+anthropic-beta, nous Portal tags) re-resolve their own
credentials by name and ignore a forwarded base_url/api_key, so forwarding is
safe even for bedrock's placeholder 'aws-sdk' key.

Verified via real-import E2E: _slot_runtime -> _resolve_task_provider_model
preserves openai-codex/xai-oauth/bedrock/anthropic/nous (+openrouter control) —
none collapse to custom. Tests updated to assert the pipeline invariant against
the real resolver instead of the removed whitelist's bare-return shape.
waefrebeorn pushed a commit to waefrebeorn/slermes that referenced this pull request Jul 2, 2026
waefrebeorn pushed a commit to waefrebeorn/slermes that referenced this pull request Jul 2, 2026
…okepoint (NousResearch#55991)

_slot_runtime maintained a hand-listed name-preservation set
({nous, anthropic, openai-codex, xai-oauth, bedrock}) that returned bare
provider+model to avoid call_llm collapsing an explicit base_url to the generic
'custom' route. That duplicated _resolve_task_provider_model's
_preserve_provider_with_base_url guard (a provider-catalog capability check)
and had to be extended by hand for every provider with custom auth/signing —
the exact drift that produced the anthropic (NousResearch#54609) and bedrock (NousResearch#54912) 429/
empty-response bugs.

Removes the whitelist: _slot_runtime now forwards the resolved base_url/api_key/
api_mode for every slot, and the single chokepoint
(_resolve_task_provider_model -> _preserve_provider_with_base_url) decides
identity preservation. Behavior is unchanged for the five providers — their
provider branches (codex Responses+Cloudflare, xai-oauth, bedrock SigV4,
anthropic OAuth Bearer+anthropic-beta, nous Portal tags) re-resolve their own
credentials by name and ignore a forwarded base_url/api_key, so forwarding is
safe even for bedrock's placeholder 'aws-sdk' key.

Verified via real-import E2E: _slot_runtime -> _resolve_task_provider_model
preserves openai-codex/xai-oauth/bedrock/anthropic/nous (+openrouter control) —
none collapse to custom. Tests updated to assert the pipeline invariant against
the real resolver instead of the removed whitelist's bare-return shape.
Jasper6439 pushed a commit to Jasper6439/hermes-agent that referenced this pull request Jul 5, 2026
Jasper6439 pushed a commit to Jasper6439/hermes-agent that referenced this pull request Jul 5, 2026
…okepoint (NousResearch#55991)

_slot_runtime maintained a hand-listed name-preservation set
({nous, anthropic, openai-codex, xai-oauth, bedrock}) that returned bare
provider+model to avoid call_llm collapsing an explicit base_url to the generic
'custom' route. That duplicated _resolve_task_provider_model's
_preserve_provider_with_base_url guard (a provider-catalog capability check)
and had to be extended by hand for every provider with custom auth/signing —
the exact drift that produced the anthropic (NousResearch#54609) and bedrock (NousResearch#54912) 429/
empty-response bugs.

Removes the whitelist: _slot_runtime now forwards the resolved base_url/api_key/
api_mode for every slot, and the single chokepoint
(_resolve_task_provider_model -> _preserve_provider_with_base_url) decides
identity preservation. Behavior is unchanged for the five providers — their
provider branches (codex Responses+Cloudflare, xai-oauth, bedrock SigV4,
anthropic OAuth Bearer+anthropic-beta, nous Portal tags) re-resolve their own
credentials by name and ignore a forwarded base_url/api_key, so forwarding is
safe even for bedrock's placeholder 'aws-sdk' key.

Verified via real-import E2E: _slot_runtime -> _resolve_task_provider_model
preserves openai-codex/xai-oauth/bedrock/anthropic/nous (+openrouter control) —
none collapse to custom. Tests updated to assert the pipeline invariant against
the real resolver instead of the removed whitelist's bare-return shape.
habarmc1223-sudo pushed a commit to habarmc1223-sudo/hermes-agent-fluxmem that referenced this pull request Jul 8, 2026
habarmc1223-sudo pushed a commit to habarmc1223-sudo/hermes-agent-fluxmem that referenced this pull request Jul 8, 2026
…okepoint (NousResearch#55991)

_slot_runtime maintained a hand-listed name-preservation set
({nous, anthropic, openai-codex, xai-oauth, bedrock}) that returned bare
provider+model to avoid call_llm collapsing an explicit base_url to the generic
'custom' route. That duplicated _resolve_task_provider_model's
_preserve_provider_with_base_url guard (a provider-catalog capability check)
and had to be extended by hand for every provider with custom auth/signing —
the exact drift that produced the anthropic (NousResearch#54609) and bedrock (NousResearch#54912) 429/
empty-response bugs.

Removes the whitelist: _slot_runtime now forwards the resolved base_url/api_key/
api_mode for every slot, and the single chokepoint
(_resolve_task_provider_model -> _preserve_provider_with_base_url) decides
identity preservation. Behavior is unchanged for the five providers — their
provider branches (codex Responses+Cloudflare, xai-oauth, bedrock SigV4,
anthropic OAuth Bearer+anthropic-beta, nous Portal tags) re-resolve their own
credentials by name and ignore a forwarded base_url/api_key, so forwarding is
safe even for bedrock's placeholder 'aws-sdk' key.

Verified via real-import E2E: _slot_runtime -> _resolve_task_provider_model
preserves openai-codex/xai-oauth/bedrock/anthropic/nous (+openrouter control) —
none collapse to custom. Tests updated to assert the pipeline invariant against
the real resolver instead of the removed whitelist's bare-return shape.
santhreal pushed a commit to santhreal/hermes-agent that referenced this pull request Jul 13, 2026
santhreal pushed a commit to santhreal/hermes-agent that referenced this pull request Jul 13, 2026
…okepoint (NousResearch#55991)

_slot_runtime maintained a hand-listed name-preservation set
({nous, anthropic, openai-codex, xai-oauth, bedrock}) that returned bare
provider+model to avoid call_llm collapsing an explicit base_url to the generic
'custom' route. That duplicated _resolve_task_provider_model's
_preserve_provider_with_base_url guard (a provider-catalog capability check)
and had to be extended by hand for every provider with custom auth/signing —
the exact drift that produced the anthropic (NousResearch#54609) and bedrock (NousResearch#54912) 429/
empty-response bugs.

Removes the whitelist: _slot_runtime now forwards the resolved base_url/api_key/
api_mode for every slot, and the single chokepoint
(_resolve_task_provider_model -> _preserve_provider_with_base_url) decides
identity preservation. Behavior is unchanged for the five providers — their
provider branches (codex Responses+Cloudflare, xai-oauth, bedrock SigV4,
anthropic OAuth Bearer+anthropic-beta, nous Portal tags) re-resolve their own
credentials by name and ignore a forwarded base_url/api_key, so forwarding is
safe even for bedrock's placeholder 'aws-sdk' key.

Verified via real-import E2E: _slot_runtime -> _resolve_task_provider_model
preserves openai-codex/xai-oauth/bedrock/anthropic/nous (+openrouter control) —
none collapse to custom. Tests updated to assert the pipeline invariant against
the real resolver instead of the removed whitelist's bare-return shape.
Gravezzz pushed a commit to Gravezzz/hermes-agent that referenced this pull request Jul 21, 2026
Gravezzz pushed a commit to Gravezzz/hermes-agent that referenced this pull request Jul 21, 2026
…okepoint (NousResearch#55991)

_slot_runtime maintained a hand-listed name-preservation set
({nous, anthropic, openai-codex, xai-oauth, bedrock}) that returned bare
provider+model to avoid call_llm collapsing an explicit base_url to the generic
'custom' route. That duplicated _resolve_task_provider_model's
_preserve_provider_with_base_url guard (a provider-catalog capability check)
and had to be extended by hand for every provider with custom auth/signing —
the exact drift that produced the anthropic (NousResearch#54609) and bedrock (NousResearch#54912) 429/
empty-response bugs.

Removes the whitelist: _slot_runtime now forwards the resolved base_url/api_key/
api_mode for every slot, and the single chokepoint
(_resolve_task_provider_model -> _preserve_provider_with_base_url) decides
identity preservation. Behavior is unchanged for the five providers — their
provider branches (codex Responses+Cloudflare, xai-oauth, bedrock SigV4,
anthropic OAuth Bearer+anthropic-beta, nous Portal tags) re-resolve their own
credentials by name and ignore a forwarded base_url/api_key, so forwarding is
safe even for bedrock's placeholder 'aws-sdk' key.

Verified via real-import E2E: _slot_runtime -> _resolve_task_provider_model
preserves openai-codex/xai-oauth/bedrock/anthropic/nous (+openrouter control) —
none collapse to custom. Tests updated to assert the pipeline invariant against
the real resolver instead of the removed whitelist's bare-return shape.
leewenjie pushed a commit to leewenjie/hermes-agent that referenced this pull request Aug 7, 2026
leewenjie pushed a commit to leewenjie/hermes-agent that referenced this pull request Aug 7, 2026
…okepoint (NousResearch#55991)

_slot_runtime maintained a hand-listed name-preservation set
({nous, anthropic, openai-codex, xai-oauth, bedrock}) that returned bare
provider+model to avoid call_llm collapsing an explicit base_url to the generic
'custom' route. That duplicated _resolve_task_provider_model's
_preserve_provider_with_base_url guard (a provider-catalog capability check)
and had to be extended by hand for every provider with custom auth/signing —
the exact drift that produced the anthropic (NousResearch#54609) and bedrock (NousResearch#54912) 429/
empty-response bugs.

Removes the whitelist: _slot_runtime now forwards the resolved base_url/api_key/
api_mode for every slot, and the single chokepoint
(_resolve_task_provider_model -> _preserve_provider_with_base_url) decides
identity preservation. Behavior is unchanged for the five providers — their
provider branches (codex Responses+Cloudflare, xai-oauth, bedrock SigV4,
anthropic OAuth Bearer+anthropic-beta, nous Portal tags) re-resolve their own
credentials by name and ignore a forwarded base_url/api_key, so forwarding is
safe even for bedrock's placeholder 'aws-sdk' key.

Verified via real-import E2E: _slot_runtime -> _resolve_task_provider_model
preserves openai-codex/xai-oauth/bedrock/anthropic/nous (+openrouter control) —
none collapse to custom. Tests updated to assert the pipeline invariant against
the real resolver instead of the removed whitelist's bare-return shape.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P3 Low — cosmetic, nice to have provider/bedrock AWS Bedrock (boto3, IAM) type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants