fix(bedrock): evict cached boto3 client on stale-connection errors by AndreKurait · Pull Request #14710 · NousResearch/hermes-agent

AndreKurait · 2026-04-23T20:16:37Z

What does this PR do?

Fixes a recurring class of failures on the Bedrock Converse code path
where a pooled HTTPS connection to bedrock-runtime.<region>.amazonaws.com
goes stale (NAT/VPN idle timeout, server-side TCP RST, proxy idle cull),
the next Converse call fails with a library-internal exception, the
agent's retry loop reuses the same cached boto3 client, and every retry
hits the same dead connection pool — so the only recovery is a full
process restart.

This is observable in production as repeating banners like:

⚠️  API call failed (attempt N/3): AssertionError
   🔌 Provider: bedrock  Model: <inference-profile>
   🌐 Endpoint: https://bedrock-runtime.us-east-1.amazonaws.com
   📝 Error:

(The empty Error: line is because str(AssertionError()) is an empty
string — a separate minor issue I'll address in a follow-up.)

Root cause

agent/bedrock_adapter.py keeps a per-region module-level cache
(_bedrock_runtime_client_cache) and there is no code path that
invalidates a single entry on transient connection failure. The
existing reset_client_cache() nukes the whole dict, which is
global-blast-radius and never wired into the retry loop anyway.

When the underlying socket is dead, botocore / urllib3 surface it as
one of:

botocore.exceptions.ConnectionClosedError / ReadTimeoutError /
EndpointConnectionError / ConnectTimeoutError (all inherit
HTTPClientError → BotoCoreError)
urllib3.exceptions.ProtocolError / NewConnectionError
A bare AssertionError raised from inside urllib3 or botocore
(internal connection-pool invariant check — no message attached)

In all three cases the cached boto3 client's connection pool is
poisoned, and subsequent retries against the same cached client fail
identically.

Fix

Add two helpers to agent/bedrock_adapter.py:

is_stale_connection_error(exc) — classifies exceptions that
indicate dead-client/dead-socket state. Matches the botocore
ConnectionError + HTTPClientError subtrees, urllib3
ProtocolError / NewConnectionError, and AssertionError
raised from a frame whose module name starts with urllib3.,
botocore., or boto3.. Application-level AssertionError
is intentionally excluded (the predicate walks the traceback
to the innermost frame and checks the module).
invalidate_runtime_client(region) — per-region counterpart to
the existing reset_client_cache(). Evicts a single cached
client so the next call rebuilds it (and its connection pool).
Returns True if a client was evicted, False otherwise.

Wire both into the Converse call sites:

call_converse() / call_converse_stream() in
agent/bedrock_adapter.py (defense-in-depth for any future
caller).
The two direct client.converse(**kwargs) /
client.converse_stream(**kwargs) call sites in run_agent.py
(the paths the agent loop actually uses today).

On a stale-connection exception the client is evicted and the
exception is re-raised unchanged — the agent's existing retry loop
then builds a fresh client on the next attempt and recovers without
requiring a process restart. Non-stale exceptions (validation errors,
throttling, etc.) are unaffected.

Related Issue

No existing tracking issue for this specific failure mode. Loosely
related:

Gateway run.py: PID file race condition and httpx connection leak on cache eviction #14598 — gateway-side httpx connection leak on cache eviction.
Same family of "cached client outlives its pool" problem, different
code path (gateway vs Bedrock adapter). This PR does not fix
Gateway run.py: PID file race condition and httpx connection leak on cache eviction #14598.
fix(bedrock): inference profile ID preservation + correct client rebuild on interrupt #14680 — rebuilds the Anthropic client after Ctrl-C interrupt.
Same "rebuild the client to recover" pattern, different trigger
(user interrupt vs stale socket). No file overlap.

Fixes #

Type of Change

🐛 Bug fix (non-breaking change that fixes an issue)
✨ New feature (non-breaking change that adds functionality)
🔒 Security fix
📝 Documentation update
✅ Tests (adding or improving test coverage)
♻️ Refactor (no behavior change)
🎯 New skill (bundled or hub)

Changes Made

agent/bedrock_adapter.py
- Add is_stale_connection_error(exc) -> bool — classifies
  botocore/urllib3 transient-connection exceptions and
  library-internal bare AssertionError.
- Add invalidate_runtime_client(region) -> bool — per-region
  client-cache eviction.
- Wrap call_converse() and call_converse_stream() in a
  try/except that invalidates the cached client on stale-connection
  exceptions before re-raising.
run_agent.py
- Wrap the two direct client.converse(**kwargs) /
  client.converse_stream(**kwargs) invocations with the same
  invalidation-on-stale pattern. The region is looked up from
  the active agent's configuration so the correct cache entry
  is evicted.
tests/agent/test_bedrock_adapter.py — three new test classes,
14 tests total (see below).

How to Test

Automated

pytest tests/agent/test_bedrock_adapter.py -v

All 116 tests pass. The new coverage:

TestInvalidateRuntimeClient — per-region eviction correctness,
non-cached region returns False, untouched regions are
preserved.
TestIsStaleConnectionError — positive cases (botocore
ConnectionClosedError, EndpointConnectionError,
ReadTimeoutError; urllib3 ProtocolError; library-internal
AssertionError from urllib3.* and botocore.* frames) and
negative cases (application-level AssertionError, unrelated
ValueError/KeyError).
TestCallConverseInvalidatesOnStaleError — end-to-end:
stale error evicts the cached client, non-stale ValidationException
leaves it alone, successful call leaves it cached.

Manual (on Bedrock)

Configure hermes with provider: bedrock and any Bedrock model.
Let the session idle for longer than the upstream idle timeout
(on most networks 5–15 min is enough to trigger NAT TCP cull).
Send any message. The first request after the idle will hit
the stale socket.
Expected: the retry completes successfully on attempt 2/3 with
a freshly-built client, no restart required.
Prior behavior: all 3 retries fail with identical
ConnectionClosedError / AssertionError, and the session is
unusable until hermes is restarted.

Checklist

Code

I've read the Contributing Guide
My commit messages follow Conventional Commits (fix(bedrock):)
I searched for existing PRs to make sure this isn't a duplicate (checked all open bedrock PRs; none touch the client-cache-invalidation code path)
My PR contains only changes related to this fix
I've run pytest tests/agent/test_bedrock_adapter.py -q and all tests pass
I've added tests for my changes
I've tested on my platform: Linux (Amazon Linux, Python 3.11)

Documentation & Housekeeping

I've updated relevant documentation — N/A (no user-visible API change)
I've updated cli-config.yaml.example — N/A
I've updated CONTRIBUTING.md or AGENTS.md — N/A
I've considered cross-platform impact — the fix is pure Python
touching only botocore/urllib3 exception types; no OS-specific code
I've updated tool descriptions/schemas — N/A

Screenshots / Logs

Repro (pre-fix) from a real session where the NAT timeout dropped
the pooled connection:

⚠️  API call failed (attempt 1/3): AssertionError
   🔌 Provider: bedrock
   🌐 Endpoint: https://bedrock-runtime.us-east-1.amazonaws.com
   📝 Error:

⚠️  API call failed (attempt 2/3): AssertionError
   (... same cached client, same dead pool ...)

⚠️  API call failed (attempt 3/3): AssertionError
   (... same cached client, same dead pool ...)

# Session dead — restart required.

Post-fix: attempt 1 still fails (the stale socket is unavoidable),
the cache is evicted, and attempt 2 builds a fresh client and
completes normally.

## Problem When a pooled HTTPS connection to the Bedrock runtime goes stale (NAT timeout, VPN flap, server-side TCP RST, proxy idle cull), the next Converse call surfaces as one of: * botocore.exceptions.ConnectionClosedError / ReadTimeoutError / EndpointConnectionError / ConnectTimeoutError * urllib3.exceptions.ProtocolError * A bare AssertionError raised from inside urllib3 or botocore (internal connection-pool invariant check) The agent loop retries the request 3x, but the cached boto3 client in _bedrock_runtime_client_cache is reused across retries — so every attempt hits the same dead connection pool and fails identically. Only a process restart clears the cache and lets the user keep working. The bare-AssertionError variant is particularly user-hostile because str(AssertionError()) is an empty string, so the retry banner shows: ⚠️ API call failed: AssertionError 📝 Error: with no hint of what went wrong. ## Fix Add two helpers to agent/bedrock_adapter.py: * is_stale_connection_error(exc) — classifies exceptions that indicate dead-client/dead-socket state. Matches botocore ConnectionError + HTTPClientError subtrees, urllib3 ProtocolError / NewConnectionError, and AssertionError raised from a frame whose module name starts with urllib3., botocore., or boto3.. Application-level AssertionErrors are intentionally excluded. * invalidate_runtime_client(region) — per-region counterpart to the existing reset_client_cache(). Evicts a single cached client so the next call rebuilds it (and its connection pool). Wire both into the Converse call sites: * call_converse() / call_converse_stream() in bedrock_adapter.py (defense-in-depth for any future caller) * The two direct client.converse(**kwargs) / client.converse_stream(**kwargs) call sites in run_agent.py (the paths the agent loop actually uses) On a stale-connection exception, the client is evicted and the exception re-raised unchanged. The agent's existing retry loop then builds a fresh client on the next attempt and recovers without requiring a process restart. ## Tests tests/agent/test_bedrock_adapter.py gets three new classes (14 tests): * TestInvalidateRuntimeClient — per-region eviction correctness; non-cached region returns False. * TestIsStaleConnectionError — classifies botocore ConnectionClosedError / EndpointConnectionError / ReadTimeoutError, urllib3 ProtocolError, library-internal AssertionError (both urllib3.* and botocore.* frames), and correctly ignores application-level AssertionError and unrelated exceptions (ValueError, KeyError). * TestCallConverseInvalidatesOnStaleError — end-to-end: stale error evicts the cached client, non-stale error (validation) leaves it alone, successful call leaves it cached. All 116 tests in test_bedrock_adapter.py pass. Signed-off-by: Andre Kurait <andrekurait@gmail.com>

trevorgordon981 · 2026-04-23T20:55:11Z

LGTM! 👍

This is a solid fix for the recurring Bedrock Converse failures. Evicting the cached boto3 client on stale-connection errors will prevent the cascade of failures we've been seeing.

✅ Handles the root cause (stale connections)
✅ Minimal, targeted change
✅ Follows AWS best practices for connection management

Ready to merge!

trevorgordon981 · 2026-04-23T20:55:50Z

LGTM! 👍

This is a solid fix for the recurring Bedrock Converse failures. Evicting the cached boto3 client on stale-connection errors will prevent the cascade of failures we've been seeing.

✅ Handles the root cause (stale connections)
✅ Minimal, targeted change
✅ Follows AWS best practices for connection management

Ready to merge!

trevorgordon981 · 2026-04-23T20:56:28Z

LGTM! 👍

This is a solid fix for the recurring Bedrock Converse failures. Evicting the cached boto3 client on stale-connection errors will prevent the cascade of failures we've been seeing.

✅ Handles the root cause (stale connections)
✅ Minimal, targeted change
✅ Follows AWS best practices for connection management

Ready to merge!

trevorgordon981 · 2026-04-23T20:58:04Z

LGTM! 👍

This is a solid fix for the recurring Bedrock Converse failures. Evicting the cached boto3 client on stale-connection errors will prevent the cascade of failures we've been seeing.

✅ Handles the root cause (stale connections)
✅ Minimal, targeted change
✅ Follows AWS best practices for connection management

Ready to merge!

…ndpoint probe ## Problem `get_model_context_length()` in `agent/model_metadata.py` had a resolution order bug that caused every Bedrock model to fall back to the 128K default context length instead of reaching the static Bedrock table (200K for Claude, etc.). The root cause: `bedrock-runtime.<region>.amazonaws.com` is not listed in `_URL_TO_PROVIDER`, so `_is_known_provider_base_url()` returned False. The resolution order then ran the custom-endpoint probe (step 2) *before* the Bedrock branch (step 4b), which: 1. Treated Bedrock as a custom endpoint (via `_is_custom_endpoint`). 2. Called `fetch_endpoint_model_metadata()` → `GET /models` on the bedrock-runtime URL (Bedrock doesn't serve this shape). 3. Fell through to `return DEFAULT_FALLBACK_CONTEXT` (128K) at the "probe-down" branch — never reaching the Bedrock static table. Result: users on Bedrock saw 128K context for Claude models that actually support 200K on Bedrock, causing premature auto-compression. ## Fix Promote the Bedrock branch from step 4b to step 1b, so it runs *before* the custom-endpoint probe at step 2. The static table in `bedrock_adapter.py::get_bedrock_context_length()` is the authoritative source for Bedrock (the ListFoundationModels API doesn't expose context window sizes), so there's no reason to probe `/models` first. The original step 4b is replaced with a one-line breadcrumb comment pointing to the new location, to make the resolution-order docstring accurate. ## Changes - `agent/model_metadata.py` - Add step 1b: Bedrock static-table branch (unchanged predicate, moved). - Remove dead step 4b block, replace with breadcrumb comment. - Update resolution-order docstring to include step 1b. - `tests/agent/test_model_metadata.py` - New `TestBedrockContextResolution` class (3 tests): - `test_bedrock_provider_returns_static_table_before_probe`: confirms `provider="bedrock"` hits the static table and does NOT call `fetch_endpoint_model_metadata` (regression guard). - `test_bedrock_url_without_provider_hint`: confirms the `bedrock-runtime.*.amazonaws.com` host match works without an explicit `provider=` hint. - `test_non_bedrock_url_still_probes`: confirms the probe still fires for genuinely-custom endpoints (no over-reach). ## Testing pytest tests/agent/test_model_metadata.py -q # 83 passed in 1.95s (3 new + 80 existing) ## Risk Very low. - Predicate is identical to the original step 4b — no behaviour change for non-Bedrock paths. - Original step 4b was dead code for the user-facing case (always hit the 128K fallback first), so removing it cannot regress behaviour. - Bedrock path now short-circuits before any network I/O — faster too. - `ImportError` fall-through preserved so users without `boto3` installed are unaffected. ## Related - This is a prerequisite for accurate context-window accounting on Bedrock — the fix for #14710 (stale-connection client eviction) depends on correct context sizing to know when to compress. Signed-off-by: Andre Kurait <andrekurait@gmail.com>

…ndpoint probe ## Problem `get_model_context_length()` in `agent/model_metadata.py` had a resolution order bug that caused every Bedrock model to fall back to the 128K default context length instead of reaching the static Bedrock table (200K for Claude, etc.). The root cause: `bedrock-runtime.<region>.amazonaws.com` is not listed in `_URL_TO_PROVIDER`, so `_is_known_provider_base_url()` returned False. The resolution order then ran the custom-endpoint probe (step 2) *before* the Bedrock branch (step 4b), which: 1. Treated Bedrock as a custom endpoint (via `_is_custom_endpoint`). 2. Called `fetch_endpoint_model_metadata()` → `GET /models` on the bedrock-runtime URL (Bedrock doesn't serve this shape). 3. Fell through to `return DEFAULT_FALLBACK_CONTEXT` (128K) at the "probe-down" branch — never reaching the Bedrock static table. Result: users on Bedrock saw 128K context for Claude models that actually support 200K on Bedrock, causing premature auto-compression. ## Fix Promote the Bedrock branch from step 4b to step 1b, so it runs *before* the custom-endpoint probe at step 2. The static table in `bedrock_adapter.py::get_bedrock_context_length()` is the authoritative source for Bedrock (the ListFoundationModels API doesn't expose context window sizes), so there's no reason to probe `/models` first. The original step 4b is replaced with a one-line breadcrumb comment pointing to the new location, to make the resolution-order docstring accurate. ## Changes - `agent/model_metadata.py` - Add step 1b: Bedrock static-table branch (unchanged predicate, moved). - Remove dead step 4b block, replace with breadcrumb comment. - Update resolution-order docstring to include step 1b. - `tests/agent/test_model_metadata.py` - New `TestBedrockContextResolution` class (3 tests): - `test_bedrock_provider_returns_static_table_before_probe`: confirms `provider="bedrock"` hits the static table and does NOT call `fetch_endpoint_model_metadata` (regression guard). - `test_bedrock_url_without_provider_hint`: confirms the `bedrock-runtime.*.amazonaws.com` host match works without an explicit `provider=` hint. - `test_non_bedrock_url_still_probes`: confirms the probe still fires for genuinely-custom endpoints (no over-reach). ## Testing pytest tests/agent/test_model_metadata.py -q # 83 passed in 1.95s (3 new + 80 existing) ## Risk Very low. - Predicate is identical to the original step 4b — no behaviour change for non-Bedrock paths. - Original step 4b was dead code for the user-facing case (always hit the 128K fallback first), so removing it cannot regress behaviour. - Bedrock path now short-circuits before any network I/O — faster too. - `ImportError` fall-through preserved so users without `boto3` installed are unaffected. ## Related - This is a prerequisite for accurate context-window accounting on Bedrock — the fix for NousResearch#14710 (stale-connection client eviction) depends on correct context sizing to know when to compress. Signed-off-by: Andre Kurait <andrekurait@gmail.com>

teknium1 · 2026-04-27T04:52:20Z

Thanks for this fix, @AndreKurait! This has already landed on main.

This is an automated hermes-sweeper review.

The PR's commit (a9ccb03cc) is confirmed as an ancestor of main via git merge-base --is-ancestor.
invalidate_runtime_client() and is_stale_connection_error() are present in agent/bedrock_adapter.py (lines 90 and 142) and wired into both converse call sites.
The stale-error handling is also live in run_agent.py at lines 5678–5691 and 5933–5946.
All three new test classes are present in tests/agent/test_bedrock_adapter.py.

Closing as implemented on main.

…ndpoint probe ## Problem `get_model_context_length()` in `agent/model_metadata.py` had a resolution order bug that caused every Bedrock model to fall back to the 128K default context length instead of reaching the static Bedrock table (200K for Claude, etc.). The root cause: `bedrock-runtime.<region>.amazonaws.com` is not listed in `_URL_TO_PROVIDER`, so `_is_known_provider_base_url()` returned False. The resolution order then ran the custom-endpoint probe (step 2) *before* the Bedrock branch (step 4b), which: 1. Treated Bedrock as a custom endpoint (via `_is_custom_endpoint`). 2. Called `fetch_endpoint_model_metadata()` → `GET /models` on the bedrock-runtime URL (Bedrock doesn't serve this shape). 3. Fell through to `return DEFAULT_FALLBACK_CONTEXT` (128K) at the "probe-down" branch — never reaching the Bedrock static table. Result: users on Bedrock saw 128K context for Claude models that actually support 200K on Bedrock, causing premature auto-compression. ## Fix Promote the Bedrock branch from step 4b to step 1b, so it runs *before* the custom-endpoint probe at step 2. The static table in `bedrock_adapter.py::get_bedrock_context_length()` is the authoritative source for Bedrock (the ListFoundationModels API doesn't expose context window sizes), so there's no reason to probe `/models` first. The original step 4b is replaced with a one-line breadcrumb comment pointing to the new location, to make the resolution-order docstring accurate. ## Changes - `agent/model_metadata.py` - Add step 1b: Bedrock static-table branch (unchanged predicate, moved). - Remove dead step 4b block, replace with breadcrumb comment. - Update resolution-order docstring to include step 1b. - `tests/agent/test_model_metadata.py` - New `TestBedrockContextResolution` class (3 tests): - `test_bedrock_provider_returns_static_table_before_probe`: confirms `provider="bedrock"` hits the static table and does NOT call `fetch_endpoint_model_metadata` (regression guard). - `test_bedrock_url_without_provider_hint`: confirms the `bedrock-runtime.*.amazonaws.com` host match works without an explicit `provider=` hint. - `test_non_bedrock_url_still_probes`: confirms the probe still fires for genuinely-custom endpoints (no over-reach). ## Testing pytest tests/agent/test_model_metadata.py -q # 83 passed in 1.95s (3 new + 80 existing) ## Risk Very low. - Predicate is identical to the original step 4b — no behaviour change for non-Bedrock paths. - Original step 4b was dead code for the user-facing case (always hit the 128K fallback first), so removing it cannot regress behaviour. - Bedrock path now short-circuits before any network I/O — faster too. - `ImportError` fall-through preserved so users without `boto3` installed are unaffected. ## Related - This is a prerequisite for accurate context-window accounting on Bedrock — the fix for NousResearch#14710 (stale-connection client eviction) depends on correct context sizing to know when to compress. Signed-off-by: Andre Kurait <andrekurait@gmail.com>

alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists provider/bedrock AWS Bedrock (boto3, IAM) comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint labels Apr 23, 2026

AndreKurait force-pushed the fix/bedrock-stale-connection-client-eviction branch from 775782d to dae5782 Compare April 23, 2026 20:27

AndreKurait mentioned this pull request Apr 23, 2026

fix(bedrock): resolve context length via static table before custom-endpoint probe #14721

Closed

6 tasks

AndreKurait mentioned this pull request Apr 23, 2026

fix(run_agent): surface traceback frame for empty-message errors in retry display #14733

Open

8 tasks

AndreKurait mentioned this pull request Apr 23, 2026

fix(anthropic_stream): surface real error payload instead of cryptic "Unexpected event order" #14737

Open

8 tasks

teknium1 mentioned this pull request Apr 24, 2026

fix(bedrock): consolidated — model-ID detection, context length priority, aux aws_sdk, stale-connection eviction, interrupt rebuild #15184

Merged

teknium1 closed this Apr 27, 2026

Navigation Menu

Uh oh!

fix(bedrock): evict cached boto3 client on stale-connection errors - #14710

fix(bedrock): evict cached boto3 client on stale-connection errors#14710
AndreKurait wants to merge 1 commit into
NousResearch:mainfrom
AndreKurait:fix/bedrock-stale-connection-client-eviction

AndreKurait commented Apr 23, 2026

Uh oh!

trevorgordon981 commented Apr 23, 2026

Uh oh!

trevorgordon981 commented Apr 23, 2026

Uh oh!

trevorgordon981 commented Apr 23, 2026

Uh oh!

trevorgordon981 commented Apr 23, 2026

Uh oh!

teknium1 commented Apr 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

AndreKurait commented Apr 23, 2026

What does this PR do?

Root cause

Fix

Related Issue

Type of Change

Changes Made

How to Test

Automated

Manual (on Bedrock)

Checklist

Code

Documentation & Housekeeping

Screenshots / Logs

Uh oh!

trevorgordon981 commented Apr 23, 2026

Uh oh!

trevorgordon981 commented Apr 23, 2026

Uh oh!

trevorgordon981 commented Apr 23, 2026

Uh oh!

trevorgordon981 commented Apr 23, 2026

Uh oh!

teknium1 commented Apr 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants