fix(telegram): auto-reconnect polling after network interruption by Mibayy · Pull Request #2477 · NousResearch/hermes-agent

Mibayy · 2026-03-22T12:09:46Z

Summary

The Telegram polling error callback only handled Conflict errors (409). All other errors — including NetworkError and TimedOut that python-telegram-bot raises when the host loses connectivity — were logged and silently discarded. The long-poll connection dies, the gateway process stays alive, and the bot stops responding until manually restarted.

Root Cause

def _polling_error_callback(error: Exception) -> None:
    if not self._looks_like_polling_conflict(error):
        logger.error(...)
        return  # network error → logged, dropped, polling stays dead
    ...

Fix

_looks_like_network_error() — classifies NetworkError, TimedOut, OSError, ConnectionError as transient reconnectable errors
_handle_polling_network_error() — exponential back-off reconnect (5s → 10s → 20s → 40s → 60s cap, up to 10 attempts); resets counter on success; marks adapter retryable-fatal after exhaustion so launchd/systemd can restart the process
_polling_error_callback() — refactored to route conflict → _handle_polling_conflict, network → _handle_polling_network_error, other → log only

Behavior after fix

Scenario	Before	After
Mac sleep/wake	Bot dead until manual restart	Auto-reconnects within 5s
WiFi switch	Bot dead	Auto-reconnects
VPN reconnect	Bot dead	Auto-reconnects
Persistent outage (>10 retries)	Bot dead forever	Gateway restarts via supervisor
Conflict (409)	Retried (existing behavior)	Unchanged

Closes NousResearch#2476 The polling error callback previously only handled Conflict errors (409 from multiple getUpdates callers). All other errors, including NetworkError and TimedOut that python-telegram-bot raises when the host loses connectivity (Mac sleep, WiFi switch, VPN reconnect), were logged and silently discarded. The bot would stop responding until manually restarted. Fix: - Add _looks_like_network_error() to classify transient connectivity errors (NetworkError, TimedOut, OSError, ConnectionError). - Add _handle_polling_network_error() with exponential back-off reconnect: retries up to 10 times with delays 5s, 10s, 20s, 40s, 60s (capped). On exhaustion, marks the adapter retryable-fatal so launchd/systemd can restart the gateway process. - Refactor _polling_error_callback() to route network errors to the new handler before falling through to a generic error log. - Track _polling_network_error_count (reset on successful reconnect) independently from _polling_conflict_count.

teknium1 · 2026-03-22T16:19:11Z

Merged via #2517. Cherry-picked with authorship preserved. Clean fix — thanks!

WhatsApp changed their server protocol for property queries, causing 400 bad-request errors in fetchProps/executeInitQueries on every reconnect (Baileys issue #2477). The fix in PR #2473 changes the IQ namespace from 'w' to 'abt' and protocol from '2' to '1'. Pin to the fix branch until the next Baileys release includes it.

WhatsApp changed their server protocol for property queries, causing 400 bad-request errors in fetchProps/executeInitQueries on every reconnect (Baileys issue NousResearch#2477). The fix in PR NousResearch#2473 changes the IQ namespace from 'w' to 'abt' and protocol from '2' to '1'. Pin to the fix branch until the next Baileys release includes it.

… for Hermes (#4) * feat(pmoves-bootstrap): loader + tools_bridge + subscriber stub CLAIM the Hermes-agent fork's slice of the Mavis harness v0 (3-repo coordinated). Companion to POWERFULMOVES/PMOVES.AI PR NousResearch#2477 and POWERFULMOVES/PMOVES-pinokio PR #1 (feat/pmoves-app-launcher). The CGP (pmoves.bootstrap/v1) is the contract that ties the 3 forks together: PMOVES.AI writes it, PMOVES-hermes-agent reads it at session init + registers PMOVES tools alongside the native toolset, PMOVES-pinokio reads it when launching a PMOVES-tagged app. What this commit ships: - pmoves_bootstrap/loader.py - the CGP reader. Accepts both YAML and JSON (Hermes has pyyaml==6.0.3 in core deps, so YAML is the natural format; JSON is supported for PMOVES_BOOTSTRAP_CGP raw- string env vars). 4 input sources in priority order: path arg, source arg, PMOVES_BOOTSTRAP_CGP[_PATH] env var, vendored example. Validates structurally (the vendored v1.schema.json is the source of truth, but no jsonschema dep is added - the thin structural check covers the 80% case). Returns a typed Bootstrap object with has_tool/has_mcp/has_constraint/service/route_for accessors. Stub Bootstrap (safe defaults, all 6 constraints) when no CGP is present - the non-breaking fallback. - pmoves_bootstrap/tools_bridge.py - the PMOVES tools bridge. Reads bootstrap.tools and resolves each entry against the v0 tool registry (Python scripts + CLI binaries). Returns a BridgeResult with registered/skipped/disabled lists. The session init code (a future slice) merges registered tools into Hermes's active toolset. PMOVES_TOOLS_DISABLE env var is a per-tool deny list. Per the 'tagged-services-are-advisory' constraint, unknown tools are silently skipped (warning, not error). - pmoves_bootstrap/subscriber.py - the optional NATS subscriber. v0 is a STUB: no nats-py in Hermes's core deps (adding it would be a meaningful blast-radius change; the deps list warns against it after the Mini Shai-Hulud worm). subscribe() always returns a SubscriberStatus with enabled=False and a clear reason. The TaskEnvelope/ResultEnvelope dataclasses document the wire contract so a future slice can wire in nats-py without changing the public surface. Subjects: pmoves.agent.task.v1 (input), pmoves.agent.result.v1 (output), pmoves.bpm.phase.v1 + pmoves.bpm.pomodoro.v1 (observability, not consumed by Hermes). - pmoves_bootstrap/__init__.py - the public surface. Re-exports load_bootstrap, stub_bootstrap, export_env, register_pmoves_tools, subscribe, and the typed shapes. Future Mavis / Spark / Knuckles sessions do 'from pmoves_bootstrap import load_bootstrap, register_pmoves_tools, subscribe'. - pmoves_bootstrap/cgp_schema/v1.schema.json - vendored copy of the PMOVES.AI schema. The hermes-agent fork doesn't depend on the PMOVES.AI repo at install time. - pmoves_bootstrap/cgp_schema/example.cgp.yaml - vendored YAML example (the same data as the PMOVES.AI example.cgp.yaml). Non-breaking test pair: - No CGP present -> load_bootstrap() returns the stub Bootstrap, register_pmoves_tools() returns BridgeResult(registered=[]), subscribe() returns SubscriberStatus(enabled=False). Existing Hermes behavior unchanged. - CGP present -> load_bootstrap() validates and returns the real Bootstrap, register_pmoves_tools() adds PMOVES tools alongside the native Hermes toolset, subscribe() is a no-op (v0) or picks up Mavis-orchestrator tasks (future slice). Cross-fork plan: - PMOVES.AI PR NousResearch#2477 (writer) - PMOVES-hermes-agent PR feat/pmoves-bootstrap-consumer (agent, this PR) - PMOVES-pinokio PR feat/pmoves-app-launcher (app launcher) All three read the same v1.schema.json - the schema is the contract. The 6 constraints baked into the CGP are honored by the loader's behavior: - no-override-existing-config: the loader never writes to Hermes's own config (cli-config.yaml, hermes_state, etc.) - tagged-services-are-advisory: missing services are skipped in tools_bridge, not failed - no-chit-bypass: no CHIT signing code in this package; the Mavis orchestrator does the signing - no-force-push: this PR's commits use rebase, never --force - no-ci-bypass: PR is in DRAFT, no --admin to skip CI - preserve-existing-tools: tools_bridge adds PMOVES tools alongside Hermes's native toolset, never in place of Tests: 33/33 pass (tests/test_pmoves_bootstrap.py, run with 'python -m pytest tests/test_pmoves_bootstrap.py -o addopts='). No new core dependencies. pyyaml is already a Hermes core dep (see pyproject.toml); no new packages are added. * test(pmoves-bootstrap): 33 pytest tests across 9 groups The test suite for the hermes-side CGP consumer. Mirrors the PMOVES.AI side test taxonomy (load_from_example / load_from_source / validation_failure / stub_fallback / export_env / typed_accessor + tools_bridge + subscriber) but with 33 tests total (vs 22 on the PMOVES.AI side) because the hermes-side has more surface (YAML + JSON parsing, env-var handling, tools_bridge registry resolution, subscriber wire contract). Test groups: - A. LoadFromExampleTests (5) - the vendored example loads + validates, identity, services, routing, constraints - B. LoadFromSourceTests (4) - raw YAML, raw JSON, PMOVES_BOOTSTRAP_CGP env var, PMOVES_BOOTSTRAP_CGP_PATH env var - C. ValidationFailureTests (5) - wrong spec, missing top-level field, missing identity.agent, bad role, non-empty super_nodes - D. StubFallbackTests (2) - no CGP returns the stub; stub has all 6 constraints - E. ExportEnvTests (3) - identity vars, services + routing vars, custom env dict (no process side-effect) - F. TypedAccessorTests (3) - has_tool/has_mcp/has_constraint, service() returns None for missing, route_for() returns None for missing - G. ToolsBridgeTests (6) - stub returns empty, real CGP registers known tools, disable list excludes, unknown goes to skipped, callables are invokable, registry populated at import - H. SubscriberTests (3) - subscribe is safe no-op when disabled, TaskEnvelope round-trip, ResultEnvelope round-trip - I. Constants and subject surfaces (2) - subjects match the orchestrator, KNOWN_TARGETS contains the expected agents Run with: python -m pytest tests/test_pmoves_bootstrap.py -v -o addopts= (the -o addopts= is needed on Windows where the project-level pytest-timeout addopts expects SIGALRM which doesn't exist on Windows; the override disables the addopts so pytest-timeout isn't required for these tests). The autouse _isolate_env fixture strips PMOVES_BOOTSTRAP_*, PMOVES_SUBSCRIBER_*, and PMOVES_TOOLS_* env vars before every test, so the tests are order-independent and don't leak state across test files. * docs(pmoves-bootstrap): README + integration notes The high-level map of the pmoves_bootstrap package + the 3 files (loader, tools_bridge, subscriber) + the non-breaking contract + the design choices (YAML+JSON, no jsonschema, nats-py as follow-up) + the cross-fork plan. Future Mavis / Spark / Knuckles sessions hit this file first to understand the integration. What's in the README: - Why this exists - the 3-repo harness v0 slice, hermes-side role as the heaviest of the three (read CGP, register tools, optional subscriber) - What this slice ships - 8 files (4 .py + 2 vendored schema files + 1 test file + 1 README) - Non-breaking contract - the 6 constraints, the no-CGP fallback, the explicit-source error behavior - Public API - the 5 public functions + 4 typed shapes - Resolution order - the 4 sources in priority order - Why YAML (in addition to JSON) - Hermes has pyyaml in core deps - Why no nats-py in v0 - the blast-radius comment in pyproject.toml warns against adding new packages; the v0 subscriber is a stub with a stable wire contract documented via dataclasses - Tests - 33/33 pass with pytest, 9 test groups - What this slice does NOT do - the 4 intentional follow-ups (wiring into run_agent.py, real nats-py, CHIT trail signing, per-session tool allow-list) - Cross-fork plan - the 3 PRs and the schema as the contract * fix(bootstrap): rename Bootstrap.source to load_source, guard non-string tool_ids, drop dead imports The verifier's review of PR #4 surfaced 6 pre-merge findings; this commit applies the cleanup for all 6: 1. Semantic-naming drift (Bootstrap.source to load_source): the field name 'source' collided with meta.source (the producer). Renamed the load-source attribute to load_source; updated the dataclass field, the _from_dict factory, the stub_bootstrap factory, the docstring, the public surface in __init__.py, and the 5 call sites in test_pmoves_bootstrap.py. 2. Reasoning gap (2-line guard in register_pmoves_tools): a malformed CGP with non-string entries in the tools array (e.g. an object {inject: evil} or an int 42) used to crash the bridge with TypeError on the `in disable` check. Now the bridge skips non-string entries to the `skipped` bucket with a warning log, and the LOG.info(skipped) call uses key=str to sort mixed-type lists. 3. Defense-in-depth (sort key=str): the LOG.info(skipped) call was crashing on sorted([int, str, None]) due to int < str comparison. Added key=str to handle mixed types. The new test_G7 proves the bridge no longer crashes on tools=[gh, dict, 42, None]. 4. Cleanup (dead imports in loader.py): dropped unused re, sys, Iterable from the typing import. 5. Cleanup (test count drift in docstrings): the test file header and the README both said 31 tests / 8 groups; the actual is 33 tests / 9 groups. Updated to 33 / 9. 6. Nit (line endings on vendored JSON): added pmoves_bootstrap/cgp_schema/*.json text eol=lf to .gitattributes so Windows checkouts do not reintroduce CRLF and produce a false-positive drift signal on the SHA-256 byte-compare against the canonical PMOVES.AI copy. Also re-vendored v1.schema.json with the PMOVES.AI side new super_nodes-required + services/routing additionalProperties tightening (the Pinokio fork got the same re-vendor in its separate commit). Test count: 33 to 34 (added test_G7_non_string_tool_id_does_not_crash_bridge). All 34 pass. * chore(mailmap): add Mavis@pmoves.local -> Mavis@users.noreply.github.com The Contributor Attribution Check (CI) was failing because my local commit author (Mavis@pmoves.local) wasn't in the .mailmap. The fix is the standard mailmap format: canonical name + canonical noreply + commit email. Future Mavis commits against this fork will now be attributed correctly. The Hermes fork's contributor graph is otherwise stable; this is a no-op for attribution counting. * chore(release): add Mavis@pmoves.local to AUTHOR_MAP The Contributor Attribution Check (CI) was failing because my commit author Mavis@pmoves.local isn't in scripts/release.py AUTHOR_MAP. The check uses AUTHOR_MAP (not the .mailmap, which is for git shortlog) to attribute commits to GitHub usernames. Added the mapping Mavis@pmoves.local -> Mavis-PMOVES. The .github/PULL_REQUEST_TEMPLATE.md / CONTRIBUTING.md author guidance will surface the canonical username in future PR bodies if the operator wants to backfill the real GitHub handle. Also added a .mailmap entry (commit 35224e5) for git shortlog / GitHub contributor graph, even though the CI check doesn't read the .mailmap. --------- Co-authored-by: Mavis <Mavis@pmoves.local>

teknium1 mentioned this pull request Mar 22, 2026

fix(telegram): auto-reconnect polling after network interruption #2517

Merged

teknium1 closed this Mar 22, 2026

This was referenced Aug 3, 2026

Telegram polling conflict: self-perpetuating retry loop from drop_pending_updates=False #75017

Closed

fix(homeassistant): bound every wedgeable await and watchdog the listener #68540

Open

POWERFULMOVES mentioned this pull request Aug 8, 2026

feat(pmoves-bootstrap): CGP consumer + tools_bridge + subscriber stub for Hermes POWERFULMOVES/PMOVES-hermes-agent#4

Merged

Navigation Menu

Uh oh!

fix(telegram): auto-reconnect polling after network interruption - #2477

fix(telegram): auto-reconnect polling after network interruption#2477
Mibayy wants to merge 1 commit into
NousResearch:mainfrom
Mibayy:fix/telegram-network-reconnect-2476

Mibayy commented Mar 22, 2026

Uh oh!

teknium1 commented Mar 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

Mibayy commented Mar 22, 2026

Summary

Root Cause

Fix

Behavior after fix

Uh oh!

teknium1 commented Mar 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants