Skip to content

feat(gateway): make still-working heartbeats configurable - #7446

Open
thesammygit wants to merge 72 commits into
NousResearch:mainfrom
thesammygit:sam/signal-heartbeat-config
Open

feat(gateway): make still-working heartbeats configurable#7446
thesammygit wants to merge 72 commits into
NousResearch:mainfrom
thesammygit:sam/signal-heartbeat-config

Conversation

@thesammygit

@thesammygit thesammygit commented Apr 10, 2026

Copy link
Copy Markdown

what

  • add display.still_working_interval for gateway long-running heartbeat cadence
  • add display.still_working_overrides for per-platform control (for example: disable Signal, shorten Telegram)
  • keep the default behavior at 600s / 10 minutes when unset
  • treat 0, false, or off as disable
  • ignore invalid per-platform overrides so they inherit the global interval
  • preserve the existing activity-detail fallback in heartbeat messages when no current tool is active
  • document the new config keys
  • add targeted tests for both config loading and runtime heartbeat behavior

why

Signal users can already quiet tool progress with display.tool_progress_overrides, but the separate hardcoded 10-minute ⏳ Still working... heartbeat remained noisy. This patch makes that heartbeat configurable without changing existing defaults.

related context:

how to test

targeted tests

  • python -m pytest tests/gateway/test_still_working_notifications.py tests/gateway/test_background_process_notifications.py tests/gateway/test_gateway_inactivity_timeout.py -q
  • local result on this branch after the latest fix: 36 passed in 16.09s

full suite

  • python -m pytest tests/ -v
  • local result on this branch (macOS 26.3 / Python 3.11.14): 9956 passed, 34 skipped, 55 failed, 35 errors in 164.68s
  • local result on a clean origin/main worktree on the same machine: 9896 passed, 34 skipped, 45 failed, 94 errors in 147.76s
  • current upstream main is therefore not green in this environment either; the changed-path targeted tests above pass

manual verification

  • exercised the gateway _run_agent path locally with a long-running fake agent
  • with display.still_working_interval: 0.05, Telegram emitted still-working heartbeats
  • with display.still_working_overrides.signal: off, Signal emitted zero still-working heartbeats

platforms tested

  • macOS 26.3 (build 25D125)
  • Python 3.11.14

cross-platform impact

  • pure Python config parsing / gateway notification scheduling only
  • no shell, path, or subprocess semantics changed
  • not manually tested on Windows or Linux

Copilot AI review requested due to automatic review settings April 10, 2026 23:28

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds configuration for the gateway’s long-running “⏳ Still working...” heartbeat so users can tune or disable it globally and per messaging platform, while preserving the existing default cadence (600s) when unset.

Changes:

  • Add display.still_working_interval (global) and display.still_working_overrides (per-platform) with disable semantics for 0/false/off.
  • Wire the configured interval into the gateway’s long-running heartbeat loop.
  • Document the new keys and add focused tests for config-loading behavior.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
gateway/run.py Loads configurable heartbeat interval from config.yaml and uses it to schedule/disable periodic “still working” messages.
hermes_cli/config.py Adds the new display keys to the default managed config structure.
tests/gateway/test_still_working_notifications.py Adds unit tests covering interval parsing, disable values, and override inheritance behavior.
website/docs/user-guide/messaging/index.md Documents how to configure long-running agent heartbeats in the messaging gateway guide.
website/docs/user-guide/configuration.md Adds the new config keys to the display section and documents global + per-platform behavior.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.


### Long-running agent heartbeats

When an agent turn itself runs for a long time (for example deep research, delegation, or many tool calls), the gateway can send periodic `⏳ Still working...` heartbeats. Control this with `display.still_working_interval`:
Comment thread gateway/run.py Outdated
Comment on lines +7432 to +7435
_iter_n = _act.get("api_call_count", 0)
_iter_max = _act.get("max_iterations", 0)
if _cur_tool:
_status_detail = f" — iteration {_iter_n}/{_iter_max}, running: {_cur_tool}"
@thesammygit
thesammygit marked this pull request as draft April 11, 2026 00:24
@thesammygit
thesammygit marked this pull request as ready for review April 11, 2026 01:05
@thesammygit

Copy link
Copy Markdown
Author

addendum after another local verification pass on macos 26.3 / python 3.11.14.

what i verified:

  • i found one real regression in this pr and fixed it before marking the pr ready: the new still-working path had dropped the old last_activity_desc fallback when current_tool was absent, which made some heartbeat messages lose useful activity detail. that is now restored.
  • after that fix, the heartbeat-adjacent targeted batch is green locally:
    python -m pytest tests/gateway/test_still_working_notifications.py tests/gateway/test_background_process_notifications.py tests/gateway/test_gateway_inactivity_timeout.py -q
    result: 36 passed in 16.17s

full-suite context from the same machine:

  • this branch: 9964 passed, 34 skipped, 51 failed, 33 errors
  • clean origin/main worktree: 9897 passed, 34 skipped, 43 failed, 96 errors

why i do not currently think this pr is introducing additional meaningful regressions in the area it touches:

  • the changed-path targeted gateway batch above is green after the fix.
  • a representative unrelated red file, tests/gateway/test_run_progress_topics.py, still fails on both this branch and clean origin/main with the same AttributeError: 'GatewayRunner' object has no attribute '_session_model_overrides' failure pattern.
  • so the repo is still not green in this local environment even on clean main, and the remaining red tests i rechecked are not specific evidence against the still-working heartbeat patch.

in other words: i did find one patch-local issue, fixed it, and after that i have not found evidence of further deterministic regressions tied to this change.

@thesammygit

Copy link
Copy Markdown
Author

clarification on the local full-suite numbers, because looking only at failed counts is misleading.

on the same machine:

  • this branch: 51 failed, 33 errors
  • clean origin/main: 43 failed, 96 errors

so while this branch shows 8 more failed, it actually has far fewer total red tests locally (82 vs 139). a lot of cases that are hard errors on clean main show up as ordinary failed results on this branch instead.

for the 12 tests that appeared as branch-only failed in the full-suite summary:

  • 7 are already broken on clean main too, just as error rather than failed
  • the remaining 5 all passed when rerun directly in isolation:
    • tests/gateway/test_background_process_notifications.py::test_run_process_watcher_respects_notification_mode[error-sessions5-1-finished with exit code 1]
    • tests/gateway/test_background_process_notifications.py::test_run_process_watcher_respects_notification_mode[result-sessions3-1-finished with exit code 0]
    • tests/gateway/test_background_process_notifications.py::test_thread_id_passed_to_send
    • tests/tools/test_managed_media_gateways.py::test_openai_tts_uses_managed_audio_gateway_when_direct_key_absent
    • tests/tools/test_managed_media_gateways.py::test_transcription_uses_model_specific_response_formats

so i do not currently have evidence that this PR is introducing additional meaningful deterministic regressions beyond the one heartbeat-detail fallback issue i already fixed.

@thesammygit

Copy link
Copy Markdown
Author

resolved the merge conflict by merging current origin/main into this branch and reconciling the touched gateway/config/docs areas.

what changed in the conflict resolution pass:

  • kept the configurable still-working heartbeat behavior
  • preserved backward compatibility with legacy agent.gateway_notify_interval / HERMES_AGENT_NOTIFY_INTERVAL when the new display keys are unset
  • carried forward the newer interim_assistant_messages / display.platforms config/docs changes from main
  • fixed the docs grammar nit from the copilot comment

local verification after the merge commit (497921a4):

  • uv run pytest tests/gateway/test_still_working_notifications.py tests/gateway/test_background_process_notifications.py tests/gateway/test_gateway_inactivity_timeout.py tests/hermes_cli/test_config.py -q
  • result: 80 passed in 16.15s

pr is no longer in a merge-conflict state on my side.

@thesammygit

Copy link
Copy Markdown
Author

resolved the new origin/main merge conflict and pushed a follow-up merge commit.

what i changed:

  • merged current origin/main into this branch
  • reconciled the hermes_cli/config.py v16→17 migration so it keeps the still-working display defaults from this pr and the legacy compression.summary_* → auxiliary.compression migration from main
  • added a focused config regression test covering both behaviors together

validation:

  • uv run --active pytest tests/gateway/test_still_working_notifications.py tests/gateway/test_background_process_notifications.py tests/gateway/test_gateway_inactivity_timeout.py tests/gateway/test_display_config.py tests/hermes_cli/test_config.py -q
  • 108 passed in 26.06s

@thesammygit

thesammygit commented Apr 16, 2026

Copy link
Copy Markdown
Author

resolved the current origin/main merge conflict and pushed a follow-up merge commit.

what i changed:

  • merged current origin/main into this branch
  • reconciled tests/hermes_cli/test_config.py so it keeps this pr's still-working heartbeat migration coverage and main's newer discord channel prompts migration coverage

validation:

  • uv run --active pytest tests/gateway/test_still_working_notifications.py tests/gateway/test_background_process_notifications.py tests/gateway/test_gateway_inactivity_timeout.py tests/gateway/test_display_config.py tests/hermes_cli/test_config.py -q
  • result: 127 passed in 27.88s

@thesammygit

Copy link
Copy Markdown
Author

merged current origin/main into this branch to keep the heartbeat/config/docs/test area current with main.

what changed:

  • merged origin/main into sam/signal-heartbeat-config
  • carried forward main's newer updates across the touched gateway/config/docs/tests area

validation:

  • uv run --active pytest tests/gateway/test_still_working_notifications.py tests/gateway/test_background_process_notifications.py tests/gateway/test_gateway_inactivity_timeout.py tests/gateway/test_display_config.py tests/hermes_cli/test_config.py -q
  • result: 127 passed in 27.40s

@thesammygit

Copy link
Copy Markdown
Author

merged current origin/main into this branch to keep the heartbeat/config/docs/test area current with main.

what changed:

  • merged origin/main into sam/signal-heartbeat-config
  • accepted the clean merge for the touched gateway/config/docs/tests files with no manual conflict edits needed

validation:

  • source .venv/bin/activate && uv run --active pytest tests/gateway/test_still_working_notifications.py tests/hermes_cli/test_config.py -q
  • result: 66 passed in 1.25s

@thesammygit

Copy link
Copy Markdown
Author

merged current origin/main into this branch and fixed one stale test expectation from the touched config/test area.

what changed:

  • merged origin/main into sam/signal-heartbeat-config
  • updated tests/hermes_cli/test_config.py to assert against DEFAULT_CONFIG["_config_version"] after main moved the config schema to v19

validation:

  • uv run --active pytest tests/gateway/test_still_working_notifications.py tests/hermes_cli/test_config.py -q
  • result: 66 passed in 1.21s

@thesammygit

Copy link
Copy Markdown
Author

merged current origin/main into this branch to keep the heartbeat/config/docs/test area current with main.

what changed:

  • merged origin/main into sam/signal-heartbeat-config
  • accepted the clean merge; the touched pr file that moved on main was gateway/run.py

validation:

  • uv run --active pytest tests/gateway/test_still_working_notifications.py tests/gateway/test_background_process_notifications.py tests/gateway/test_gateway_inactivity_timeout.py tests/gateway/test_display_config.py tests/hermes_cli/test_config.py -q
  • result: 127 passed in 26.62s

@thesammygit

thesammygit commented Apr 19, 2026

Copy link
Copy Markdown
Author

merged current origin/main into this branch to keep the heartbeat/config area current with main.

what changed:

  • merged origin/main into sam/signal-heartbeat-config
  • accepted the clean merge after main moved both gateway/run.py and hermes_cli/config.py

validation:

  • source .venv/bin/activate && uv run --active pytest tests/gateway/test_still_working_notifications.py tests/gateway/test_background_process_notifications.py tests/gateway/test_gateway_inactivity_timeout.py tests/gateway/test_display_config.py tests/hermes_cli/test_config.py -q
  • result: 127 passed in 26.88s

@thesammygit

Copy link
Copy Markdown
Author

merged current origin/main into this branch to keep the heartbeat/config/docs/test area current with main.

what changed:

  • merged origin/main into sam/signal-heartbeat-config
  • accepted the clean merge after main moved files in the touched heartbeat/config/docs/test area

validation:

  • uv run --active pytest tests/gateway/test_still_working_notifications.py tests/gateway/test_background_process_notifications.py tests/gateway/test_gateway_inactivity_timeout.py tests/gateway/test_display_config.py tests/hermes_cli/test_config.py -q
  • result: 127 passed in 26.68s

@thesammygit

thesammygit commented Apr 19, 2026

Copy link
Copy Markdown
Author

merged current origin/main into this branch to keep the heartbeat/config/docs/test area current with main.

what changed:

  • merged origin/main into sam/signal-heartbeat-config
  • accepted the clean merge after main moved the touched heartbeat/config/docs/test files, including gateway/run.py, hermes_cli/config.py, the focused tests, and the docs pages

validation:

  • source .venv/bin/activate && uv run --active pytest tests/gateway/test_still_working_notifications.py tests/gateway/test_background_process_notifications.py tests/gateway/test_gateway_inactivity_timeout.py tests/gateway/test_display_config.py tests/hermes_cli/test_config.py -q
  • result: 127 passed in 26.63s

@thesammygit

Copy link
Copy Markdown
Author

merged current origin/main into sam/signal-heartbeat-config to keep the gateway heartbeat/config/docs/tests branch current after origin/main moved gateway/run.py, hermes_cli/config.py, and website/docs/user-guide/messaging/index.md.

what changed:

  • merged origin/main at 5d3be89 into the pr branch
  • clean merge; no unresolved conflict markers in the pr-touched files

validation:

  • uv run --active python -m pytest tests/gateway/test_still_working_notifications.py tests/hermes_cli/test_config.py -q → 69 passed in 2.16s

@thesammygit

thesammygit commented May 6, 2026

Copy link
Copy Markdown
Author

merged current origin/main into sam/signal-heartbeat-config to keep the gateway heartbeat/config/docs/tests branch current after origin/main moved the pr-touched files.

what changed:

  • merged origin/main at cd2cbc7 into the pr branch and pushed merge commit 2b503c1
  • merge was clean; no line-anchored conflict markers in the pr-touched files

validation:

  • command (sanitized): source .venv/bin/activate && uv run --active python -m pytest tests/gateway/test_still_working_notifications.py tests/gateway/test_background_process_notifications.py tests/gateway/test_gateway_inactivity_timeout.py tests/gateway/test_display_config.py tests/hermes_cli/test_config.py -q
  • result: 131 passed, 8 warnings in 28.97s

@thesammygit

thesammygit commented May 8, 2026

Copy link
Copy Markdown
Author

merged current origin/main into sam/signal-heartbeat-config to keep the gateway heartbeat/config/docs/tests branch current after origin/main moved the pr-touched files.

what changed:

  • merged origin/main at 839cdd1 into the fork branch (merge commit 390e8dd)
  • merge was clean; line-anchored conflict marker scan over the pr-touched files found none
  • validation-induced uv.lock dirt was discarded before pushing

validation:

  • source .venv/bin/activate && uv run --active python -m pytest tests/gateway/test_still_working_notifications.py tests/gateway/test_background_process_notifications.py tests/gateway/test_gateway_inactivity_timeout.py tests/gateway/test_display_config.py tests/hermes_cli/test_config.py -q
  • result: 137 passed in 29.04s

@thesammygit

Copy link
Copy Markdown
Author

merged current origin/main into sam/signal-heartbeat-config to keep the gateway heartbeat/config/docs/tests branch current after main touched overlapping files.

validation: uv run --active python -m pytest tests/gateway/test_still_working_notifications.py tests/hermes_cli/test_config.py -q → 69 passed.

@thesammygit

Copy link
Copy Markdown
Author

cron watcher update:

  • merged current origin/main into sam/signal-heartbeat-config to keep the pr branch current after upstream touched the same gateway/config/test/docs files.
  • validation: source .venv/bin/activate && uv run --active python -m pytest tests/gateway/test_still_working_notifications.py tests/hermes_cli/test_config.py -q -> 72 passed.

@thesammygit

Copy link
Copy Markdown
Author

maintenance update: merged current origin/main into sam/signal-heartbeat-config after main moved through the same gateway/config/docs areas touched by this PR.

validation: uv run --active python -m pytest tests/gateway/test_still_working_notifications.py tests/hermes_cli/test_config.py -q -> 72 passed.

pushed merge commit 5af6fbecb.

@thesammygit

Copy link
Copy Markdown
Author

merged current origin/main into sam/signal-heartbeat-config because upstream moved through files touched by this pr (gateway/run.py, hermes_cli/config.py, and website/docs/user-guide/configuration.md).

validation: uv run --active python -m pytest tests/gateway/test_still_working_notifications.py tests/hermes_cli/test_config.py -q -> 72 passed.

@teknium1

Copy link
Copy Markdown
Contributor

Thanks for carrying the per-platform cadence idea through repeated gateway/config changes. As @alt-glitch noted, #8572 already landed the global interval; current main also already supports per-platform heartbeat suppression via display.platforms.<platform>.long_running_notifications (gateway/run.py:19222-19228, gateway/display_config.py:191-225).

Problems

  • The added loader reads import-time _hermes_home, but multiplexed turns enter _profile_runtime_scope before loading the per-turn config (gateway/run.py:16826-16920; _hermes_home is initialized at gateway/run.py:1301-1305). Secondary-profile settings would be ignored.
  • The patch restores a 600-second effective fallback, while current main intentionally defaults to 180 seconds in both hermes_cli/config.py:1124-1131 and gateway/run.py:19215-19221.

Suggested changes

  • Rework cadence resolution to consume the already loaded user_config and extend the existing display.platforms resolver rather than add a raw-file loader/parallel override namespace.
  • Preserve the current 180-second fallback and add multiplex-profile regression coverage.

This is an automated hermes-sweeper review.

@alt-glitch alt-glitch added P3 Low — cosmetic, nice to have sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages labels Jul 12, 2026
@teknium1 teknium1 added sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform labels Jul 12, 2026
@alt-glitch alt-glitch removed the sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state label Jul 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/config Config system, migrations, profiles comp/gateway Gateway runner, session dispatch, delivery P3 Low — cosmetic, nice to have sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants