Skip to content

fix(gateway): keep a served profile's delivery-ledger rows in the launch state.db - #119974

Closed
jonpol01 wants to merge 1 commit into
NousResearch:mainfrom
jonpol01:sweep/gwcore-ledger-secondary-profile-store-split
Closed

jonpol01 wants to merge 1 commit into
NousResearch:mainfrom
jonpol01:sweep/gwcore-ledger-secondary-profile-store-split

Conversation

@jonpol01

Copy link
Copy Markdown
Contributor

What does this PR do?

On a multiplexed gateway, each served profile's adapter is connected inside that profile's _profile_runtime_scope. The receive loop the adapter starts while connecting inherits the profile's home override, so every final reply that bot sends is recorded from that context. The delivery ledger resolved its path with get_hermes_home(), which follows the override, so those rows landed in profiles/<name>/state.db.

The boot sweep (sweep_recoverable) and the boot flood-timer arming (pending_retries) run in the launch context and open the launch state.db, so they never saw the rows. A served bot's reply cut off by a crash or SIGKILL between finalize and the platform's ACK was never redelivered. A flood-refused reply whose wait spanned a restart was never retried. resume_pending stayed set for a conversation whose answer was in the ledger.

The ledger is already designed as one shared store: the boot sweep scopes rows by (platform, adapter_profile), and the profile purge terminalizes rows in the shared store. Only the path resolver was out of step. This PR resolves _db_path() from get_process_hermes_home(), which is how gateway/status.py resolves the gateway's other process-level files.

I left out the HERMES_HOME-unset fallback to get_hermes_home() that lifecycle_ledger uses on purpose. A default gateway started in the foreground has no HERMES_HOME, and that fallback would follow the override again. The test covers that case.

Rows an earlier build already wrote to a profile's state.db stay where they are. This PR does not migrate them.

Related Issue

Fixes #119973

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)
  • ✨ New feature (non-breaking change that adds functionality)
  • 🔒 Security fix
  • 📝 Documentation update
  • ✅ Tests (adding or improving test coverage)
  • ♻️ Refactor (no behavior change)
  • 🎯 New skill (bundled or hub)

Changes Made

  • gateway/delivery_ledger.py: _db_path() now resolves from get_process_hermes_home() instead of get_hermes_home(). The get_hermes_home import this left unused is gone.
  • tests/gateway/test_delivery_ledger_process_home.py (new):
    • Records a final reply through the real BasePlatformAdapter._record_delivery_obligation, from a task created inside the real _profile_runtime_scope, then kills the owner.
    • Runs GatewayRunner._redeliver_pending_obligations from the launch context. It checks that the served profile's adapter sends the reply with the recovered-reply marker and that resume_pending is cleared.
    • Is parametrized over HERMES_HOME set and unset (the platform default).
    • Leaves _db_path unpatched. Every other ledger test replaces it, which is why the existing suite never saw this bug.
  • website/docs/user-guide/messaging/index.md: one sentence on where the ledger lives on a multiplexed gateway.

How to Test

  1. Run pytest tests/gateway/test_delivery_ledger_process_home.py -q: 2 passed. With gateway/delivery_ledger.py from main, both cases fail with assert 0 == 1, because the boot sweep claims nothing.
  2. Replace the new line with the lifecycle_ledger pattern (get_process_hermes_home() if HERMES_HOME is set else get_hermes_home()). Only the platform-default case fails.
  3. Run the repro script from the issue:
    • On main, the row lands in profiles/research/state.db and the boot sweep claims [].
    • On this branch, the row lands in the launch state.db and is claimed as ('ob-1', 'research', True).
  4. End to end, I used a real BasePlatformAdapter subclass whose connect() spawns its poll loop, connected through _connect_initial_adapter_with_timeout inside _profile_runtime_scope. The inbound message went through handle_message, and a send never returned. On this branch the row is in the launch store, and _claim_pending_obligations claims it with the marker and clears resume_pending, the same as a single-profile gateway.

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix/feature (no unrelated commits)
  • I've run pytest tests/ -q and all tests pass. I did not run the full suite. I ran these files one at a time, all green:
    • tests/gateway/: test_delivery_ledger_process_home.py, test_delivery_ledger.py, test_delivery_ledger_producer.py, test_delivery_ledger_single_connection.py, test_delivery_ledger_fd_leak.py, test_queued_final_ledger.py, test_delivery_flood_invariants.py, test_diagnostic_wake_presentation.py, test_discord_missed_message_backfill.py, test_multiplex_adapter_registry.py, test_owner_liveness_drift.py, test_platform_reconnect.py, test_restart_notice_replay.py, test_restart_resume_pending.py, test_silent_partial_delivery.py, test_profile_isolation_runtime.py
    • tests/hermes_state/: test_journal_mode_config.py, test_purge_profile_state.py, test_rekey_profile_state.py
    • tests/hermes_cli/: test_sqlite_util_canonical.py, test_session_recovery.py
  • I've added tests for my changes (required for bug fixes, strongly encouraged for features)
  • I've tested on my platform: macOS 27.0

Documentation & Housekeeping

  • I've updated relevant documentation (README, docs/, docstrings) — or N/A
  • I've updated cli-config.yaml.example if I added/changed config keys — or N/A
  • I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A
  • I've considered cross-platform impact (Windows, macOS) per the compatibility guide — or N/A. The path comes from the existing get_process_hermes_home() helper, which already handles the per-platform default home.
  • I've updated tool descriptions/schemas if I changed tool behavior — or N/A

Screenshots / Logs

Repro script from the issue, on main:

launch state.db : <no file>
profile state.db: [('ob-1', 'attempting', 'research')]
boot sweep claimed: []

On this branch:

launch state.db : [('ob-1', 'attempting', 'research')]
profile state.db: <no file>
boot sweep claimed: [('ob-1', 'research', True)]

…nch state.db

A multiplexed gateway connects each served profile's adapter inside
_profile_runtime_scope(<profile home>). The receive loop an adapter starts
while connecting inherits that home override, so every final reply the bot
sends is recorded from it. The ledger resolved its path through
get_hermes_home(), which follows the override, and the rows landed in
profiles/<name>/state.db. The boot sweep (sweep_recoverable) and the boot
flood-timer arming (pending_retries) run in the launch context and open the
launch state.db, so they never saw those rows. A served bot's reply cut off by
a crash or SIGKILL between finalize and platform ACK was never redelivered, a
flood-refused reply that spanned a restart was never retried, and
resume_pending was not cleared for a session whose answer sat in the ledger.

The ledger is meant to be one shared store: the boot sweep already scopes rows
by (platform, adapter_profile), and the profile purge terminalizes rows in the
shared store. _db_path now resolves from get_process_hermes_home(), as the
gateway's other process-level files do (gateway.status). It deliberately skips
the get_hermes_home() fallback that lifecycle_ledger uses when HERMES_HOME is
unset: a default gateway started in the foreground has no HERMES_HOME, and that
fallback would follow the override again.

Rows an earlier build already wrote to a profile's state.db stay where they are.
@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/gateway Gateway runner, session dispatch, delivery area/profiles Multi-profile isolation, HERMES_HOME scoping sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state labels Sep 23, 2026
@teknium1

Copy link
Copy Markdown
Collaborator

Landed on main via #120103 (merge bd970b0588), cherry-picked with your authorship preserved — thank you @jonpol01. Closing this PR since its commits are now on main; the follow-up fixes and tests from review are in #120103's body.

@teknium1 teknium1 closed this Sep 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/profiles Multi-profile isolation, HERMES_HOME scoping comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state type/bug Something isn't working

Projects

None yet

3 participants