Skip to content

fix(slack): integrate durable Ring 2 intake - #110466

Open
m0ntydad0n wants to merge 1 commit into
NousResearch:mainfrom
m0ntydad0n:fix/ring2-current-main-integration
Open

m0ntydad0n wants to merge 1 commit into
NousResearch:mainfrom
m0ntydad0n:fix/ring2-current-main-integration

Conversation

@m0ntydad0n

Copy link
Copy Markdown

Summary

  • Add a durable, metadata-only Slack intake ledger with per-install HMAC correlation, bounded SQLite work, validated WAL handling, capacity controls, and degraded fallback receipts.
  • Integrate observer and claim ownership across message, edit, file-share fallback, reaction, retry, and duplicate paths without replacing current-main Slack behavior.
  • Scope transport identity by workspace/channel/event and semantic ownership by workspace/channel/original message timestamp; publish completion only after successful downstream dispatch.
  • Add deterministic concurrency, cancellation, retry, cross-entry, persistence, privacy, and 74-case adapter-wiring coverage.

Reliability contract

  • Same-receipt waiters share the owner outcome without allowing waiter cancellation to cancel the owner or clear its claim.
  • Failed, rejected, or cancelled owners release only their own claim so a healthy sibling can proceed.
  • Accepted edit/message/file/reaction siblings serialize on one semantic-message boundary and dispatch at most once.
  • Socket Mode remains available after acknowledgement-time ledger degradation while emitting bounded, fixed-vocabulary, metadata-only diagnostics.
  • Raw Slack message bodies and raw identifiers are not persisted in the intake ledger.

Verification

  • scripts/run_tests.sh --file-retries 0 tests/gateway/test_slack_intake_ledger.py tests/gateway/test_slack_intake_adapter_observability.py tests/gateway/test_slack_intake_adapter_wiring.py -W error -q — 392 passed.
  • Expanded Slack/current-main selection through scripts/run_tests.sh --file-retries 0 ... -q — 980 passed across 43 files at the canonical 28 workers.
  • Python compilation, compatibility-pointer check, commit diff check, final-newline check, and scoped sensitive-data scan passed.
  • The rebased patch has the same stable patch ID as the previously reviewed candidate.
  • Independent serial specification/integration and quality/security reviews passed on commit 160a46aca91c4592ec2cf06c9a356e6fe05c217a.

Local limitations

  • A repository-wide Python run on the earlier candidate timed out and was not counted as passing; it was not repeated after the current-main rebase.
  • npm run check and Ruff were unavailable in the local environment and were not counted as passing. No dependencies were installed to manufacture a green receipt.
  • No live Slack traffic, restart, canary, fault injection, or deployment was performed.

@m0ntydad0n

Copy link
Copy Markdown
Author

CI is currently waiting for maintainer approval to run workflows for this fork PR. The branch is rebased onto current main and the reviewed head is 160a46aca91c4592ec2cf06c9a356e6fe05c217a. Please approve the workflow runs when convenient.

@alt-glitch alt-glitch added type/feature New feature or request P3 Low — cosmetic, nice to have comp/gateway Gateway runner, session dispatch, delivery comp/plugins Plugin system and bundled plugins platform/slack Slack app adapter sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages labels Sep 14, 2026

@andrexibiza andrexibiza left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed exact head 160a46aca91c4592ec2cf06c9a356e6fe05c217a against base/current main ee4452991d17534aa561f31ee55596d082aa94e7. The overall direction is strong: the semantic-message ownership boundary, same-receipt waiter behavior, accepted-only-after-dispatch rule, metadata-only HMAC correlation, fail-closed HTTP intake, and opened-object/WAL verification are all materially better than a best-effort in-memory dedup layer. I also checked the new ledger end to end, the Slack adapter integration, current multiplex scoping contract, exact-head CI state, existing review state, commit attribution, and adjacent Slack/dedup work.

I found three blockers on this exact head:

  1. The ledger worker drops the routed profile context before resolving its storage path. _db_path() and _correlation_key() intentionally call get_hermes_home() at operation time, but _run_ledger_work() submits a bare callable to a ThreadPoolExecutor. Hermes's profile home is a ContextVar; a bare executor worker starts with an empty context. In a multiplex gateway, a secondary Slack adapter therefore enters the ledger worker without its _profile_runtime_scope, and the worker can resolve the process/default HERMES_HOME instead of the source profile. That collapses the exact owner proof at the persistence boundary: secondary-profile receipts and the correlation key can land in the default profile's runtime/slack-intake/. Current main already has the canonical solution in tools.thread_context.propagate_context_to_thread, and the multiplexing contract explicitly requires copy_context() across executor boundaries. Please capture the caller context before submission and add a real two-profile regression: process/default home A, context-local routed home B, invoke the async safe entry point, then prove only B's ledger/key changed and A remained untouched. Please audit the process-global fallback/degraded-health buffers at the same boundary so one profile's persistence outage cannot silently become another profile's diagnostic state.

  2. The durable path is POSIX-only but the PR does not define that as a supported-platform boundary. Every transaction enters _serialized_connection_lifecycle(), which hard-fails when fcntl.flock is unavailable; _watch_opened_object() likewise only has kqueue or Linux /proc+inotify implementations. On native Windows this means the claimed durable Ring 2 path is permanently unavailable: Socket Mode degrades to volatile fallback and HTTP intake fails closed. That is not a transient local-test limitation; it is the production control flow. Please either implement an equivalent Windows-safe serialization/object-identity path with adversarial tests, or make an explicit repository-supported platform decision and scope the feature/docs/tests accordingly. The current body presents this as general Slack intake durability and does not disclose a native-Windows exclusion.

  3. This introduces a new 3,253-line production godfile and a 4,807-line test file. gateway/slack_intake_ledger.py currently owns at least five separable responsibilities: correlation/key identity, private filesystem + WAL graph pinning, schema/migration/transactions, async admission/fallback health, and diagnostic snapshot/WAL parsing. The Hermes development contract's 2K invariant is a hard ceiling, not a target, and decomposition is still expected below it. Please split this before merge behind a narrow facade and split the test matrix along the same ownership boundaries. This is especially important here because the file is a security/reliability boundary: reviewability is part of the control, not cosmetics.

Topology / interlocks: I found no direct duplicate of this Ring 2 ledger. #62861 by irresi is complementary claim-lifecycle prior art: it fixes mark-before-success dedup ownership for WeCom and explicitly calls out Slack as a follow-up class; it does not collide with this PR's files. #85699 by jerryhjones is an older draft/non-mergeable conversational Kanban intake and does collide in plugins/platforms/slack/adapter.py; it owns a feature-specific durable intake source, not this transport-wide substrate, so it should be treated as adjacent/competing surface if revived rather than silently duplicated. #54535 by sk-holmes is also an older non-mergeable Slack adapter carrier; its read-only history work is distinct, but its profile/workspace isolation contract is directly relevant to the executor-scope finding above. #108914 already hit the same off-thread HERMES_HOME failure class in another subsystem and fixed it by carrying copy_context() into the worker. The repository-level architecture in #90866 describes the same defect shape: strong source ownership is established, then degraded at a projection edge before a persistence/mutation boundary consumes it.

Attribution: this head is one commit authored/committed by John Montgomery (m0ntydad0n). I did not find a co-author chain in the commit to preserve beyond that; the adjacent PRs above remain credited to their own contributors and should stay that way if any pieces are consolidated.

Verification state: the author's local receipts are substantial (392 passed focused; 980 passed across 43 files), and the branch is based exactly on current main. But exact-head hosted CI is not green: CI, Nix, and Docker are all action_required, the CI run exposes zero jobs, and there are no combined statuses on 160a46a. So those local results are useful development evidence, not an acceptance receipt for this exact object.

Once the profile context is preserved through the ledger worker, the Windows/platform boundary is closed, the two godfiles are decomposed, and exact-head hosted checks execute green, I think this is worth another pass. The core ownership model is promising; the remaining issues are all at boundaries where this layer has to be stricter than the code it is replacing.

Comment thread gateway/slack_intake_ledger.py Outdated
@@ -0,0 +1,3253 @@
"""Metadata-only durable Slack intake stage ledger.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: this lands a 3,253-line production godfile (with a 4,807-line sibling test file). The Hermes 2K invariant is a hard ceiling, and this module currently combines correlation/key identity, filesystem/WAL pinning, schema/migration/transactions, async admission/fallback health, and diagnostic snapshot parsing. Please split these ownership domains behind a narrow facade before merge, and split the test matrix along the same seams. For a persistence/security boundary, reviewability is part of the control.

Comment thread gateway/slack_intake_ledger.py Outdated
permits.release()

try:
future = executor.submit(work)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: this bare ThreadPoolExecutor.submit(work) drops the caller's ContextVars. _db_path() / _correlation_key() resolve get_hermes_home() inside this worker, while multiplex routing installs the selected profile through a context-local HERMES_HOME override. A secondary Slack profile can therefore persist into the process/default profile's ledger/key. Current main's tools.thread_context.propagate_context_to_thread exists for exactly this class. Capture the caller context before submit and add a two-profile regression proving routed profile B touches only B's ledger/key, never default A.

Comment thread gateway/slack_intake_ledger.py Outdated
connection lifetime removes that legitimate lifecycle race without adding
another filesystem member for strict diagnostic readers to encounter.
"""
if fcntl is None or not hasattr(fcntl, "flock"):

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking cross-platform boundary: every ledger transaction enters this lifecycle lock, so native Windows (no fcntl) can never use the durable path. _watch_opened_object() is also kqueue or Linux /proc+inotify only. Today Socket Mode therefore permanently degrades to process-local fallback and HTTP intake fails closed on Windows. Please provide a Windows-equivalent trusted serialization/object-identity path with exact tests, or explicitly narrow the supported-platform contract and docs; the PR currently claims general Slack intake durability.

@m0ntydad0n
m0ntydad0n force-pushed the fix/ring2-current-main-integration branch from 160a46a to b75a190 Compare September 14, 2026 17:37

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery comp/plugins Plugin system and bundled plugins P3 Low — cosmetic, nice to have platform/slack Slack app adapter sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants