fix(gateway): hydrate connection on per-connection webhook under multi-replica

[buremba]

What was broken

The Slack Preview flow (/lobu link <code> to bind a chat to an agent) dead-ends in prod. Tested live against the working lobu-crm @Lobster bot:

• In a channel: Slackbot returns "/lobu failed because the app did not respond", and the preview claim stays unconsumed in oauth_states (no binding created).
• The lobu-crm bot replies to mentions fine, so it's not a dead connection.

Root cause

The slash command (and the per-connection Event URL) hits /api/v1/webhooks/:connectionId → ChatInstanceManager.handleWebhook(), which looks up the Chat instance in the pod-local this.instances map and 404s if it isn't warm on the receiving pod. It's the only inbound path that doesn't hydrate the connection from the store first.

Prod runs app.replicaCount: 2. With ClientIP affinity, a webhook can land on the replica that never warmed this connection → instant 404. Slack events mostly survive because Slack retries 3x and eventually hits the warm pod, but a one-shot slash command doesn't tolerate that, so it surfaced as "app did not respond" + an unconsumed claim. It worked end-to-end when the app ran a single replica.

The /slack/events coordinator (handleAppWebhook) and postMessageToChannel already do the right thing — ensureConnectionRunning() before forwarding. handleWebhook just never adopted it.

Fix

handleWebhook now calls ensureConnectionRunning(connectionId) before giving up — the same store-backed lazy-start the coordinator and postMessageToChannel use. It's a no-op when the instance is already running, so the coordinator's existing pre-call stays harmless. Any replica can now serve any connection's inbound webhook.

Reproducer (red → green)

packages/server/src/gateway/__tests__/chat-instance-manager-slack.test.ts — "ChatInstanceManager.handleWebhook (multi-replica)":

• Red (without the fix): a cold-pod webhook returns 404, restartConnection never called.
• Green (with the fix): the cold-pod webhook hydrates from the store and is handled (200). A stopped connection still 404s (no auto-revive).

 2 pass / 0 fail   (handleWebhook multi-replica)

Remaining (separate follow-ups, not in this PR)

• Preview claim hardening: platform-scope the claim; check allowedSurfaces before deleting the code.
• previewMode provisioning seam (declare via lobu apply instead of a manual DB edit).
• The bot's AI-app DM is a threaded surface; Slack blocks slash commands there, so DM-based linking needs a message-based path. Channel linking works with this fix.

Live verification still owed

Prod runs the old code; the live /lobu link retest happens after this merges and Flux rolls out the new image.

Summary by CodeRabbit

• Bug Fixes

  • Enhanced webhook delivery reliability in multi-instance deployments. Chat connections are now automatically loaded on-demand when processing incoming webhooks, improving availability and responsiveness.

• Tests

  • Added test coverage for webhook handling in multi-replica scenarios, including connection startup and error handling.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 0b77c0c7-22bc-4af3-9376-277b40bf5df4

📥 Commits

Reviewing files that changed from the base of the PR and between 2eecc76 and cd36b33.

📒 Files selected for processing (2)

• packages/server/src/gateway/__tests__/chat-instance-manager-slack.test.ts
• packages/server/src/gateway/connections/chat-instance-manager.ts

---

📝 Walkthrough

Walkthrough

This PR enables multi-replica webhook delivery by making ChatInstanceManager.handleWebhook lazily start chat connections from the connection store when they are not already running on the current pod, instead of immediately returning 404.

Changes

Multi-replica webhook delivery

Layer / File(s)	Summary
handleWebhook lazy-start implementation packages/server/src/gateway/connections/chat-instance-manager.ts	handleWebhook now calls ensureConnectionRunning when the target connection is not found in memory, then re-fetches the instance from registry before proceeding with webhook handler resolution and dispatch.
Lazy-load webhook handler tests packages/server/src/gateway/__tests__/chat-instance-manager-slack.test.ts	Two new test cases verify that handleWebhook restarts a cold connection and invokes the stored webhook handler, returning the handler's response; and that a stopped connection that cannot be started yields HTTP 404.

Sequence Diagram

sequenceDiagram
  participant Client
  participant handleWebhook
  participant ensureConnectionRunning
  participant ConnectionStore
  participant WebhookHandler
  Client->>handleWebhook: POST webhook (connectionId)
  handleWebhook->>handleWebhook: lookup instance in memory
  alt instance not found
    handleWebhook->>ensureConnectionRunning: start connection from store
    ensureConnectionRunning->>ConnectionStore: fetch and boot connection
    ConnectionStore-->>ensureConnectionRunning: connection started
    handleWebhook->>handleWebhook: re-fetch instance from registry
  end
  handleWebhook->>WebhookHandler: invoke handler with instance
  WebhookHandler-->>handleWebhook: response
  handleWebhook-->>Client: HTTP response

Loading

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

Possibly related PRs

• lobu-ai/lobu#881: Both PRs change ChatInstanceManager to start/recover chat connections from the connection store, where the main PR's new handleWebhook lazy-start path likely triggers the startInstance/org-context boot logic fixed in the retrieved PR.

Poem

🐰 A webhook knocks on a distant pod door,
No instance inside—but no worry anymore!
We fetch and we start what once lay asleep,
Then handle the message, promises to keep.
Multi-replica dreams now run deep.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The description is comprehensive, covering what was broken, root cause analysis, the fix, test coverage, and known follow-ups. However, the Test plan section (required per template) is entirely missing and none of the validation checkboxes are marked.	Add a Test plan section with checked items from the template to document what validation was performed before submitting.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely describes the main change: enabling webhook hydration for multi-replica scenarios by connecting to the store-backed lazy-start mechanism.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/slack-preview-fix

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 ESLint

If the error stems from missing dependencies, add them to the package.json file. For unrecoverable errors (e.g., due to private dependencies), disable the tool in the CodeRabbit configuration.

ESLint skipped: no ESLint configuration detected in root package.json. To enable, add eslint to devDependencies.

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[buremba]

pi review — SHIP-WITH-NITS

The fix correctly adds the missing lazy hydration in handleWebhook (mirrors postMessageToChannel and the /slack/events coordinator). No auth bypass (signature verification still runs in the adapter after this point), no warm-path regression (warm instances skip DB/restart entirely), and the red→green reproducer is valid.

Nits — all pre-existing, none blocking, tracked as follow-ups:

• ensureConnectionRunning() is check-then-act with no per-connection in-flight guard, so two cold webhooks racing on the same pod can double-start. This already affects the coordinator and postMessageToChannel; this PR extends it to the per-connection route. Follow-up: a per-connection start lock.
• On a failed lazy start (exists but can't start) we return 404; the coordinator returns 503 for the same case. Minor.
• An unauthenticated request to a valid cold connection id now does a DB lookup/start before signature verification fails (extra pre-auth work, not a bypass).

Live /lobu link retest is owed after this rolls out to prod (it currently runs the old code).

[buremba]

bug_free 84, simplicity 93, slop 0, bugs 0, 0 blockers

Typecheck and unit passed. Canonical integration ran the affected chat-instance-manager Slack tests before later failing in untouched agent-routes-apply. [env] embedded Postgres init hit shared-memory exhaustion; exploratory narrow rerun of the affected test hit the same initdb shm failure before tests.

Full verdict JSON

{
  "bug_free_confidence": 84,
  "bugs": 0,
  "slop": 0,
  "simplicity": 93,
  "blockers": [],
  "change_type": "fix",
  "behavior_change_risk": "medium",
  "tests_adequate": true,
  "suggested_fixes": [],
  "notes": "Typecheck and unit passed. Canonical integration ran the affected chat-instance-manager Slack tests before later failing in untouched agent-routes-apply. [env] embedded Postgres init hit shared-memory exhaustion; exploratory narrow rerun of the affected test hit the same initdb shm failure before tests.",
  "categories": {
    "src": 17,
    "tests": 55,
    "docs": 0,
    "config": 0,
    "deps": 0,
    "migrations": 0,
    "ci": 0,
    "generated": 0
  }
}

Local review gate — branch protection can require the pi-review commit status. See docs/REVIEW_SCHEMA.md.

[buremba]

Correcting the record: this PR was opened on the belief it fixed the Slack /lobu link preview flow. It did not. That failure was a misconfigured Slash Command Request URL on the Slack app — it pointed at an old community.lobu.ai host (and the Interactivity URL pointed at a non-existent slack-lobu-preview connection). Both were corrected in the Slack app config, and /lobu link now works end-to-end (verified live: link → bind → agent reply).

This change is still valid, as multi-replica hardening: the per-connection webhook (/api/v1/webhooks/:id) now lazily hydrates the connection via ensureConnectionRunning instead of returning 404 when it is not warm on the receiving replica — matching what the /slack/events coordinator and postMessageToChannel already do. Scope is narrow: existing connections are warmed on every replica at boot (initialize()), so this only affects the window when a connection is created/restarted on one replica and a webhook hits another before it reconciles.