-
Notifications
You must be signed in to change notification settings - Fork 46.7k
test(slack): add peer-agent smoke target #51637
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,226 @@ | ||
| """ | ||
| Repeatable smoke tests for Slack peer-agent routing invariants. | ||
|
|
||
| Run this file directly after Slack gateway config changes or deploys. | ||
| """ | ||
|
|
||
| import sys | ||
| from unittest.mock import AsyncMock, MagicMock | ||
|
|
||
| import pytest | ||
|
|
||
| from gateway.config import PlatformConfig | ||
|
|
||
|
|
||
| def _ensure_slack_mock(): | ||
| """Install mock slack modules so SlackAdapter can be imported.""" | ||
| if "slack_bolt" in sys.modules and hasattr(sys.modules["slack_bolt"], "__file__"): | ||
| return | ||
|
|
||
| slack_bolt = MagicMock() | ||
| slack_bolt.async_app.AsyncApp = MagicMock | ||
| slack_bolt.adapter.socket_mode.async_handler.AsyncSocketModeHandler = MagicMock | ||
|
|
||
| slack_sdk = MagicMock() | ||
| slack_sdk.web.async_client.AsyncWebClient = MagicMock | ||
|
|
||
| for name, mod in [ | ||
| ("slack_bolt", slack_bolt), | ||
| ("slack_bolt.async_app", slack_bolt.async_app), | ||
| ("slack_bolt.adapter", slack_bolt.adapter), | ||
| ("slack_bolt.adapter.socket_mode", slack_bolt.adapter.socket_mode), | ||
| ("slack_bolt.adapter.socket_mode.async_handler", slack_bolt.adapter.socket_mode.async_handler), | ||
| ("slack_sdk", slack_sdk), | ||
| ("slack_sdk.web", slack_sdk.web), | ||
| ("slack_sdk.web.async_client", slack_sdk.web.async_client), | ||
| ]: | ||
| sys.modules.setdefault(name, mod) | ||
|
|
||
| sys.modules.setdefault("aiohttp", MagicMock()) | ||
|
|
||
|
|
||
| _ensure_slack_mock() | ||
|
|
||
| import plugins.platforms.slack.adapter as _slack_mod | ||
|
|
||
| _slack_mod.SLACK_AVAILABLE = True | ||
|
|
||
| from plugins.platforms.slack.adapter import SlackAdapter # noqa: E402 | ||
|
|
||
|
|
||
| BOT_USER_ID = "U_TARGET_BOT" | ||
| PEER_USER_ID = "U_PEER_BOT" | ||
| TEAM_ID = "T_SMOKE" | ||
| CHANNEL_ID = "C_SMOKE" | ||
| THREAD_TS = "1700000000.100000" | ||
| REPLY_TS = "1700000000.200000" | ||
|
|
||
|
|
||
| def _make_event( | ||
| *, | ||
| text: str, | ||
| user: str, | ||
| ts: str, | ||
| thread_ts: str | None = None, | ||
| bot_id: str | None = None, | ||
| ) -> dict: | ||
| event = { | ||
| "text": text, | ||
| "user": user, | ||
| "channel": CHANNEL_ID, | ||
| "channel_type": "channel", | ||
| "team": TEAM_ID, | ||
| "ts": ts, | ||
| } | ||
| if thread_ts is not None: | ||
| event["thread_ts"] = thread_ts | ||
| if bot_id is not None: | ||
| event["bot_id"] = bot_id | ||
| return event | ||
|
|
||
|
|
||
| def _assert_peer_agent_preflight(adapter: SlackAdapter) -> None: | ||
| assert adapter._slack_require_mention() is True, ( | ||
| "config: slack.require_mention must be true for the peer-agent smoke target" | ||
| ) | ||
| assert adapter._slack_strict_mention() is True, ( | ||
| "config: slack.strict_mention must be true so stale thread state cannot wake peers" | ||
| ) | ||
| assert str(adapter.config.extra.get("allow_bots", "")).lower().strip() == "mentions", ( | ||
| "config: slack.allow_bots must be 'mentions' for peer-agent routing smoke" | ||
| ) | ||
| assert adapter._slack_allowed_channels() == set(), ( | ||
| "config: slack.allowed_channels must stay empty for the documented smoke profile" | ||
| ) | ||
| assert adapter._app is not None and getattr(adapter._app, "client", None) is not None, ( | ||
| "platform_connectivity: Slack client must be initialized before routing smoke runs" | ||
| ) | ||
| assert adapter._bot_user_id, ( | ||
| "bot_identity: Slack auth_test must resolve a bot user id before routing smoke runs" | ||
| ) | ||
|
|
||
|
|
||
| @pytest.fixture() | ||
| def smoke_adapter(): | ||
| config = PlatformConfig( | ||
| enabled=True, | ||
| token="xoxb-fake-token", | ||
| extra={ | ||
| "require_mention": True, | ||
| "strict_mention": True, | ||
| "allow_bots": "mentions", | ||
| "allowed_channels": "", | ||
| }, | ||
| ) | ||
| adapter = SlackAdapter(config) | ||
| adapter._app = MagicMock() | ||
| adapter._app.client = AsyncMock() | ||
| adapter._bot_user_id = BOT_USER_ID | ||
| adapter._team_bot_user_ids = {TEAM_ID: BOT_USER_ID} | ||
| adapter._channel_team = {} | ||
| adapter._running = True | ||
| adapter.handle_message = AsyncMock() | ||
| adapter._resolve_user_name = AsyncMock(return_value="Smoke User") | ||
| adapter._fetch_thread_context = AsyncMock(return_value="") | ||
| adapter._fetch_thread_parent_text = AsyncMock(return_value="Parent mention") | ||
| return adapter | ||
|
|
||
|
|
||
| class TestSlackPeerAgentSmoke: | ||
| def test_peer_agent_smoke_preflight_contract(self, smoke_adapter): | ||
| _assert_peer_agent_preflight(smoke_adapter) | ||
|
|
||
| @pytest.mark.asyncio | ||
| async def test_human_message_with_current_mention_routes(self, smoke_adapter): | ||
| event = _make_event( | ||
| text=f"<@{BOT_USER_ID}> summarize the deploy status", | ||
| user="U_HUMAN", | ||
| ts=REPLY_TS, | ||
| ) | ||
|
|
||
| await smoke_adapter._handle_slack_message(event) | ||
|
|
||
| smoke_adapter.handle_message.assert_awaited_once() | ||
| msg_event = smoke_adapter.handle_message.await_args.args[0] | ||
| assert msg_event.text == "summarize the deploy status", ( | ||
| "routing_logic: human @mentions must route and strip the current target mention" | ||
| ) | ||
| assert msg_event.source.thread_id == REPLY_TS | ||
| smoke_adapter._fetch_thread_context.assert_not_awaited() | ||
|
|
||
| @pytest.mark.asyncio | ||
| async def test_peer_bot_without_current_mention_is_ignored_despite_thread_state( | ||
| self, smoke_adapter | ||
| ): | ||
| smoke_adapter._bot_message_ts.add(THREAD_TS) | ||
| smoke_adapter._mentioned_threads.add(THREAD_TS) | ||
| smoke_adapter._has_active_session_for_thread = MagicMock(return_value=True) | ||
|
|
||
| event = _make_event( | ||
| text="status: work finished", | ||
| user=PEER_USER_ID, | ||
| bot_id="B_PEER", | ||
| ts=REPLY_TS, | ||
| thread_ts=THREAD_TS, | ||
| ) | ||
|
|
||
| await smoke_adapter._handle_slack_message(event) | ||
|
|
||
| smoke_adapter.handle_message.assert_not_called() | ||
| smoke_adapter._fetch_thread_context.assert_not_awaited() | ||
|
|
||
| @pytest.mark.asyncio | ||
| async def test_peer_bot_with_current_explicit_mention_routes(self, smoke_adapter): | ||
| smoke_adapter._bot_message_ts.add(THREAD_TS) | ||
| smoke_adapter._has_active_session_for_thread = MagicMock(return_value=False) | ||
|
|
||
| event = _make_event( | ||
| text=f"<@{BOT_USER_ID}> please take over this thread", | ||
| user=PEER_USER_ID, | ||
| bot_id="B_PEER", | ||
| ts=REPLY_TS, | ||
| thread_ts=THREAD_TS, | ||
| ) | ||
|
|
||
| await smoke_adapter._handle_slack_message(event) | ||
|
|
||
| smoke_adapter.handle_message.assert_awaited_once() | ||
| msg_event = smoke_adapter.handle_message.await_args.args[0] | ||
| assert msg_event.text == "please take over this thread", ( | ||
| "routing_logic: explicit peer-bot @mentions must still route in allow_bots=mentions mode" | ||
| ) | ||
| assert smoke_adapter._mentioned_threads == set(), ( | ||
| "routing_logic: strict peer-agent mode must not persist thread mentions after routing" | ||
| ) | ||
| smoke_adapter._fetch_thread_context.assert_awaited_once() | ||
|
|
||
| @pytest.mark.asyncio | ||
| @pytest.mark.parametrize( | ||
| ("text", "case_id"), | ||
| [ | ||
| ("ack: sent the summary", "ack"), | ||
| ("status: waiting for approval", "status"), | ||
| ("error: tool call failed", "error"), | ||
| ], | ||
| ids=["ack", "status", "error"], | ||
| ) | ||
| async def test_passive_peer_bot_messages_do_not_route( | ||
| self, smoke_adapter, text, case_id | ||
| ): | ||
| smoke_adapter._bot_message_ts.add(THREAD_TS) | ||
| smoke_adapter._mentioned_threads.add(THREAD_TS) | ||
| smoke_adapter._has_active_session_for_thread = MagicMock(return_value=True) | ||
|
|
||
| event = _make_event( | ||
| text=text, | ||
| user=PEER_USER_ID, | ||
| bot_id="B_PEER", | ||
| ts=REPLY_TS, | ||
| thread_ts=THREAD_TS, | ||
| ) | ||
|
|
||
| await smoke_adapter._handle_slack_message(event) | ||
|
|
||
| assert smoke_adapter.handle_message.await_count == 0, ( | ||
| f"routing_logic: passive peer bot {case_id} messages must never create bot-to-bot loops" | ||
| ) | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -394,6 +394,35 @@ Set this to `true` in busy workspaces where Slack's default "the bot remembers t | |
| Slack supports both patterns: `@mention` required to start a conversation by default, but you can opt specific channels out via `SLACK_FREE_RESPONSE_CHANNELS` (comma-separated channel IDs) or `slack.free_response_channels` in `config.yaml`. Once the bot has an active session in a thread, subsequent thread replies do not require a mention. In DMs the bot always responds without needing a mention. | ||
| ::: | ||
|
|
||
| ### Peer-Agent Smoke Check | ||
|
|
||
| For multi-bot Slack deployments that rely on strict per-turn mentions, keep the following profile: | ||
|
|
||
| ```yaml | ||
| slack: | ||
| require_mention: true | ||
| strict_mention: true | ||
| allow_bots: mentions | ||
| allowed_channels: "" | ||
| ``` | ||
|
|
||
| After gateway config changes, deploys, or restarts, run this synthetic smoke target: | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The target constructs its own passing config, mocked client, and bot identity, so it cannot validate a deployed gateway after a restart or distinguish those runtime failure buckets. Please either narrow this wording to an adapter regression test or add a runtime-aware check. |
||
|
|
||
| ```bash | ||
| uv run --frozen pytest -q tests/gateway/test_slack_peer_agent_smoke.py -o addopts='' | ||
| ``` | ||
|
|
||
| This target uses in-process synthetic Slack events only. It does not send live Slack messages and does not require real bot tokens by default. | ||
|
|
||
| Failure buckets: | ||
|
|
||
| - `config:` `test_peer_agent_smoke_preflight_contract` caught a profile mismatch (`require_mention`, `strict_mention`, `allow_bots`, or `allowed_channels`). | ||
| - `platform_connectivity:` the adapter/client was not initialized, so routing smoke is not a trustworthy signal yet. | ||
| - `bot_identity:` the adapter never resolved its bot user ID, so current-message mention checks cannot work. | ||
| - `routing_logic:` the Slack adapter regressed on one of the peer-agent invariants (human mention routing, peer-bot ignore, explicit peer mention admit, or passive ack/status/error suppression). | ||
|
|
||
| If this target passes but a live workspace still misroutes messages, investigate Slack token/workspace connectivity and runtime deployment state outside the routing logic itself. | ||
|
|
||
| ### Channel allowlist (`allowed_channels`) | ||
|
|
||
| Restrict the bot to a fixed set of Slack channels — useful when the bot is invited to many channels but should only respond in a few. When set, messages from channels NOT in this list are **silently ignored**, even if the bot is `@mentioned`. | ||
|
|
||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This only exercises the already-recognized
bot_idevent shape. Please add the ordinary peer bot-user event shape (auserID withoutbot_idorsubtype) if this smoke target is meant to cover the routing incident described by #51627.