test: E2E tests for Admin API and Responses API by pranavraja99 · Pull Request #2174 · nearai/ironclaw

pranavraja99 · 2026-04-08T22:39:58Z

Summary

Adds pytest E2E scenarios for:

Admin API (test_admin_api.py): user CRUD (create, list, get, update), suspend/activate, secret lifecycle (create, list, delete), delete + verify 404
Responses API (test_responses_api.py): non-streaming (text + messages input), multi-turn via previous_response_id, GET by ID, streaming (OpenAI client + raw SSE), context injection (approval/rejection), error cases

Also adds SECRETS_MASTER_KEY to the ironclaw_server fixture so admin secret provisioning tests work.

Test plan

cd tests/e2e && source .venv/bin/activate
pytest scenarios/test_admin_api.py scenarios/test_responses_api.py -v

🤖 Generated with Claude Code

- test_admin_api.py: user CRUD, suspend/activate, secret lifecycle, delete - test_responses_api.py: non-streaming, streaming, multi-turn, GET by ID, context injection, error cases (no auth, empty input) - SECRETS_MASTER_KEY added to ironclaw_server fixture Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

Adds new E2E scenario coverage for the Admin API and OpenAI-compatible Responses API, and updates the E2E server fixture environment to support secrets provisioning.

Changes:

Add Admin API E2E scenarios for user CRUD, suspend/activate, and secrets lifecycle.
Add Responses API E2E scenarios for non-streaming, multi-turn, retrieval, streaming (SDK + raw SSE), and error cases.
Update E2E ironclaw_server fixture env (adds SECRETS_MASTER_KEY) and remove some fixtures/waits from conftest.py.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.

File	Description
`tests/e2e/scenarios/test_admin_api.py`	New E2E coverage for admin user/secrets endpoints.
`tests/e2e/scenarios/test_responses_api.py`	New E2E coverage for `/v1/responses` including streaming and error cases.
`tests/e2e/conftest.py`	Adjusts server env for secrets; removes SSE wait and removes Slack E2E fixtures.

Comments suppressed due to low confidence (1)

tests/e2e/conftest.py:1145

The Slack E2E fixtures (fake_slack_server / slack_e2e_server) were removed from conftest.py, but there are still scenarios that depend on them (e.g. tests/e2e/scenarios/test_slack_e2e.py references slack_e2e_server). This will cause the Slack E2E tests to fail at collection time. Either restore these fixtures or remove/update the dependent Slack scenarios in the same PR.



# ── Telegram E2E fixtures ────────────────────────────────────────────────

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-08T22:43:12Z

    await pg.goto(f"{ironclaw_server}/?token={AUTH_TOKEN}")
    # Wait for the app to initialize (auth screen hidden, SSE connected)
    await pg.wait_for_selector("#auth-screen", state="hidden", timeout=15000)
-    # Wait for SSE connection (onopen sets sseHasConnectedBefore = true)
-    await pg.wait_for_function(
-        "() => typeof sseHasConnectedBefore !== 'undefined' && sseHasConnectedBefore === true",
-        timeout=10000,
-    )
    yield pg


The comment says "Wait for the app to initialize (auth screen hidden, SSE connected)" but the fixture no longer waits for SSE to connect (the sseHasConnectedBefore wait was removed). Either reintroduce an explicit SSE-ready wait here, or update the comment to match what the fixture actually guarantees to reduce flakiness/confusion in tests that rely on page being fully connected.

Copilot · 2026-04-08T22:43:13Z

+import httpx
+import pytest
+from openai import OpenAI
+
+from helpers import AUTH_TOKEN


These tests import OpenAI from the openai Python package, but tests/e2e/pyproject.toml doesn't list openai as a dependency (it currently includes pytest/httpx/playwright/etc.). Running the stated test plan in a fresh tests/e2e virtualenv will fail with ModuleNotFoundError. Add openai to the E2E project dependencies (or switch to direct httpx calls only) so the suite is self-contained.

Copilot · 2026-04-08T22:43:13Z

+@pytest.fixture()
+async def openai_client(responses_user):
+    """OpenAI client pointed at the test IronClaw instance."""
+    base_url, user_token = responses_user
+    return OpenAI(api_key=user_token, base_url=f"{base_url}/v1")
+


openai_client is used from async def tests, but it returns the synchronous OpenAI client. Its network calls (including streaming iteration) will block the pytest-asyncio event loop, which can lead to slow/flaky behavior when other async fixtures/tasks are running. Prefer AsyncOpenAI (or run sync calls in a thread via anyio.to_thread.run_sync) so the tests remain non-blocking.

Copilot · 2026-04-08T22:43:13Z

+            json={
+                "input": "Go ahead with the transfer",
+                "x_context": {
+                    "notification_response": {
+                        "notification_id": "msg_456",
+                        "action": "approved",
+                        "original_signal": "convert_now",
+                        "score": 72,
+                    }
+                },
+                "stream": False,
+            },


The tests labeled as "Context injection" send an x_context field, but the server's POST /v1/responses request struct (src/channels/web/responses_api.rs:48+) doesn't define or forward x_context into the agent loop (unknown fields are ignored by serde). As written, these tests will pass without actually validating context injection behavior. Use the real field name the API supports (if any) and/or add assertions that demonstrate the injected context changed the response, or update the PR description if context injection isn't actually covered here.

Copilot · 2026-04-08T22:43:13Z

+    r = await admin_client.post("/api/admin/users", json={
+        "display_name": "Delete Me",
+        "email": email,
+        "role": "member",
+    })
+    uid = r.json()["id"]
+


test_delete_user_and_verify_gone reads r.json()["id"] without first asserting the create-user request succeeded. If the POST fails (non-200 or non-JSON), this will raise and hide the real failure. Add a status_code (and ideally schema) assertion before accessing the JSON body, consistent with the other tests in this file.

gemini-code-assist

Code Review

This pull request introduces comprehensive E2E integration tests for the Admin and Responses APIs, covering user CRUD operations, secret management, and OpenAI-compatible streaming responses. It also refactors the test configuration by removing unused Slack fixtures and adding a master key for secrets encryption. Feedback was provided regarding the use of predictable hardcoded keys, potential flakiness from removing SSE connection waits, and the recommendation to use pytest fixtures for user lifecycle management in the new test files to improve isolation and cleanup.

gemini-code-assist · 2026-04-08T22:46:02Z

+    email = f"test-{uuid.uuid4().hex[:8]}@example.com"
+    r = await admin_client.post("/api/admin/users", json={
+        "display_name": "Create Test",
+        "email": email,
+        "role": "member",
+    })
+    assert r.status_code == 200
+    data = r.json()
+    assert "id" in data
+    assert "token" in data
+    assert data["status"] == "active"
+    assert data["role"] == "member"
+    # Cleanup
+    await admin_client.delete(f"/api/admin/users/{data['id']}")


To ensure test isolation and proper cleanup, use a pytest fixture to manage the creation and deletion of the test user instead of manual try...finally blocks. This eliminates setup code duplication and ensures resources are cleaned up even if assertions fail.

References

In pytest, use fixtures to achieve test isolation and eliminate setup code duplication, rather than relying on sequential test execution and shared class state.

gemini-code-assist · 2026-04-08T22:46:02Z

+    email = f"test-{uuid.uuid4().hex[:8]}@example.com"
+    r = await admin_client.post("/api/admin/users", json={
+        "display_name": "Delete Me",
+        "email": email,
+        "role": "member",
+    })
+    uid = r.json()["id"]
+
+    r = await admin_client.delete(f"/api/admin/users/{uid}")
+    assert r.status_code == 200
+    assert r.json()["deleted"] is True
+
+    r = await admin_client.get(f"/api/admin/users/{uid}")
+    assert r.status_code == 404


Similar to other user creation tests, use a pytest fixture to handle resource lifecycle. This ensures the test user is cleaned up regardless of the test outcome, following the repository's preference for fixtures to achieve test isolation.

References

In pytest, use fixtures to achieve test isolation and eliminate setup code duplication, rather than relying on sequential test execution and shared class state.

gemini-code-assist · 2026-04-08T22:46:02Z

        "HEARTBEAT_ENABLED": "false",
        "EMBEDDING_ENABLED": "false",
+        # Secrets encryption (needed for admin secret provisioning)
+        "SECRETS_MASTER_KEY": "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",


Using a weak, predictable secret for SECRETS_MASTER_KEY in tests is a security risk. This pattern could be accidentally copied into production code. It's better to use a more random key, even if it's hardcoded for tests. You can generate one using openssl rand -hex 32.

gemini-code-assist · 2026-04-08T22:46:02Z

@@ -1113,11 +1114,6 @@ async def page(ironclaw_server, browser):
    await pg.goto(f"{ironclaw_server}/?token={AUTH_TOKEN}")
    # Wait for the app to initialize (auth screen hidden, SSE connected)
    await pg.wait_for_selector("#auth-screen", state="hidden", timeout=15000)


Removing the explicit wait for the SSE connection could introduce flakiness. Instead of a fixed timeout, prefer using page.wait_for_function to poll for a specific condition that indicates the SSE connection is established. This ensures the test waits only as long as necessary for the expected state change.

await page.wait_for_function("window.sseReady === true")

References

In tests, prefer page.wait_for_function to poll for a specific condition instead of using a fixed wait_for_timeout.

* test(e2e): add Admin and Responses API scenarios Salvaged from #2174 commit 663a388. The original PR also edited tests/e2e/conftest.py, including removal of Slack E2E fixtures and SSE wait logic. Those stale fixture edits are intentionally omitted because current main still has Slack E2E tests and current conftest already configures SECRETS_MASTER_KEY for the shared server. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * test(e2e): avoid optional OpenAI SDK dependency --------- Co-authored-by: Pranav Raja <pranavraja99@gmail.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

serrrfirat · 2026-05-05T21:34:37Z

Salvaged via #3267 onto current main.

Your original work is credited in the salvage PR body, including the original commit SHA and co-author trailer. Stale conftest.py fixture edits were omitted because current main still needs the Slack E2E fixtures/SSE wait logic, and a maintainer follow-up adapted the Responses API tests to avoid the optional OpenAI Python SDK dependency. Thanks!

…earai#3267) * test(e2e): add Admin and Responses API scenarios Salvaged from nearai#2174 commit 663a388. The original PR also edited tests/e2e/conftest.py, including removal of Slack E2E fixtures and SSE wait logic. Those stale fixture edits are intentionally omitted because current main still has Slack E2E tests and current conftest already configures SECRETS_MASTER_KEY for the shared server. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * test(e2e): avoid optional OpenAI SDK dependency --------- Co-authored-by: Pranav Raja <pranavraja99@gmail.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Copilot AI review requested due to automatic review settings April 8, 2026 22:40

github-actions Bot added size: L 200-499 changed lines risk: low Changes to docs, tests, or low-risk modules contributor: new First-time contributor labels Apr 8, 2026

Copilot started reviewing on behalf of pranavraja99 April 8, 2026 22:40 View session

Copilot AI reviewed Apr 8, 2026

View reviewed changes

gemini-code-assist Bot reviewed Apr 8, 2026

View reviewed changes

pranavraja99 mentioned this pull request Apr 14, 2026

fix(responses-api): thread creation, GET by ID, streaming delta, context injection #2167

Merged

henrypark133 changed the base branch from staging to main May 1, 2026 06:16

serrrfirat mentioned this pull request May 5, 2026

test: salvage Admin and Responses API E2E coverage from #2174 #3267

Merged

serrrfirat closed this May 5, 2026

github-actions Bot mentioned this pull request May 24, 2026

chore(ironclaw): release v0.29.2 JZKK720/ironclaw#4

Open

Navigation Menu

Repository navigation

test: E2E tests for Admin API and Responses API - #2174

test: E2E tests for Admin API and Responses API#2174
pranavraja99 wants to merge 1 commit into
mainfrom
test/admin-responses-api-e2e

pranavraja99 commented Apr 8, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Apr 8, 2026

Uh oh!

Copilot AI Apr 8, 2026

Uh oh!

Copilot AI Apr 8, 2026

Uh oh!

Copilot AI Apr 8, 2026

Uh oh!

Copilot AI Apr 8, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

gemini-code-assist Bot Apr 8, 2026

Uh oh!

gemini-code-assist Bot Apr 8, 2026

Uh oh!

gemini-code-assist Bot Apr 8, 2026

Uh oh!

gemini-code-assist Bot Apr 8, 2026

Uh oh!

serrrfirat commented May 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

pranavraja99 commented Apr 8, 2026

Summary

Test plan

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

serrrfirat commented May 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants