-
Notifications
You must be signed in to change notification settings - Fork 1.5k
[codex] Tighten auth flows and unify live canary coverage #2367
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
95 commits
Select commit
Hold shift + click to select a range
8b37eeb
ci: add live canary regression lanes
serrrfirat d6f5ec5
test: tighten live zizmor canary prompt
serrrfirat 78750c1
feat(auth): harden extension auth and unify canary lanes
ilblackdragon d25a4b3
merge: integrate upstream live canary lanes
ilblackdragon 87b6e50
refactor(canary): unify auth live canary framework
ilblackdragon 7b96690
fix(mcp): share stdio runtime state across user views
ilblackdragon 442877a
Merge remote-tracking branch 'origin/staging' into codex/auth-oauth-c…
ilblackdragon 793d021
fix(ci): mark root crate unpublished
ilblackdragon 19da65c
merge: sync origin/staging
ilblackdragon d59426e
fix(auth): address oauth canary review findings
serrrfirat bb78a0e
Merge remote-tracking branch 'origin/staging' into codex/auth-oauth-c…
ilblackdragon f4015be
refactor: unify canary runners, restore post-merge user-isolation reg…
ilblackdragon 65e78d9
Merge origin/codex/auth-oauth-canary-unification
ilblackdragon 0f0aaec
fix: resolve unbound variable error in live-canary dispatcher
nickpismenkov 86e8c0b
ci: enable live-canary workflow on PRs
nickpismenkov 3e9aa07
Merge remote-tracking branch 'origin/staging' into codex/auth-oauth-c…
ilblackdragon 1f4187f
Merge remote-tracking branch 'origin/codex/auth-oauth-canary-unificat…
ilblackdragon 1982ee8
Merge remote-tracking branch 'origin/staging' into codex/auth-oauth-c…
ilblackdragon e8dbbcf
ci: enable live-canary on both main and staging PRs
nickpismenkov e8ca597
ci: enable all canary lanes to run on pull requests
nickpismenkov e8bc940
Merge remote-tracking branch 'origin/staging' into codex/auth-oauth-c…
ilblackdragon 3e70291
fix: address PR #2367 Copilot review findings
ilblackdragon 089b070
Merge remote-tracking branch 'origin/codex/auth-oauth-canary-unificat…
ilblackdragon 0095afd
Merge remote-tracking branch 'origin/staging' into codex/auth-oauth-c…
ilblackdragon 5342489
fix(bridge): pass CredentialName as &str to setup instructions lookup
ilblackdragon ab17505
fix(e2e): unblock two auth-matrix canary tests
ilblackdragon d7b272c
fix(e2e): resolve remaining auth-matrix canary failures
ilblackdragon f9b1f0f
ci: keep only mock-backed canary lanes on PRs
ilblackdragon fe6ffeb
fix: deterministic replay
nickpismenkov 468d8a6
Merge branch 'staging' into codex/auth-oauth-canary-unification
nickpismenkov 7c21b70
ci: remove mission test from deterministic-replay lane
nickpismenkov 162160e
ci: remove persona tests from deterministic-replay lane
nickpismenkov 459f3bc
ci: temporarily enable public-smoke on PRs for testing
nickpismenkov d0a345f
ci: use existing ANTHROPIC_API_KEY secret for live canary
nickpismenkov 0269008
Merge remote-tracking branch 'origin/staging' into codex/auth-oauth-c…
nickpismenkov 68c6bba
fix: codestyle
nickpismenkov 2410ad2
Merge remote-tracking branch 'origin/staging' into codex/auth-oauth-c…
nickpismenkov 56d2e9e
style: apply cargo fmt
nickpismenkov d51c317
fix(e2e): update assertion to match new mock MCP response format
nickpismenkov b4688da
fix(e2e): accept response content as proof zizmor ran
nickpismenkov 4da8331
Merge remote-tracking branch 'origin/staging' into codex/auth-oauth-c…
nickpismenkov d10822c
fix: update auth_manager path in chat test helper
nickpismenkov 2679640
Merge remote-tracking branch 'origin/staging' into codex/auth-oauth-c…
nickpismenkov cff7c6a
ci: temporarily enable auth-live-seeded on PRs for testing
nickpismenkov 016fa73
ci: use repo-level secrets for auth-live-seeded
nickpismenkov dcc670f
fix(e2e): print mock LLM port before modifying app state
nickpismenkov 4430181
fix(e2e): fall back to default scopes when env var is empty
nickpismenkov 7dbfcac
feat(e2e): auth-live-seeded uses real OAuth flow instead of DB seeding
nickpismenkov 387a2c8
fix(e2e): complete OAuth flow for all Google extensions, not just Gmail
nickpismenkov c8bbd79
feat(e2e): support Notion MCP DCR credentials in auth-live-seeded
nickpismenkov dcc817d
Merge branch 'staging' into codex/auth-oauth-canary-unification
nickpismenkov 53ad644
fix(e2e): preflight-refresh Google access token before auth-live-seeded
nickpismenkov dffb7bd
fix(e2e): case-insensitive expected_text matching in auth-live-seeded
nickpismenkov c83856f
fix(e2e): add Gmail canned response + move non-sensitive vars from se…
nickpismenkov 8ae8cf3
ci: remove short-value secrets that corrupt CI logs
nickpismenkov c684d78
ci: add Notion DCR client secrets to auth-live-seeded workflow
nickpismenkov f6178ce
fix(e2e): add Notion preflight token refresh with proper User-Agent
nickpismenkov 3b7c38e
fix(e2e): use tool name as expected_text instead of canned response s…
nickpismenkov cff7ee7
ci: temporarily enable all canary lanes on PRs for testing
nickpismenkov 7baa2a5
ci: disable auth-browser-consent and private-oauth on PRs
nickpismenkov 99686e2
fix(ci): read LIVE_OPENAI_COMPATIBLE_BASE_URL from vars not secrets
nickpismenkov de1fa2e
fix variable
nickpismenkov c3fc1be
feat(e2e): add lifecycle canary tests for Gmail, Calendar, and Notion
nickpismenkov 8831659
fix(e2e): relax persona keyword checks + pre-install zizmor in CI
nickpismenkov 5e9ebda
ci: remove temporary PR triggers from all live canary lanes
nickpismenkov aa67f42
Merge remote-tracking branch 'origin/staging' into codex/auth-oauth-c…
nickpismenkov 974fe0e
fix(e2e): use tool_name_matches for negative recovery-loop assertions
nickpismenkov fe0dea0
Merge branch 'staging' into codex/auth-oauth-canary-unification
nickpismenkov b1443a1
Merge remote-tracking branch 'origin/staging' into codex/auth-oauth-c…
nickpismenkov 39c3ac3
fix(e2e): correct bearer token prefix in multi-user MCP assertion
nickpismenkov d93243b
fix(mcp): resolve per-user client at tool-call time to stop cross-ten…
nickpismenkov ab6bb11
infra(runner): add Railway-hosted self-hosted runner for private-oaut…
nickpismenkov dca10fe
fix(mcp): partition Mcp-Session-Id by (user_id, server_name)
nickpismenkov edeed74
fix(mcp): close activate-vs-remove TOCTOU on shared MCP servers
nickpismenkov ec5c2a7
fix(canary): materialise sensitive auth secrets to files, out of job env
nickpismenkov e618848
fix(oauth): make token-body parser content-type-aware + validate token
nickpismenkov 127f6ba
Merge branch 'staging' into codex/auth-oauth-canary-unification
nickpismenkov bf770c0
fix(runner): install libicu + kerberos + lttng deps for actions/runner
nickpismenkov 09c738e
feat(runner): RUNNER_FORCE_REREGISTER env for re-registration recovery
nickpismenkov a49cbb9
feat(runner): IRONCLAW_DB_B64 env for one-shot libsql DB bootstrap
nickpismenkov d23983b
feat(runner): IRONCLAW_DB_URL fallback when base64 env exceeds plan l…
nickpismenkov 32228d8
infra(runner): add seed-runner-db.sh for one-shot DB transfer
nickpismenkov 5d50c1f
fix(runner): install python3 + python3-dev for pyo3 build
nickpismenkov f0d8b29
Merge remote-tracking branch 'origin/staging' into codex/auth-oauth-c…
nickpismenkov 16fb498
fix(app): remove dead MCP_MAX_SESSIONS env-var parsing
nickpismenkov db36e3c
fix(extensions): clean up MCP client on tool-wrapper-construction fai…
nickpismenkov 0aa54dd
style: apply cargo fmt
nickpismenkov f4834d2
fix(oauth): route all error-response body reads through a single trun…
nickpismenkov fb2f4fa
fix(canary): skip drive_auth_gate_roundtrip until WASM pre-flight gat…
nickpismenkov 0df70e4
fix(canary,mcp,docs): address review findings + harden MCP registry i…
nickpismenkov 96639e9
fix(e2e,docs): scope live-token override to Google + grammar typo
nickpismenkov e605627
docs(canary): document repo-scope secrets (no env isolation today)
nickpismenkov 63b79fa
fix(runner): checkpoint WAL before copying DB in seed-runner-db.sh
nickpismenkov 13b7638
fix(mcp): three review findings on MCP registry / process / startup p…
nickpismenkov 23e0ac2
fix(mcp,canary): annotation-aware fingerprint + lock/await hygiene + …
nickpismenkov File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,142 @@ | ||
| # Live Canary Regression Lanes | ||
|
|
||
| IronClaw now has two complementary regression systems: | ||
|
|
||
| - deterministic CI, which replays committed tests and traces without depending | ||
| on real third-party providers for the main blocking path; | ||
| - live canaries, which use real providers, real browser consent flows, or | ||
| selected real LLM lanes to catch provider drift, refresh failures, release | ||
| upgrade problems, and auth regressions that mocks will miss. | ||
|
|
||
| The implementation lives in: | ||
|
|
||
| - `.github/workflows/test.yml` for the normal blocking test lanes; | ||
| - `.github/workflows/live-canary.yml` for scheduled and manual live lanes; | ||
| - `scripts/live-canary/run.sh` for lane dispatch; | ||
| - `scripts/live-canary/scrub-artifacts.sh` for artifact scanning; | ||
| - `scripts/live-canary/upgrade-canary.sh` for previous-release upgrade checks. | ||
|
|
||
| The auth-specific executors used by the unified live-canary wrapper are: | ||
|
|
||
| - `scripts/auth_canary/run_canary.py` | ||
| - `scripts/auth_live_canary/run_live_canary.py` (both seeded and browser-consent | ||
| flows; selected with `--mode {seeded,browser}`) | ||
|
|
||
| Their shared auth-lane framework lives in: | ||
|
|
||
| - `scripts/live_canary/common.py` | ||
| - `scripts/live_canary/auth_registry.py` | ||
| - `scripts/live_canary/auth_runtime.py` | ||
|
|
||
| Future auth canaries should extend that shared framework and the canonical | ||
| account guide rather than introducing another bespoke runner layout. | ||
|
|
||
| ## Lane Summary | ||
|
|
||
| | Lane | Scope | Runner | Trigger | Blocking | | ||
| | --- | --- | --- | --- | --- | | ||
| | `deterministic-replay` | Replays `tests/e2e_live*.rs` fixtures without live LLM calls | GitHub-hosted | PR/staging via `test.yml`; manual via `live-canary.yml` | Yes in `test.yml` | | ||
| | `public-smoke` | Real LLM plus public tools such as `zizmor_scan` and mission digest | GitHub-hosted | Daily and manual | Opens issue on scheduled failure | | ||
| | `persona-rotating` | Real LLM multi-turn persona workflow, one persona per day | GitHub-hosted | Daily and manual | Opens issue on scheduled failure | | ||
| | `private-oauth` | Google Drive auth gate and transparent refresh against a dedicated test account | Self-hosted `ironclaw-live` runner | Manual; scheduled only when enabled | Opens issue on scheduled failure | | ||
| | `provider-matrix` | Same live behavior against multiple provider adapters | GitHub-hosted | Weekly and manual | Opens issue on scheduled failure | | ||
| | `release-public-full` | Full public live suite for release candidates | GitHub-hosted | Manual | Release checklist gate | | ||
| | `upgrade-canary` | Previous release DB opened by current checkout | GitHub-hosted | Manual | Release checklist gate | | ||
| | `auth-smoke` | Fresh-machine mock-backed auth smoke: hosted OAuth, MCP OAuth, and multi-user MCP isolation | GitHub-hosted | Hourly and manual | No | | ||
| | `auth-full` | Larger mock-backed auth matrix including failure and refresh cases | GitHub-hosted | Manual | No | | ||
| | `auth-channels` | WASM channel auth diagnostic lane | GitHub-hosted | Manual | No | | ||
| | `auth-live-seeded` | Real-provider runtime checks using seeded tokens against a clean DB | GitHub-hosted | Hourly and manual | No | | ||
| | `auth-browser-consent` | Real browser-consent OAuth using Playwright against provider login UIs | GitHub-hosted | Nightly and manual | No | | ||
|
|
||
| ## Required Repository Configuration | ||
|
|
||
| ### Public live LLM lanes | ||
|
|
||
| Secrets: | ||
|
|
||
| - `LIVE_ANTHROPIC_API_KEY` | ||
| - `LIVE_OPENAI_COMPATIBLE_API_KEY` | ||
| - `LIVE_OPENAI_COMPATIBLE_BASE_URL` | ||
|
|
||
| Variables: | ||
|
|
||
| - `LIVE_ANTHROPIC_MODEL` | ||
| - `LIVE_OPENAI_COMPATIBLE_MODEL` | ||
| - `LIVE_CANARY_PRIVATE_OAUTH_ENABLED` | ||
|
|
||
| ### Auth live-seeded lane | ||
|
|
||
| Secrets and dedicated account material are documented in | ||
| [scripts/live-canary/ACCOUNTS.md](../../scripts/live-canary/ACCOUNTS.md). | ||
|
|
||
| Current provider material includes: | ||
|
|
||
| - Google OAuth client credentials and seeded access/refresh tokens | ||
| - GitHub seeded token plus a stable issue fixture | ||
| - Notion seeded access token and a stable query fixture | ||
|
|
||
| ### Auth browser-consent lane | ||
|
|
||
| Secrets and browser session material are documented in | ||
| [scripts/live-canary/ACCOUNTS.md](../../scripts/live-canary/ACCOUNTS.md). | ||
|
|
||
| Current provider material includes: | ||
|
|
||
| - Google OAuth app credentials plus browser storage state | ||
| - GitHub OAuth app credentials plus browser storage state and issue fixture | ||
| - Notion browser storage state | ||
|
|
||
| ## Commands | ||
|
|
||
| Run public live smoke locally: | ||
|
|
||
| ```bash | ||
| IRONCLAW_LIVE_TEST=1 \ | ||
| LLM_BACKEND=anthropic \ | ||
| ANTHROPIC_API_KEY=... \ | ||
| LANE=public-smoke \ | ||
| scripts/live-canary/run.sh | ||
| ``` | ||
|
|
||
| Run a private OAuth lane on the dedicated runner: | ||
|
|
||
| ```bash | ||
| LANE=private-oauth scripts/live-canary/run.sh | ||
| ``` | ||
|
|
||
| Run the auth smoke lane: | ||
|
|
||
| ```bash | ||
| LANE=auth-smoke scripts/live-canary/run.sh | ||
| ``` | ||
|
|
||
| Run the seeded auth live lane: | ||
|
|
||
| ```bash | ||
| LANE=auth-live-seeded scripts/live-canary/run.sh | ||
| ``` | ||
|
|
||
| Run the browser-consent auth lane: | ||
|
|
||
| ```bash | ||
| LANE=auth-browser-consent scripts/live-canary/run.sh | ||
| ``` | ||
|
|
||
| Run selected auth provider cases only: | ||
|
|
||
| ```bash | ||
| LANE=auth-live-seeded CASES=gmail,github scripts/live-canary/run.sh | ||
| LANE=auth-browser-consent CASES=google,github scripts/live-canary/run.sh | ||
| ``` | ||
|
|
||
| ## Artifact Policy | ||
|
|
||
| Artifacts are written under `artifacts/live-canary/`. | ||
|
|
||
| Before upload, the workflow runs `scripts/live-canary/scrub-artifacts.sh`. | ||
| That script is a guardrail against uploading obvious token-shaped strings from | ||
| logs or result files. | ||
|
|
||
| Private OAuth lanes should continue to avoid uploading raw OAuth logs. The | ||
| auth-browser-consent and auth-live-seeded lanes may capture screenshots and JSON | ||
| results, but should not upload long-lived credential material. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These links use an absolute local path (
/home/illia/ironclaw/...) so they'll be broken in GitHub-rendered docs. Use repo-relative links instead (e.g.,/scripts/live-canary/ACCOUNTS.md).There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed in
3e702910. Both occurrences indocs/internal/live-canary.mdnow use the repo-relative../../scripts/live-canary/ACCOUNTS.md.