feat(ci): add E2E smoke test workflows for Claude and Codex#1255
feat(ci): add E2E smoke test workflows for Claude and Codex#1255
Conversation
Adds real workflow execution to CI, verifying the full engine works end-to-end with both providers. Organized into 4 tiers: deterministic (0 API calls), Claude, Codex, and mixed-provider tests. New workflows: - e2e-deterministic: bash, script (bun/uv), conditions, trigger rules - e2e-skills-mcp: skills injection, MCP server, effort, systemPrompt - Enhanced existing e2e-claude-smoke, e2e-codex-smoke, e2e-mixed-providers - Fixed e2e-all-nodes (was broken due to script node syntax) Supporting files: - e2e-echo-command.md (test command file) - echo-args.py (Python script for uv runtime test) - e2e-test-skill/SKILL.md (minimal skill for injection test) - e2e-filesystem.json (MCP config for filesystem server test) GitHub Actions: .github/workflows/e2e-smoke.yml - Runs on push to main/dev only (no PR trigger to avoid API cost abuse) - Uses haiku (Claude) and gpt-5.1-codex-mini (Codex) for cost efficiency Closes #1254 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds end-to-end smoke test CI: a new GitHub Actions workflow, multiple Archon workflow files (new/updated), a command doc, a Python echo script, and removal of one legacy Archon workflow. Changes
Sequence Diagram(s)sequenceDiagram
participant GH as GitHub Actions
participant Runner as CI Runner
participant ArchonCLI as Archon CLI
participant Workflow as Archon Workflow DAG
participant Script as Local Script (bun/python)
participant Claude as Claude Provider
participant Codex as Codex Provider
GH->>Runner: trigger e2e-* job
Runner->>ArchonCLI: invoke workflow (e.g., e2e-deterministic / e2e-claude / e2e-codex / e2e-mixed)
ArchonCLI->>Workflow: load DAG and start nodes
Workflow->>Script: execute script nodes (bun, python) and return outputs
Workflow->>Claude: run Claude nodes (model: haiku)
Workflow->>Codex: run Codex nodes (model: gpt-5.1-codex-mini)
Claude-->>Workflow: node outputs
Codex-->>Workflow: node outputs
Script-->>Workflow: deterministic outputs
Workflow-->>ArchonCLI: aggregate/verify results
ArchonCLI-->>Runner: exit status and logs
Runner-->>GH: job result
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Poem
🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (2 inconclusive)
✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (5)
.archon/scripts/echo-args.py (1)
1-7: LGTM! Clean and fit for purpose.The script correctly implements a simple test fixture: it reads a command-line argument (with a sensible default), serializes it to JSON with a UTC timestamp, and prints the result. The logic is straightforward and the code is syntactically correct.
Optional: Consider adding a shebang for portability
While not required if invoked via
uv runorpython, adding a shebang would make the script directly executable and improve portability:+#!/usr/bin/env python3 """Simple script node test — echoes input as JSON (uv/Python runtime).""" import json🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In @.archon/scripts/echo-args.py around lines 1 - 7, Add a POSIX shebang that invokes the system Python 3 via env as the very first line of echo-args.py so the script can be executed directly, and update the file mode to be executable (e.g., set the executable bit); keep the existing logic in the module (reading sys.argv[1], defaulting to "no-input", and printing the JSON with datetime.now(timezone.utc).isoformat()) unchanged..github/workflows/e2e-smoke.yml (3)
59-64: Claude job runs 3 workflow files sequentially.The
e2e-claudejob runse2e-claude-smoke,e2e-all-nodes, ande2e-skills-mcpsequentially. If one fails, subsequent tests won't run. Consider whether you wantcontinue-on-error: trueon individual steps or if failing fast is preferred.Also applies to: 66-71, 73-78
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In @.github/workflows/e2e-smoke.yml around lines 59 - 64, The Claude job currently runs three sequential steps ("Run Claude smoke test", the step that runs `e2e-all-nodes`, and the step that runs `e2e-skills-mcp`) and will stop on the first failure; update the workflow to make the desired behavior explicit: either add continue-on-error: true to each individual step (`name: Run Claude smoke test`, the `e2e-all-nodes` step, and the `e2e-skills-mcp` step) so later tests run even if one fails, or keep failing-fast by leaving them as-is; modify the relevant steps in the e2e-claude job to include continue-on-error where you want non-fatal failures.
51-54: Pin Claude CLI version in install script for reproducibility and stability.The remote install script may change or be updated to install different versions over time. Specify an explicit version or cache the CLI installation to ensure consistent behavior across CI runs.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In @.github/workflows/e2e-smoke.yml around lines 51 - 54, Update the "Install Claude Code CLI" workflow step to pin the CLI to a specific, immutable release and/or use workflow caching instead of calling the remote install.sh unpinned; for example invoke the installer with an explicit version argument (or download a specific release artifact) so the curl | bash line is replaced with a deterministic install of a named version, and keep the existing echo to GITHUB_PATH; this ensures reproducible installs across CI runs.
97-98: Pin Codex CLI to a specific version for reproducible CI builds.Using
npm install -g@openai/codex`` without a version specifier causes non-deterministic CI behavior when the package is updated. This conflicts with the pinned versions used elsewhere in the workflow (Bun 1.3.11, Node 22, etc.).🔧 Proposed fix
- name: Install Codex CLI - run: npm install -g `@openai/codex` + run: npm install -g `@openai/codex`@0.1.2505141022Replace
0.1.2505141022with the desired version. Check available versions withnpm view@openai/codexversions.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In @.github/workflows/e2e-smoke.yml around lines 97 - 98, The Install Codex CLI step uses a floating dependency which makes CI non-deterministic; update the run command in the "Install Codex CLI" step to pin the package to a specific version (e.g., change npm install -g `@openai/codex` to npm install -g `@openai/codex`@0.1.2505141022 or your chosen semver), and verify the chosen version exists with npm view `@openai/codex` versions before committing..archon/test-fixtures/mcp/e2e-filesystem.json (1)
1-6: Consider the unpredictability of/tmpcontents in CI.The MCP server targets
/tmp, which may contain unpredictable or empty contents in CI environments. While this configuration is valid, the correspondingmcp-testnode (ine2e-skills-mcp.yaml) asks to "list the contents of /tmp" without specifying expected results, which could lead to non-deterministic test behavior.For a smoke test, verifying the MCP server loads and responds without errors may be sufficient. However, consider whether a more controlled test directory (e.g., creating a temp subdirectory with known files) would improve test reliability.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In @.archon/test-fixtures/mcp/e2e-filesystem.json around lines 1 - 6, The MCP fixture uses an uncontrolled /tmp directory via the "filesystem" object (command: "npx", args: [..., "/tmp"]) which makes CI tests non-deterministic; change the args target to a dedicated test subdirectory (e.g., "/tmp/mcp-test-fixture" or a dynamically created temp dir) and update the test setup so that the mcp-test node in e2e-skills-mcp.yaml populates that directory with known files before the test runs (or validate only server readiness instead of listing contents) so tests are reliable.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In @.archon/workflows/e2e-codex-smoke.yaml:
- Line 6: Replace the deprecated model token 'gpt-5.1-codex-mini' with
'gpt-5.3-codex' in the workflow configuration so the pipeline uses the supported
GPT-5.3-Codex model; update the value for the key that currently reads "model:
gpt-5.1-codex-mini" to "model: gpt-5.3-codex" ensuring no other fields are
changed.
In @.archon/workflows/e2e-skills-mcp.yaml:
- Around line 41-52: The workflow places context-shared-setup in the same
parallel layer as mcp-test, effort-test, and system-prompt-test (all depending
only on skill-test), so its session is not preserved for context-shared-verify;
change the dependency graph so the two context nodes run sequentially: either
give context-shared-setup a unique dependency not shared with those parallel
nodes (so context-shared-verify can depend on it), or chain context-shared-setup
-> context-shared-verify and ensure neither depends directly on the parallel
nodes (adjust the depends_on entries for context-shared-setup and
context-shared-verify and remove conflicting shared dependencies with
skill-test, mcp-test, effort-test, and system-prompt-test).
In @.github/workflows/e2e-smoke.yml:
- Around line 3-5: Update the GitHub Actions trigger in the workflow's on: push:
branches list to remove the 'feat/e2e-smoke-tests' entry so pushes only on
'main' and 'dev' run the e2e-smoke workflow; edit the branches array that
currently contains [main, dev, feat/e2e-smoke-tests] and delete the
'feat/e2e-smoke-tests' item, keeping the rest unchanged.
---
Nitpick comments:
In @.archon/scripts/echo-args.py:
- Around line 1-7: Add a POSIX shebang that invokes the system Python 3 via env
as the very first line of echo-args.py so the script can be executed directly,
and update the file mode to be executable (e.g., set the executable bit); keep
the existing logic in the module (reading sys.argv[1], defaulting to "no-input",
and printing the JSON with datetime.now(timezone.utc).isoformat()) unchanged.
In @.archon/test-fixtures/mcp/e2e-filesystem.json:
- Around line 1-6: The MCP fixture uses an uncontrolled /tmp directory via the
"filesystem" object (command: "npx", args: [..., "/tmp"]) which makes CI tests
non-deterministic; change the args target to a dedicated test subdirectory
(e.g., "/tmp/mcp-test-fixture" or a dynamically created temp dir) and update the
test setup so that the mcp-test node in e2e-skills-mcp.yaml populates that
directory with known files before the test runs (or validate only server
readiness instead of listing contents) so tests are reliable.
In @.github/workflows/e2e-smoke.yml:
- Around line 59-64: The Claude job currently runs three sequential steps ("Run
Claude smoke test", the step that runs `e2e-all-nodes`, and the step that runs
`e2e-skills-mcp`) and will stop on the first failure; update the workflow to
make the desired behavior explicit: either add continue-on-error: true to each
individual step (`name: Run Claude smoke test`, the `e2e-all-nodes` step, and
the `e2e-skills-mcp` step) so later tests run even if one fails, or keep
failing-fast by leaving them as-is; modify the relevant steps in the e2e-claude
job to include continue-on-error where you want non-fatal failures.
- Around line 51-54: Update the "Install Claude Code CLI" workflow step to pin
the CLI to a specific, immutable release and/or use workflow caching instead of
calling the remote install.sh unpinned; for example invoke the installer with an
explicit version argument (or download a specific release artifact) so the curl
| bash line is replaced with a deterministic install of a named version, and
keep the existing echo to GITHUB_PATH; this ensures reproducible installs across
CI runs.
- Around line 97-98: The Install Codex CLI step uses a floating dependency which
makes CI non-deterministic; update the run command in the "Install Codex CLI"
step to pin the package to a specific version (e.g., change npm install -g
`@openai/codex` to npm install -g `@openai/codex`@0.1.2505141022 or your chosen
semver), and verify the chosen version exists with npm view `@openai/codex`
versions before committing.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 35a914ed-b6f9-464f-838a-62244d02b982
📒 Files selected for processing (11)
.archon/commands/e2e-echo-command.md.archon/scripts/echo-args.py.archon/test-fixtures/mcp/e2e-filesystem.json.archon/workflows/e2e-all-nodes.yaml.archon/workflows/e2e-claude-smoke.yaml.archon/workflows/e2e-codex-smoke.yaml.archon/workflows/e2e-deterministic.yaml.archon/workflows/e2e-mixed-providers.yaml.archon/workflows/e2e-skills-mcp.yaml.claude/skills/e2e-test-skill/SKILL.md.github/workflows/e2e-smoke.yml
| name: e2e-codex-smoke | ||
| description: "E2E smoke test for Codex provider. Runs a simple prompt + structured output node." | ||
| provider: codex | ||
| model: gpt-5.1-codex-mini |
There was a problem hiding this comment.
🧩 Analysis chain
🌐 Web query:
OpenAI Codex gpt-5.1-codex-mini model availability
💡 Result:
The gpt-5.1-codex-mini model from OpenAI, a smaller version of gpt-5.1-codex optimized for coding tasks, was released around November 2025 and made available via the OpenAI API with pricing at $0.25/1M input tokens and $2.00/1M output tokens. It supports text and image input, text output, 400k context window, and features like function calling and structured outputs. It was also integrated into GitHub Copilot in public preview in November 2025. However, as of April 1, 2026, it has been deprecated in GitHub Copilot (recommended replacement: GPT-5.3-Codex). Current OpenAI API model lists (as of April 2026) do not prominently feature it among active coding models (e.g., GPT-5-Codex, GPT-5.3-Codex), suggesting it may be legacy or deprecated, though dedicated model pages still exist. It is not listed as retired in official deprecation notices, unlike some other models. Availability may depend on the platform (API vs. Copilot), with potential continued access via API for existing users but not recommended for new use.
Citations:
- 1: https://developers.openai.com/api/docs/models/gpt-5.1-codex-mini
- 2: https://platform.openai.com/docs/models/gpt-5.1-codex-mini
- 3: https://developers.openai.com/api/docs/models/gpt-5.1-codex
- 4: https://developers.openai.com/api/docs/models/all
- 5: https://github.blog/changelog/2025-11-13-openais-gpt-5-1-gpt-5-1-codex-and-gpt-5-1-codex-mini-are-now-in-public-preview-for-github-copilot/
- 6: https://developers.openai.com/api/docs/models/all/
- 7: https://github.blog/changelog/2026-04-03-gpt-5-1-codex-gpt-5-1-codex-max-and-gpt-5-1-codex-mini-deprecated
Replace deprecated model gpt-5.1-codex-mini with gpt-5.3-codex.
The gpt-5.1-codex-mini model was deprecated on April 3, 2026. Use GPT-5.3-Codex instead.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In @.archon/workflows/e2e-codex-smoke.yaml at line 6, Replace the deprecated
model token 'gpt-5.1-codex-mini' with 'gpt-5.3-codex' in the workflow
configuration so the pipeline uses the supported GPT-5.3-Codex model; update the
value for the key that currently reads "model: gpt-5.1-codex-mini" to "model:
gpt-5.3-codex" ensuring no other fields are changed.
- Rename echo-args.py → echo-py.py to avoid duplicate script name conflict with echo-args.js (script discovery uses base name, not extension) - Add CODEX_API_KEY env var to codex and mixed CI jobs (Codex CLI requires this, not OPENAI_API_KEY, for headless auth) - Sequentialize all Claude AI nodes via depends_on chains to prevent concurrent CLI subprocess idle timeouts in CI - Increase idle_timeout from 60s to 120s on all AI nodes for CI headroom - Override MCP test node to model: sonnet (Haiku doesn't support MCP tool search) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Actionable comments posted: 3
♻️ Duplicate comments (1)
.github/workflows/e2e-smoke.yml (1)
5-5:⚠️ Potential issue | 🟡 MinorRemove the temporary feature-branch trigger before merge.
Line 5 still includes
feat/e2e-smoke-tests, so pushes to that branch will keep running paid E2E jobs.🔧 Proposed fix
on: push: - branches: [main, dev, feat/e2e-smoke-tests] + branches: [main, dev]🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In @.github/workflows/e2e-smoke.yml at line 5, Remove the temporary branch trigger string "feat/e2e-smoke-tests" from the branches array in the workflow branches definition so pushes to that feature branch no longer trigger the E2E job; locate the branches: [main, dev, feat/e2e-smoke-tests] entry and update it to only include the permanent branches (e.g., main and dev).
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In @.github/workflows/e2e-smoke.yml:
- Line 106: The workflow is wiring CODEX_API_KEY to secrets.OPENAI_API_KEY,
coupling two independent credentials; update the environment variable
assignments where CODEX_API_KEY is set (currently using secrets.OPENAI_API_KEY)
to use a dedicated secret (e.g. secrets.CODEX_API_KEY) instead, and ensure the
repository/CI secrets contain CODEX_API_KEY; apply this same replacement for
every occurrence of CODEX_API_KEY = ${{ secrets.OPENAI_API_KEY }} in the
workflow.
- Line 98: The workflow currently installs `@openai/codex` unpinned via the run
steps that contain "run: npm install -g `@openai/codex`"; update both occurrences
(the step at the shown run line and the other similar install step) to pin the
package to version 0.121.0 by changing the command to install
`@openai/codex`@0.121.0 so CI runs are reproducible.
- Line 62: The env entry CLAUDE_BIN_PATH currently uses a literal tilde
(~/.local/bin/claude) which is not expanded by Node.js; update the workflow so
CLAUDE_BIN_PATH is exported in the shell (run:) with $HOME expansion instead of
in the env: block — replace occurrences where CLAUDE_BIN_PATH is set in env (the
steps that invoke Claude at symbols CLAUDE_BIN_PATH in the e2e-smoke workflow)
and move/export them inside the run: command (e.g., export
CLAUDE_BIN_PATH="$HOME/.local/bin/claude") so the shell expands the path before
Node.js reads the variable.
---
Duplicate comments:
In @.github/workflows/e2e-smoke.yml:
- Line 5: Remove the temporary branch trigger string "feat/e2e-smoke-tests" from
the branches array in the workflow branches definition so pushes to that feature
branch no longer trigger the E2E job; locate the branches: [main, dev,
feat/e2e-smoke-tests] entry and update it to only include the permanent branches
(e.g., main and dev).
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: cffeafe8-3799-4fee-a070-9ff6486131fe
📒 Files selected for processing (6)
.archon/scripts/echo-py.py.archon/workflows/e2e-all-nodes.yaml.archon/workflows/e2e-claude-smoke.yaml.archon/workflows/e2e-deterministic.yaml.archon/workflows/e2e-skills-mcp.yaml.github/workflows/e2e-smoke.yml
✅ Files skipped from review due to trivial changes (1)
- .archon/scripts/echo-py.py
🚧 Files skipped from review as they are similar to previous changes (4)
- .archon/workflows/e2e-claude-smoke.yaml
- .archon/workflows/e2e-skills-mcp.yaml
- .archon/workflows/e2e-deterministic.yaml
- .archon/workflows/e2e-all-nodes.yaml
| - name: Run Claude smoke test | ||
| env: | ||
| ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} | ||
| CLAUDE_BIN_PATH: ~/.local/bin/claude |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Verify literal tilde usage in workflow env blocks
rg -n 'CLAUDE_BIN_PATH:\s*~/.local/bin/claude' .github/workflows/e2e-smoke.ymlRepository: coleam00/Archon
Length of output: 259
🏁 Script executed:
# Check the workflow file to see how CLAUDE_BIN_PATH is actually used
cat -n .github/workflows/e2e-smoke.yml | sed -n '55,85p'Repository: coleam00/Archon
Length of output: 1291
🏁 Script executed:
# Check lines around line 144 as well
cat -n .github/workflows/e2e-smoke.yml | sed -n '135,155p'Repository: coleam00/Archon
Length of output: 591
🏁 Script executed:
# Search for how CLAUDE_BIN_PATH is referenced in the codebase
rg -n 'CLAUDE_BIN_PATH' --type-listRepository: coleam00/Archon
Length of output: 448
🏁 Script executed:
# Look for any tilde expansion handling in the CLI or workflow code
rg -n '\$HOME|\$\{HOME\}|expanduser|expandtilde' packages/ --type ts --type js --type bashRepository: coleam00/Archon
Length of output: 87
🏁 Script executed:
# Search for references to CLAUDE_BIN_PATH in the codebase
rg -n 'CLAUDE_BIN_PATH' packages/ --type ts --type jsRepository: coleam00/Archon
Length of output: 3798
🏁 Script executed:
# Look for tilde/home expansion in TypeScript/JavaScript code
rg -n '\$HOME|expanduser|expandtilde|~' packages/core/src --type ts -B 2 -A 2Repository: coleam00/Archon
Length of output: 10100
🏁 Script executed:
# Check how the CLI processes environment variables for paths
rg -n 'process\.env\[|process\.env\.' packages/core/src --type ts -B 2 -A 2 | head -100Repository: coleam00/Archon
Length of output: 7816
🏁 Script executed:
# Search for any environment variable path handling
find packages -name '*.ts' -type f -exec grep -l 'env' {} \; | head -10Repository: coleam00/Archon
Length of output: 4769
Fix literal tilde in CLAUDE_BIN_PATH by expanding it in the shell before Node.js reads the variable.
GitHub Actions' env: block passes ~/.local/bin/claude as a literal string to the Node.js process, which does not automatically expand tilde. The resolver then fails file existence checks because ~ is not a valid path. Export the variable with $HOME expansion in the run: block instead.
🔧 Proposed fix (apply to all Claude-running steps)
- name: Run Claude smoke test
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
- CLAUDE_BIN_PATH: ~/.local/bin/claude
run: |
+ export CLAUDE_BIN_PATH="$HOME/.local/bin/claude"
bun run cli workflow run e2e-claude-smoke --no-worktree "smoke test"Applies to lines: 62–63, 69–70, 76–77, 144–145
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In @.github/workflows/e2e-smoke.yml at line 62, The env entry CLAUDE_BIN_PATH
currently uses a literal tilde (~/.local/bin/claude) which is not expanded by
Node.js; update the workflow so CLAUDE_BIN_PATH is exported in the shell (run:)
with $HOME expansion instead of in the env: block — replace occurrences where
CLAUDE_BIN_PATH is set in env (the steps that invoke Claude at symbols
CLAUDE_BIN_PATH in the e2e-smoke workflow) and move/export them inside the run:
command (e.g., export CLAUDE_BIN_PATH="$HOME/.local/bin/claude") so the shell
expands the path before Node.js reads the variable.
| node-version: 22 | ||
|
|
||
| - name: Install Codex CLI | ||
| run: npm install -g @openai/codex |
There was a problem hiding this comment.
🧩 Analysis chain
🌐 Web query:
What is the latest stable npm version of @openai/codex suitable for CI pinning, and are there any known breaking changes in recent releases?
💡 Result:
The latest stable npm version of @openai/codex is 0.121.0, released on 2026-04-15. It is suitable for CI pinning as it is the most recent non-pre-release version recommended in official OpenAI changelogs with install commands like npm install -g @openai/codex@0.121.0. Recent releases (0.119.0, 0.120.0, 0.121.0) primarily include new features (e.g., marketplace support, sandbox fixes), chores (e.g., supply-chain hardening, Rust optimizations), and bug fixes. No explicit breaking changes are mentioned in the release notes or changelogs; changes appear additive or non-breaking. One community report notes a review changes issue in a "latest update," but this is unconfirmed and not in official notes. For CI, pin to ^0.121.0 or exactly 0.121.0 and monitor GitHub releases for updates.
Citations:
- 1: https://www.npmjs.com/package/@openai/codex
- 2: https://github.com/openai/codex/releases
- 3: https://www.npmjs.com/package/@openai/codex?activeTab=versions
- 4: https://developers.openai.com/codex/changelog
- 5: https://help.openai.com/en/articles/11428266-codex-changelog
🏁 Script executed:
cat -n .github/workflows/e2e-smoke.yml | grep -A 2 -B 2 "npm install -g `@openai/codex`"Repository: coleam00/Archon
Length of output: 353
🏁 Script executed:
cat -n .github/workflows/e2e-smoke.yml | grep "npm install"Repository: coleam00/Archon
Length of output: 156
Pin Codex CLI version for reproducible CI runs.
Lines 98 and 134 install @openai/codex unpinned, which can introduce nondeterministic failures across runs. Pin to version 0.121.0 (latest stable) in both locations.
🔧 Proposed fix
- - name: Install Codex CLI
- run: npm install -g `@openai/codex`
+ - name: Install Codex CLI
+ run: npm install -g `@openai/codex`@0.121.0📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| run: npm install -g @openai/codex | |
| run: npm install -g `@openai/codex`@0.121.0 |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In @.github/workflows/e2e-smoke.yml at line 98, The workflow currently installs
`@openai/codex` unpinned via the run steps that contain "run: npm install -g
`@openai/codex`"; update both occurrences (the step at the shown run line and the
other similar install step) to pin the package to version 0.121.0 by changing
the command to install `@openai/codex`@0.121.0 so CI runs are reproducible.
| - name: Run Codex smoke test | ||
| env: | ||
| OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} | ||
| CODEX_API_KEY: ${{ secrets.OPENAI_API_KEY }} |
There was a problem hiding this comment.
Wire CODEX_API_KEY to its dedicated secret.
Lines 106 and 143 set CODEX_API_KEY from secrets.OPENAI_API_KEY; this couples two credentials and can break if Codex auth is rotated independently.
🔧 Proposed fix
- name: Run Codex smoke test
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
- CODEX_API_KEY: ${{ secrets.OPENAI_API_KEY }}
+ CODEX_API_KEY: ${{ secrets.CODEX_API_KEY }}
...
- name: Run mixed providers test
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
- CODEX_API_KEY: ${{ secrets.OPENAI_API_KEY }}
+ CODEX_API_KEY: ${{ secrets.CODEX_API_KEY }}
CLAUDE_BIN_PATH: ~/.local/bin/claudeAlso applies to: 143-143
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In @.github/workflows/e2e-smoke.yml at line 106, The workflow is wiring
CODEX_API_KEY to secrets.OPENAI_API_KEY, coupling two independent credentials;
update the environment variable assignments where CODEX_API_KEY is set
(currently using secrets.OPENAI_API_KEY) to use a dedicated secret (e.g.
secrets.CODEX_API_KEY) instead, and ensure the repository/CI secrets contain
CODEX_API_KEY; apply this same replacement for every occurrence of CODEX_API_KEY
= ${{ secrets.OPENAI_API_KEY }} in the workflow.
Claude CLI is slow with structured output and tool use in CI (~4 min for structured output, ~2 min for tool use). With 3 sequential workflow runs (claude-smoke, all-nodes, skills-mcp), 10 minutes is insufficient. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Claude CLI is extremely slow with structured output (~4 min) and tool use (~2 min) in CI, making the previous multi-workflow approach take 10+ min. Radical simplification: - Remove e2e-all-nodes (redundant with deterministic + claude-smoke) - Remove e2e-skills-mcp (advanced features too slow for per-commit smoke) - Remove structured output and tool use from Claude smoke test (too slow) - Strip Claude smoke to: 1 prompt + 1 command + 1 bash verify node - Keep mixed providers (simplified: 1 Claude + 1 Codex + bash verify) - All timeouts reduced to 30s, all job timeouts to 5 min - Remove MCP test fixtures and e2e-test-skill (no longer needed) Expected: Claude job ~15s of AI time, Codex ~5s, mixed ~10s Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In @.archon/workflows/e2e-mixed-providers.yaml:
- Around line 26-27: The verify step currently only echoes values and never
fails; update the "verify" step in .archon/workflows/e2e-mixed-providers.yaml to
actually compare the outputs (claude-node.output and codex-node.output) and exit
non‑zero on mismatch so the CI fails on regression; replace the simple echo with
a small bash assertion that reads the two values into variables, compares them
(e.g., string equality), prints a helpful error message including both
claude-node.output and codex-node.output when they differ, and exits 1 to
indicate failure.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: f3dbddb7-a12e-4778-8307-804128291a3e
📒 Files selected for processing (5)
.archon/workflows/e2e-all-nodes.yaml.archon/workflows/e2e-claude-smoke.yaml.archon/workflows/e2e-codex-smoke.yaml.archon/workflows/e2e-mixed-providers.yaml.github/workflows/e2e-smoke.yml
💤 Files with no reviewable changes (1)
- .archon/workflows/e2e-all-nodes.yaml
🚧 Files skipped from review as they are similar to previous changes (3)
- .archon/workflows/e2e-codex-smoke.yaml
- .github/workflows/e2e-smoke.yml
- .archon/workflows/e2e-claude-smoke.yaml
| bash: "echo 'claude=$claude-node.output codex=$codex-node.output'" | ||
| depends_on: [claude-node, codex-node] |
There was a problem hiding this comment.
verify currently doesn’t verify; it only logs.
This step always succeeds even if interpolation/output content regresses, which can produce false-green CI for the smoke test.
Suggested change
- bash: "echo 'claude=$claude-node.output codex=$codex-node.output'"
+ bash: |
+ set -euo pipefail
+ claude_out="$claude-node.output"
+ codex_out="$codex-node.output"
+ [[ "$claude_out" == "claude-ok" ]] || { echo "Unexpected claude output: $claude_out"; exit 1; }
+ [[ "$codex_out" == "codex-ok" ]] || { echo "Unexpected codex output: $codex_out"; exit 1; }
+ echo "claude=$claude_out codex=$codex_out"🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In @.archon/workflows/e2e-mixed-providers.yaml around lines 26 - 27, The verify
step currently only echoes values and never fails; update the "verify" step in
.archon/workflows/e2e-mixed-providers.yaml to actually compare the outputs
(claude-node.output and codex-node.output) and exit non‑zero on mismatch so the
CI fails on regression; replace the simple echo with a small bash assertion that
reads the two values into variables, compares them (e.g., string equality),
prints a helpful error message including both claude-node.output and
codex-node.output when they differ, and exits 1 to indicate failure.
The command-test node was missing allowed_tools: [], causing the Claude CLI to load full tool access. Without tools restricted, the subprocess hangs after responding. The simple prompt node with allowed_tools: [] completes in 4s — this should match. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Command nodes consistently produce zero output and hit the 30s idle timeout in CI, even with allowed_tools: []. This appears to be a bug in how command: nodes interact with the Claude CLI subprocess — the process never emits output. This adds 30s of wasted time to every run. The simple prompt node already verifies Claude connectivity. Command file discovery/loading is a deterministic operation that doesn't need an AI call to validate in a smoke test. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In @.archon/workflows/e2e-claude-smoke.yaml:
- Around line 2-4: The verify-output step currently just echoes $nodeId.output
and always exits zero, so it cannot detect interpolation failures; update the
verify-output step (the job/step named "verify-output" referencing
$nodeId.output and the nodeId output variable) to explicitly check that
$nodeId.output is present/non-empty and exit non‑zero when it is missing or
empty (e.g., test for emptiness and fail with a non‑zero exit) so the workflow
fails on substitution regressions instead of producing a false green.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: eeb3ac75-dc0c-494f-ac60-9faa1fe29f89
📒 Files selected for processing (1)
.archon/workflows/e2e-claude-smoke.yaml
Injects exit 1 into e2e-deterministic bash-echo node to prove the engine fix (failWorkflowRun on anyFailed) propagates to a non-zero CLI exit code and a red X in GitHub Actions. Will be reverted in the next commit. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Reverts the injected exit 1 in bash-echo (CI red X confirmed in run 24522356737). Removes feat/e2e-smoke-tests from branch triggers — ready to merge to dev. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Will remove feat/e2e-smoke-tests trigger in the final cleanup commit before merging to dev. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Removes feat/e2e-smoke-tests from E2E workflow triggers. CI failure detection verified: red X on run 24522356737 (deliberate bash exit 1), green on run 24522484762 (reverted), and credit-exhaustion failure also correctly produced exit 1. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
feat(ci): add E2E smoke test workflows for Claude and Codex
1 participant
Summary
Adds lightweight E2E smoke tests that run on every push to
main/dev, verifying the workflow engine works end-to-end with real AI providers.4 CI jobs (all parallel except mixed):
$nodeId.outputsubstitution,when:conditions,trigger_rulejoin semantics.$nodeId.outputrefs.Design decisions:
allowed_tools: []— without this, the Claude CLI subprocess hangs after respondinghaikufor Claude,gpt-5.1-codex-minifor CodexTotal AI time per commit: ~15 seconds (4s Claude + 4s Codex + 7s mixed)
Test plan
Closes #1254
🤖 Generated with Claude Code
Summary by CodeRabbit
Tests
Documentation
Chores
Tools