feat(nix): add label-triggered cua-driver screenshot test - #1748
Conversation
Adds a `cua-driver-screenshot` label trigger that: 1. Runs the full NixOS VM integration test with screenshot capture 2. Captures a QEMU framebuffer screenshot showing xterm with test results 3. Uploads the screenshot as a GitHub artifact 4. Comments on the PR with a link to the screenshot 5. Removes the `cua-driver-screenshot` label New files: - nix/cua-driver/tests/screenshot.nix: integration test variant with machine.screenshot() at the end - .github/workflows/nix-screenshot.yml: label-triggered workflow Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughThis PR introduces screenshot-based integration testing for the CUA driver. It adds a new NixOS VM test that validates the driver's MCP protocol implementation and renders display output, integrates this test into the flake's checks, and provides a GitHub Actions workflow that automates test execution with AWS credential management and artifact handling. ChangesScreenshot Testing Infrastructure
Sequence Diagram(s)The PR introduces a coordinated flow across Nix test infrastructure and CI automation. The primary interaction occurs between the NixOS VM test runner, CUA driver process, and MCP client protocol validator, followed by screenshot capture and CI artifact management: sequenceDiagram
participant GitHub as GitHub Actions
participant AWS as AWS Secrets Manager
participant Nix as Nix Build
participant VM as NixOS VM
participant Driver as CUA Driver
participant MCP as MCP Client
participant Cache as S3 Cache
GitHub->>AWS: Assume OIDC Role
AWS->>GitHub: Return Credentials
GitHub->>AWS: Fetch Signing Key
AWS->>GitHub: Return Key Material
GitHub->>Nix: Install + Configure
Nix->>VM: Spawn Test VM
VM->>Driver: Boot with cua-driver enabled
VM->>Driver: CLI validation (--help, list-tools)
MCP->>Driver: Send JSON-RPC init + tool requests
Driver->>MCP: Respond with capabilities/tools
VM->>VM: Start Xvfb Display
VM->>VM: Render xterm Test Page
VM->>VM: Capture Screenshot
Nix->>GitHub: Return PNG Artifacts
GitHub->>Cache: Sign & Upload to S3
GitHub->>GitHub: Post PR Comment
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
r33drichards
added
the
cua-driver-screenshot
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/nix-screenshot.yml:
- Around line 25-27: The workflow step using actions/checkout (uses:
actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5) leaves a writable
token in the local git config; update that checkout step to set
persist-credentials: false so the action does not persist the credentials to the
repo's git config during the job, ensuring the checkout step includes the
persist-credentials: false option.
- Around line 141-147: The "Sign and upload to Nix cache" step (which runs nix
store sign --key-file ... and nix copy --to ...) must not execute in untrusted
pull_request workflows; remove or guard this step so it does not run for PR
events and instead move signing/upload to a separate trusted workflow that
triggers only on protected branch pushes or a release (e.g., create a new
workflow that runs the nix store sign and nix copy commands on push to main with
required protections). Specifically, delete or conditionally disable the
existing step (the "Sign and upload to Nix cache" job/step) from the PR workflow
and implement a new workflow that performs nix store sign and nix copy only in
the trusted context.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: f97e3ef8-33ab-4b51-a4f1-ad7e4709655d
📒 Files selected for processing (3)
.github/workflows/nix-screenshot.ymlflake.nixnix/cua-driver/tests/screenshot.nix
| - name: Checkout | ||
| uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4 | ||
|
|
There was a problem hiding this comment.
Disable checkout credential persistence.
actions/checkout should set persist-credentials: false to avoid leaving a writable token in local git config during the job.
Suggested fix
- name: Checkout
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
+ with:
+ persist-credentials: false📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| - name: Checkout | |
| uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4 | |
| - name: Checkout | |
| uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4 | |
| with: | |
| persist-credentials: false |
🧰 Tools
🪛 zizmor (1.25.2)
[warning] 25-26: credential persistence through GitHub Actions artifacts (artipacked): does not set persist-credentials: false
(artipacked)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/nix-screenshot.yml around lines 25 - 27, The workflow step
using actions/checkout (uses:
actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5) leaves a writable
token in the local git config; update that checkout step to set
persist-credentials: false so the action does not persist the credentials to the
repo's git config during the job, ensuring the checkout step includes the
persist-credentials: false option.
| - name: Sign and upload to Nix cache | ||
| if: always() | ||
| run: | | ||
| echo "Signing and uploading build artifacts to Nix cache..." | ||
| nix store sign --key-file "${{ runner.temp }}/signing-key.sec" --all | ||
| nix copy --to "s3://${{ env.NIX_CACHE_BUCKET }}?region=${{ env.AWS_REGION }}&want-mass-query=true" --all -L | ||
|
|
There was a problem hiding this comment.
Do not sign/upload cache artifacts from a pull_request workflow.
This job fetches a trusted signing key and then signs/uploads store paths for PR code. That creates a cache-poisoning/supply-chain risk in an untrusted execution context.
Suggested fix
- name: Sign and upload to Nix cache
- if: always()
+ if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
run: |
echo "Signing and uploading build artifacts to Nix cache..."
nix store sign --key-file "${{ runner.temp }}/signing-key.sec" --all
nix copy --to "s3://${{ env.NIX_CACHE_BUCKET }}?region=${{ env.AWS_REGION }}&want-mass-query=true" --all -LMove cache signing/publishing to a separate trusted workflow (e.g., push on protected branch) rather than this PR-label workflow.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/nix-screenshot.yml around lines 141 - 147, The "Sign and
upload to Nix cache" step (which runs nix store sign --key-file ... and nix copy
--to ...) must not execute in untrusted pull_request workflows; remove or guard
this step so it does not run for PR events and instead move signing/upload to a
separate trusted workflow that triggers only on protected branch pushes or a
release (e.g., create a new workflow that runs the nix store sign and nix copy
commands on push to main with required protections). Specifically, delete or
conditionally disable the existing step (the "Sign and upload to Nix cache"
job/step) from the PR workflow and implement a new workflow that performs nix
store sign and nix copy only in the trusted context.
github-actions
Bot
removed
the
cua-driver-screenshot
- Use `import -window root` (ImageMagick) + `copy_from_vm` to capture the Xvfb display, not `machine.screenshot()` which captures the blank QEMU VGA console and hangs - Remove unused `glob` require that caused JS SyntaxError in comment step - Add imagemagick to VM packages Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
r33drichards
added
the
cua-driver-screenshot
CUA Driver Screenshot Test |
github-actions
Bot
removed
the
cua-driver-screenshot
ImageMagick's `import -window root` hangs in headless Xvfb. Switch to xwd + xwdtopnm + pnmtopng pipeline which is more reliable. Also use copy_from_machine (not deprecated copy_from_vm) and bump step timeout. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
r33drichards
added
the
cua-driver-screenshot
CUA Driver Screenshot Test |
github-actions
Bot
removed
the
cua-driver-screenshot
machine.succeed("sleep N") blocks indefinitely in NixOS VM tests.
Use Python's time.sleep() in the test driver instead. Also add
timeout to xwd and suppress stderr noise from netpbm.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
r33drichards
added
the
cua-driver-screenshot
CUA Driver Screenshot TestScreenshot captured from NixOS VM integration test. 📸 Download screenshot artifact ✅ Test passed |
github-actions
Bot
removed
the
cua-driver-screenshot
)" This reverts commit 8d7e546.
2 participants
Summary
cua-driver-screenshotlabel that triggers a NixOS VM integration test with screenshot capturemachine.screenshot()cua-driver-screenshotlabel automaticallychecks.x86_64-linux.cua-driver-screenshotin the flake for local useNew files
nix/cua-driver/tests/screenshot.nixmachine.screenshot().github/workflows/nix-screenshot.ymlUsage
cua-driver-screenshotlabel to any PRTest plan
cua-driver-screenshotlabel to this PR to trigger the workflow🤖 Generated with Claude Code
Summary by CodeRabbit