Skip to content

Add support for AI agent environment variables to quiet test output#21135

Merged
Jarred-Sumner merged 9 commits intomainfrom
claude/add-claudecode-flag
Jul 18, 2025
Merged

Add support for AI agent environment variables to quiet test output#21135
Jarred-Sumner merged 9 commits intomainfrom
claude/add-claudecode-flag

Conversation

@robobun
Copy link
Copy Markdown
Collaborator

@robobun robobun commented Jul 17, 2025

Summary

This PR adds support for AI agent environment variables to enable quiet test output in Bun's test runner. When an AI agent is detected through environment variables like CLAUDECODE=1, REPL_ID=1, or IS_CODE_AGENT=1, the test runner reduces verbosity by only showing failures while still maintaining accurate test summary counts.

Changes

  • Uses existing Output.isAIAgent() function in src/output.zig which detects:
    • CLAUDECODE=1 - Claude Code
    • REPL_ID=1 - Replit
    • IS_CODE_AGENT=1 - Generic AI agent flag
    • (Future support for Gemini, Codex, and Cursor background agents)
  • Modified test output functions in src/cli/test_command.zig:
    • writeTestStatusLine() - Skip non-failure status indicators in quiet mode
    • printTestLine() - Skip non-failure test line output (with full JUnit support)
  • Added comprehensive test suite with snapshots in test/cli/test/claudecode-flag.test.ts

Behavior

When AI agent is detected:

  • Only test failures are displayed with their full output
  • Passing, skipped, and todo test indicators are hidden
  • Summary statistics are still shown at the end
  • JUnit reporter output remains unaffected
  • Error messages are more concise (e.g., "0 test files matching..." instead of friendly hints)

When no AI agent is detected:

  • All test output is shown normally (existing behavior)

Example Output

Normal mode (no AI agent):

test.js:
(pass) passing test [1.00ms]
(fail) failing test [2.00ms]
(skip) skipped test
(todo) todo test

 1 pass
 1 skip
 1 todo
 1 fail

Quiet mode (AI agent detected):

test.js:
(fail) failing test [2.00ms]

 1 pass
 1 skip
 1 todo
 1 fail

Test plan

  • All existing tests pass
  • New comprehensive test suite with snapshots
  • Comparison tests between AI agent and normal modes
  • Debug build compilation successful
  • JUnit reporter functionality preserved
  • Summary statistics accuracy maintained

This feature is useful for AI coding assistants where reduced output verbosity improves readability and context efficiency while still providing essential failure information.

🤖 Generated with Claude Code

This adds support for the CLAUDECODE environment variable to enable quiet test output,
which reduces verbosity by only showing failures while still maintaining accurate
test summary counts.

When CLAUDECODE=1 is set:
- Only test failures are displayed with their full output
- Passing, skipped, and todo test indicators are hidden
- Summary statistics are still shown at the end
- JUnit reporter output remains unaffected

When CLAUDECODE=0 or unset:
- All test output is shown normally (existing behavior)

This feature is useful for automated testing environments where reduced output
verbosity improves readability while still providing essential failure information.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
@robobun
Copy link
Copy Markdown
Collaborator Author

robobun commented Jul 17, 2025

Updated 8:31 PM PT - Jul 17th, 2025

@Jarred-Sumner, your commit 25d749a is building: #20874

@RiskyMH
Copy link
Copy Markdown
Collaborator

RiskyMH commented Jul 17, 2025

this probably should also force disable ansi (if it somehow gets enabled, as its a waste of its tokens)

Comment thread src/cli/test_command.zig Outdated
Comment on lines 652 to 656
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it should just always run this code here then? not too good to duplicate the code

- Add filename suppression logic that only shows file paths when there's a failure
- Remove duplicate JUnit reporting code for cleaner implementation
- Track current file info in CommandLineReporter for deferred filename printing
- Add printFilenameIfNeeded function to print filename only when needed
- Maintain full JUnit reporting functionality in quiet mode

This makes the quiet mode even more focused - showing only essential failure
information while completely hiding successful test output.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
@robobun robobun force-pushed the claude/add-claudecode-flag branch from 5c480c2 to 45af2b5 Compare July 17, 2025 15:42
@mizulu
Copy link
Copy Markdown
Contributor

mizulu commented Jul 17, 2025

  1. This feature seem vaguely familiar
    https://github.com/oven-sh/bun/compare/claude/implement-agent-flag

there they use --agent
which I think is slightly better, as it is more generic(for the day you switch to another agent)

  1. some test runners implements flags to control that even more generically
    which is what we should have for bun test runner

Jest : https://jestjs.io/docs/cli#--onlyfailures

--onlyFailures

Vitest

--silent   // to remove all outputs 
--silent passed-only  //  let errors passthrough 

--no-color
https://vitest.dev/guide/cli.html#color

I think we need those explicit flags so we have more control future wise
as this feature might not be just agent specific, and can be useful
for users as well, especially the --onlyFailures

after we have proper options for the test runner,
then feel free to add some AGENT env with some convenience defaults

@190n

This comment was marked as resolved.

@Jarred-Sumner Jarred-Sumner changed the title Add CLAUDECODE=1 environment variable to quiet test output Add support for AI agent environment variables to quiet test output Jul 18, 2025
@Jarred-Sumner Jarred-Sumner merged commit 851fa7d into main Jul 18, 2025
52 of 57 checks passed
@Jarred-Sumner Jarred-Sumner deleted the claude/add-claudecode-flag branch July 18, 2025 04:20
robobun pushed a commit to robobun/bun that referenced this pull request Jul 18, 2025
Documents the AI agent integration feature that enables quieter test output
for AI coding assistants. This feature was added in PR oven-sh#21135 and supports
environment variables like CLAUDECODE=1, REPL_ID=1, and IS_CODE_AGENT=1
to minimize test output verbosity while preserving failure information.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
@robobun robobun added the claude label Aug 14, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants