Add self-test recipe for goose validation

[tlongwell-block]

This PR introduces goose-self-test.yaml, a meta-testing recipe that enables goose to validate its own capabilities through first-person integration testing.

What is First-Person Integration Testing?

Traditional testing approaches rely on external test harnesses, unit tests, or integration suites that examine a system from the outside. This recipe takes a different approach: it has a running goose instance test itself using its own tools and capabilities.

This is meta-testing - the system under test is also the tester, examining its own behavior from within an active session. For an AI agent like goose, this approach offers unique insights into behavioral consistency and tool reliability that external testing cannot provide.

Primary Use Case: Goose Testing Goose

The most powerful application of this recipe is when goose itself is developing new goose features. A goose instance working on the codebase can:

1. Make changes to the goose source code
2. Build a new goose binary with cargo build --release
3. Shell out to run the self-test recipe using the newly built binary:

   ./target/release/goose run --recipe goose-self-test.yaml

4. Analyze the results to verify the new feature works correctly

This creates a recursive development loop where goose can autonomously develop, test, and validate improvements to itself. The goose doing the development can examine test outputs, debug failures, and iterate on fixes - all while using the self-test recipe to validate each iteration.

How It Works

The self-test recipe guides goose through a structured validation process:

1. Environment Setup - Creates a workspace and establishes testing infrastructure
2. Tool Validation - Tests file operations, shell commands, and code analysis
3. Extension Testing - Validates dynamic extension management
4. Subagent Orchestration - Tests recursive self-creation and parallel execution
5. Advanced Scenarios - Explores error boundaries and security controls
6. Report Generation - Produces comprehensive documentation of results

The recipe uses goose's own capabilities to create test scenarios, execute them, and validate the outcomes. Each test phase builds on the previous, creating a comprehensive assessment of functionality.

Design Principles

What Can Be Tested

From within a running session, goose can test:

• Tool execution and error handling
• File and shell operations
• Code analysis accuracy
• Extension management
• Subagent creation and coordination
• Observable behaviors and consistency

What Cannot Be Tested

Certain aspects require external observation:

• Provider switching mid-session
• Session persistence across restarts
• Internal token counting mechanisms
• Network transport layers
• Security boundaries from outside perspective

The recipe focuses on what's testable from within, providing meaningful validation of user-facing functionality.

Key Features

Flexible Execution

The recipe supports parameterized testing:

• test_phases: Select specific test categories or run all
• test_depth: Choose between quick, standard, or exhaustive testing
• parallel_tests: Enable/disable parallel test execution
• workspace_dir: Specify test artifact location

Self-Documenting

The test generates comprehensive reports:

• Detailed logs for each phase
• Summary statistics
• Pass/fail status for each capability
• Terminal-displayed executive summary

Clean Artifacts

Test artifacts are organized in a single gooseselftest directory, which is automatically added to .gitignore to keep the repository clean.

Why This Matters

Continuous Validation

Provides a standardized method to verify goose functionality across different:

• Environments
• Configurations
• Model providers
• Operating systems

Behavioral Testing

Unlike unit tests that verify code correctness, this tests actual agent behavior - crucial for AI systems where behavior can vary with context and model.

Meta-Cognitive Assessment

The successful completion of self-testing demonstrates goose's ability to:

• Understand complex instructions
• Coordinate multiple tools
• Maintain context across operations
• Reason about its own capabilities

Quality Assurance

Enables rapid validation after:

• Code changes
• Dependency updates
• Configuration modifications
• New feature additions

Initial Validation

The recipe has been successfully tested with:

./target/release/goose run --recipe goose-self-test.yaml --params test_phases=basic --params test_depth=quick

Results from initial testing:

• 21 tests executed
• 100% pass rate
• ~3 minute execution time
• All core developer tools validated

[zanesq]

Nice! Assuming this is cli only right?

[tlongwell-block]

Nice! Assuming this is cli only right?

Yes, this one is. But @DOsinga and I were talking about using playwright to test the desktop app. Will try to explore that in a subsequent PR

[tlongwell-block]

cc @angiejones you might think this new feature is fun

[DOsinga]

can you add it to the checklist @tlongwell-block that we run on a new release?