dev: add tool for comparing benchmarks on two commits

[cpcloud]

Add a tool to compare benchmarks on two commits. Run with pixi run bench-against

[copy-pr-bot]

Auto-sync is disabled for ready for review pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[gmarkall]

/ok to test

[greptile-apps]

Greptile Overview

Greptile Summary

This PR adds a benchmark comparison tool that allows developers to compare performance between two git commits without modifying the current working directory.

• Introduces scripts/bench-against.py which uses git worktrees to safely check out and benchmark two commits
• Adds a new bench-against pixi environment with test dependencies for running the tool
• Adds pixi run bench-against task (Linux only) to invoke the script
• Suppresses PytestBenchmarkWarning about differing machine_info when comparing benchmarks across commits

The tool takes two required arguments (baseline and proposed git refs) and an optional environment flag (defaults to cu-12-9-py312).

Confidence Score: 4/5

• This PR is safe to merge - it adds a new developer tool without modifying existing functionality.
• The implementation is clean and uses git worktrees correctly with proper cleanup. The tool is isolated (new files and environment only) and doesn't affect existing code paths. Minor risk is around edge cases in error handling during worktree operations.
• No files require special attention - the main script is straightforward and follows good practices.

Important Files Changed

File Analysis

Filename	Score	Overview
scripts/bench-against.py	4/5	New benchmark comparison script using git worktrees. Creates temporary worktree to check out baseline and proposed commits, runs benchmarks on each, and compares results. Clean implementation with proper cleanup on exit.
pixi.toml	5/5	Adds new bench-against environment with test features and task to run the benchmark comparison script.
testing/pytest.ini	5/5	Adds filter to ignore PytestBenchmarkWarning about differing machine_info when comparing benchmarks across commits.
pixi.lock	5/5	Lock file updates for new bench-against environment dependencies across linux-64, linux-aarch64, and win-64 platforms.

Sequence Diagram

sequenceDiagram
    participant User
    participant Script as bench-against.py
    participant Git
    participant Pixi
    participant Pytest

    User->>Script: pixi run bench-against <baseline> <proposed>
    Script->>Git: rev-parse --show-toplevel
    Git-->>Script: git root directory
    Script->>Git: worktree add --detach <temp-path> HEAD
    Script->>Script: chdir(worktree_path)
    
    Note over Script: Baseline benchmarks
    Script->>Git: checkout <baseline>
    Script->>Pixi: reinstall -e <env> numba-cuda
    Script->>Pixi: run -e <env> bench
    Pixi->>Pytest: benchmark tests (autosave results)
    
    Note over Script: Proposed benchmarks
    Script->>Git: checkout <proposed>
    Script->>Pixi: reinstall -e <env> numba-cuda
    Script->>Pixi: run -e <env> benchcmp
    Pixi->>Pytest: benchmark tests (compare with saved)
    Pytest-->>User: Comparison results
    
    Script->>Git: worktree remove --force <temp-path>
    Script->>Script: chdir(original_dir)

Loading

[greptile-apps]

_{2 files reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}

[cpcloud]

I am not sure why pixi thinks the lockfile isn't up-to-date.

[cpcloud]

/ok to test

[greptile-apps]

_{2 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

_{3 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[cpcloud]

@gmarkall I am happy to add a smoke test for this if you'd like, but I'd like to use it on #600 before doing so if that's okay.

[greptile-apps]

_{3 files reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

_{3 files reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

_{3 files reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}

[gmarkall]

This works for me - many thanks!

[gmarkall]

/ok to test

[greptile-apps]

_{3 files reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}

[gmarkall]

/ok to test

[greptile-apps]

_{3 files reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}