[red-knot] Run py-fuzzer in CI to check for new panics

[AlexWaygood]

Summary

This PR sets up a CI job that uses the py-fuzzer script to check whether a PR introduces new panics to red-knot.

The py-fuzzer script still finds lots of panics in red-knot generally (see astral-sh/ty#228 for a tracking issue), so running it with the default arguments in CI would lead to a lot of red CI jobs. However, the py-fuzzer script also has a --only-new-bugs CLI flag! If specified, it allows you to compare two binaries and check whether any new panics have been introduced by the PR. If any have been introduced, the script fails; if there are no new panics relative to the baseline binary, however, the script exits with code 0.

The CI job here ends up looking like a cross between the fuzz-parser CI job in ci.yaml (which already runs the py-fuzzer script in CI for us, but which runs it with the ruff binary to fuzz the parser rather than red-knot) and the ruff-ecosystem CI job in ci.yaml (which, in a similar way to this new CI job, downloads two binaries and compares them to see if there's any differences).

The CI job is currently failing because there's no baseline red-knot binary for the job to download (this PR adds the CI job that will start uploading the red-knot binary for future CI jobs to download). I'm not totally sure how to address this -- I suppose I could split it up into two PRs, one that adds the job uploading the binary and then a second one that adds the job to actually use the binary? I'll wait for review on the general idea here before going ahead with that, though.

Test Plan

I did the following:

• Built the binary on main with --target-directory=target/main

• Switched to a branch that had more panics and ran cargo build -p red_knot

• Ran the following command from my terminal and observed that the script only reported new panics that did not exist on main:

  uvx \
      --python="3.12" \
      --from=./python/py-fuzzer \
      fuzz \
      --test-executable="./target/debug/red_knot" \
      --baseline-executable="./target/main/debug/red_knot" \
      --only-new-bugs \
      --bin=red_knot \
      0-500
  

[github-actions]

ruff-ecosystem results

Linter (stable)

✅ ecosystem check detected no linter changes.

Linter (preview)

✅ ecosystem check detected no linter changes.

Formatter (stable)

✅ ecosystem check detected no format changes.

Formatter (preview)

✅ ecosystem check detected no format changes.

[dhruvmanila]

The CI job is currently failing because there's no baseline red-knot binary for the job to download (this PR adds the CI job that will start uploading the red-knot binary for future CI jobs to download). I'm not totally sure how to address this -- I suppose I could split it up into two PRs, one that adds the job uploading the binary and then a second one that adds the job to actually use the binary? I'll wait for review on the general idea here before going ahead with that, though.

Or, you could create a dummy PR with either an empty commit or a random change (if it's required) to trigger CI which is stacked on top of this PR.

[AlexWaygood]

Or, you could create a dummy PR with either an empty commit or a random change (if it's required) to trigger CI which is stacked on top of this PR.

I split this PR into two -- this change is now stacked on top of #17720 (so that there is a baseline binary to download), and CI is passing on the new job 🎉 https://github.com/astral-sh/ruff/actions/runs/14740981444/job/41378854533?pr=17719#step:6:539

[carljm]

Thank you!

[dhruvmanila]

Yeah, and it's most useful to run it on a PR. We also don't run the py-fuzzer on parser on main.

[codspeed-hq]

CodSpeed Performance Report

Merging #17719 will improve performances by 4.1%

_{Comparing alex/panic-regrtest (8c563cc) with main (8c68d30)}

Summary

⚡ 1 improvements
✅ 32 untouched benchmarks

Benchmarks breakdown

	Benchmark	BASE	HEAD	Change
⚡	parser[numpy/ctypeslib.py]	924.5 µs	888 µs	+4.1%

[AlexWaygood]

Merging #17719 will improve performances by 4.1%

hmmmmm