Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Audit {check-run-results, regex-error-pattern, error-pattern, check-stdout, normalize-*, dont-check-compiler-*, run-rustfix, rustfix-only-machine-applicable, forbid-output} family of directives #134888

Open
jieyouxu opened this issue Dec 29, 2024 · 0 comments
Labels
A-compiletest Area: The compiletest test runner A-testsuite Area: The testsuite used to check the correctness of rustc C-bug Category: This is a bug. E-hard Call for participation: Hard difficulty. Experience needed to fix: A lot. E-needs-design This issue needs exploration and design to see how and if we can fix/implement it E-needs-investigation Call for partcipation: This issues needs some investigation to determine current status T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.

Comments

@jieyouxu
Copy link
Member

jieyouxu commented Dec 29, 2024

I don't know about other contributors, but I have to look at the source implementation (or trial-and-error) every time I see or try to use one of these compiletest directives. IMO, we should audit the design of these directives, and possibly revamp them entirely.

Concrete confusions:

  • check-run-results check both run stderr and stdout and puts them into snapshot files (on bless), then compares the subsequent run stderr and stdout against the snapshot.
  • error-pattern doesn't only check stderr (whose stderr? compiler? run?), it can also check stdout (or both??) depending on check-stdout, dont-check-compiler-*, and also it can check also compiler stderr or stdout I think??
  • normalize-* (that is not normalize-stdout or normalize-stderr) I believe can simultaneously apply to {compiler,run} {stderr,stdout}.
  • run-rustfix will run rustfix and try to apply all non-placeholder suggestions, including non-machine-applicable ones like MaybeIncorrect ones.
  • rustfix-only-machine-applicable is like run-rustfix but only tries to apply MachineApplicable suggestions.
  • forbid-output is like error-pattern but named completely differently. I don't remember which output pattern of {compiler,run}x{stderr,stdout} it is forbidding.

EDIT:

  • regex-error-pattern is like error-pattern but accepts a regex...
@jieyouxu jieyouxu added A-compiletest Area: The compiletest test runner C-bug Category: This is a bug. E-hard Call for participation: Hard difficulty. Experience needed to fix: A lot. E-needs-design This issue needs exploration and design to see how and if we can fix/implement it E-needs-investigation Call for partcipation: This issues needs some investigation to determine current status T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Dec 29, 2024
@rustbot rustbot added the needs-triage This issue may need triage. Remove it if it has been sufficiently triaged. label Dec 29, 2024
@jieyouxu jieyouxu removed the needs-triage This issue may need triage. Remove it if it has been sufficiently triaged. label Dec 29, 2024
@jieyouxu jieyouxu added the A-testsuite Area: The testsuite used to check the correctness of rustc label Dec 29, 2024
@jieyouxu jieyouxu changed the title Audit {check-run-results, error-pattern, check-stdout, normalize-*, dont-check-compiler-*, run-rustfix, rustfix-only-machine-applicable, forbid-output} family of directives Audit {check-run-results, regex-error-pattern, error-pattern, check-stdout, normalize-*, dont-check-compiler-*, run-rustfix, rustfix-only-machine-applicable, forbid-output} family of directives Dec 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-compiletest Area: The compiletest test runner A-testsuite Area: The testsuite used to check the correctness of rustc C-bug Category: This is a bug. E-hard Call for participation: Hard difficulty. Experience needed to fix: A lot. E-needs-design This issue needs exploration and design to see how and if we can fix/implement it E-needs-investigation Call for partcipation: This issues needs some investigation to determine current status T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.
Projects
Development

No branches or pull requests

2 participants