Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Audit run-fail ui tests to see if they need *exact* run stderr/stdout #134889

Open
jieyouxu opened this issue Dec 29, 2024 · 2 comments
Open

Audit run-fail ui tests to see if they need *exact* run stderr/stdout #134889

jieyouxu opened this issue Dec 29, 2024 · 2 comments
Labels
A-testsuite Area: The testsuite used to check the correctness of rustc C-cleanup Category: PRs that clean code up or issues documenting cleanup. E-medium Call for participation: Medium difficulty. Experience needed to fix: Intermediate. E-tedious Call for participation: An issue involves lots of work and is better handled as many small tasks. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.

Comments

@jieyouxu
Copy link
Member

compiletest triage: I'm not convinced switching error-pattern run-fail ui tests to use only the current form of check-run-results is an improvement over error-pattern unconditionally. If anything, it would be a regression to flaky tests in many cases that are sensitive to exact run stdout/stderr (line numbers, backtraces that can depend on platform-specific unwind mechanism, concrete addresses, version numbers, etc.).

I would be in favor of auditing some of these run-fail tests and sparingly add check-run-results if the exact run stderr/stdout is critical for the intention of the test, but not in a blanket fashion. As such, I'm going to close this issue in favor of a more concrete audit issue.

Originally posted by @jieyouxu in #65865

@jieyouxu jieyouxu added A-testsuite Area: The testsuite used to check the correctness of rustc E-tedious Call for participation: An issue involves lots of work and is better handled as many small tasks. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Dec 29, 2024
@rustbot rustbot added the needs-triage This issue may need triage. Remove it if it has been sufficiently triaged. label Dec 29, 2024
@jieyouxu jieyouxu removed the needs-triage This issue may need triage. Remove it if it has been sufficiently triaged. label Dec 29, 2024
@jieyouxu jieyouxu added the C-cleanup Category: PRs that clean code up or issues documenting cleanup. label Dec 29, 2024
@jieyouxu jieyouxu added the E-hard Call for participation: Hard difficulty. Experience needed to fix: A lot. label Dec 29, 2024
@jieyouxu
Copy link
Member Author

jieyouxu commented Dec 29, 2024

E-medium because you need some experience to not accidentally make a test flaky and to know if a test needs to check for exact run stderr/stdout.

@jieyouxu jieyouxu added E-medium Call for participation: Medium difficulty. Experience needed to fix: Intermediate. and removed E-hard Call for participation: Hard difficulty. Experience needed to fix: A lot. labels Dec 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-testsuite Area: The testsuite used to check the correctness of rustc C-cleanup Category: PRs that clean code up or issues documenting cleanup. E-medium Call for participation: Medium difficulty. Experience needed to fix: Intermediate. E-tedious Call for participation: An issue involves lots of work and is better handled as many small tasks. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.
Projects
None yet
Development

No branches or pull requests

2 participants