Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rewrite the compiletest directive parser #128070

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

tgross35
Copy link
Contributor

It has come up a few times that the way we handle files compile test files has some shortcomings. This is the start of an attempt to rewrite it in a way that gives us (hopefully) a lot more flexibility to identify errors.

It uses a new parser that is pretty simple, with this basic architecture:

  • Try to parse each line as an Item, which is anything we care about: header directives, UI error annotations, LLVM filecheck annotations (so we can validate those).

  • If nothing shows up, try to parse as a near miss. This should catch errors in the syntax and things like typos.

  • If it didn't match an Item nor its similar parser, ignore the line

  • Build these all into a Vec<Item>, the output of itemlist.rs

  • We can then do a handful of passes to either check for errors or perform some behavior. Some important ones:

    • Populate items into a default Config
    • Build revision-specific Config
    • Extract and validate UI directives
    • Extract and validate filecheck directives

    The goal at this point is to catch anything that is correct syntax but invalid for whatever reason, and to construct configuration to run tests.

  • Rather than panic!, errors get raised and tagged with location info (line, col, file) as they bubble up.

I think this should help solve a number of problems:

It is pretty early and I haven't started connecting everything into the existing infrastructure, but I just wanted to get your feedback before I get too far along @jieyouxu. So please don't read the code too much, not super clean :)

r? @jieyouxu

@rustbot rustbot added A-testsuite Area: The testsuite used to check the correctness of rustc S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) labels Jul 22, 2024
@rust-log-analyzer
Copy link
Collaborator

The job mingw-check-tidy failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)
Getting action download info
Download action repository 'msys2/[email protected]' (SHA:cc11e9188b693c2b100158c3322424c4cc1dadea)
Download action repository 'actions/checkout@v4' (SHA:692973e3d937129bcbf40652eb9f2f61becf3332)
Download action repository 'actions/upload-artifact@v4' (SHA:0b2256b8c012f0828dc542b3febcab082c67f72b)
Complete job name: PR - mingw-check-tidy
git config --global core.autocrlf false
shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
---
COPY scripts/sccache.sh /scripts/
RUN sh /scripts/sccache.sh

COPY host-x86_64/mingw-check/reuse-requirements.txt /tmp/
RUN pip3 install --no-deps --no-cache-dir --require-hashes -r /tmp/reuse-requirements.txt \
    && pip3 install virtualenv
COPY host-x86_64/mingw-check/validate-toolstate.sh /scripts/
COPY host-x86_64/mingw-check/validate-error-codes.sh /scripts/

# NOTE: intentionally uses python2 for x.py so we can test it still works.
# NOTE: intentionally uses python2 for x.py so we can test it still works.
# validate-toolstate only runs in our CI, so it's ok for it to only support python3.
ENV SCRIPT TIDY_PRINT_DIFF=1 python2.7 ../x.py test \
           --stage 0 src/tools/tidy tidyselftest --extra-checks=py:lint,cpp:fmt
# This file is autogenerated by pip-compile with Python 3.10
# by the following command:
#
#    pip-compile --allow-unsafe --generate-hashes reuse-requirements.in
---

#12 [5/8] COPY host-x86_64/mingw-check/reuse-requirements.txt /tmp/
#12 DONE 0.0s

#13 [6/8] RUN pip3 install --no-deps --no-cache-dir --require-hashes -r /tmp/reuse-requirements.txt     && pip3 install virtualenv
#13 0.435   Downloading binaryornot-0.4.4-py2.py3-none-any.whl (9.0 kB)
#13 0.446 Collecting boolean-py==4.0
#13 0.449   Downloading boolean.py-4.0-py3-none-any.whl (25 kB)
#13 0.460 Collecting chardet==5.1.0
---
#13 3.803 Building wheels for collected packages: reuse
#13 3.804   Building wheel for reuse (pyproject.toml): started
#13 4.147   Building wheel for reuse (pyproject.toml): finished with status 'done'
#13 4.148   Created wheel for reuse: filename=reuse-1.1.0-cp310-cp310-manylinux_2_35_x86_64.whl size=181117 sha256=f5f58750481f69515c2c0d1d503daf565e2565c370d07fc6aeb95fe3498b4269
#13 4.148   Stored in directory: /tmp/pip-ephem-wheel-cache-ig6lip_w/wheels/c2/3c/b9/1120c2ab4bd82694f7e6f0537dc5b9a085c13e2c69a8d0c76d
#13 4.150 Installing collected packages: boolean-py, binaryornot, setuptools, reuse, python-debian, markupsafe, license-expression, jinja2, chardet
#13 4.174   Attempting uninstall: setuptools
#13 4.174     Found existing installation: setuptools 59.6.0
#13 4.176     Not uninstalling setuptools at /usr/lib/python3/dist-packages, outside environment /usr
---
#13 5.457   Downloading virtualenv-20.26.3-py3-none-any.whl (5.7 MB)
#13 5.532      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.7/5.7 MB 78.1 MB/s eta 0:00:00
#13 5.588 Collecting filelock<4,>=3.12.2
#13 5.591   Downloading filelock-3.15.4-py3-none-any.whl (16 kB)
#13 5.607 Collecting distlib<1,>=0.3.7
#13 5.612   Downloading distlib-0.3.8-py2.py3-none-any.whl (468 kB)
#13 5.619      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 468.9/468.9 KB 88.3 MB/s eta 0:00:00
#13 5.646 Collecting platformdirs<5,>=3.9.1
#13 5.649   Downloading platformdirs-4.2.2-py3-none-any.whl (18 kB)
#13 5.739 Installing collected packages: distlib, platformdirs, filelock, virtualenv
#13 5.925 Successfully installed distlib-0.3.8 filelock-3.15.4 platformdirs-4.2.2 virtualenv-20.26.3
#13 DONE 6.0s

#14 [7/8] COPY host-x86_64/mingw-check/validate-toolstate.sh /scripts/
#14 DONE 0.0s
---
DirectMap4k:      219072 kB
DirectMap2M:     8169472 kB
DirectMap1G:    10485760 kB
##[endgroup]
Executing TIDY_PRINT_DIFF=1 python2.7 ../x.py test            --stage 0 src/tools/tidy tidyselftest --extra-checks=py:lint,cpp:fmt
+ TIDY_PRINT_DIFF=1 python2.7 ../x.py test --stage 0 src/tools/tidy tidyselftest --extra-checks=py:lint,cpp:fmt
    Finished `dev` profile [unoptimized] target(s) in 0.04s
##[endgroup]
downloading https://ci-artifacts.rust-lang.org/rustc-builds-alt/20f23abbecd7ac5e04dd7ccadc29c3824e28a269/rust-dev-nightly-x86_64-unknown-linux-gnu.tar.xz
extracting /checkout/obj/build/cache/llvm-20f23abbecd7ac5e04dd7ccadc29c3824e28a269-true/rust-dev-nightly-x86_64-unknown-linux-gnu.tar.xz to /checkout/obj/build/x86_64-unknown-linux-gnu/ci-llvm
---
fmt check
fmt: checked 5354 files
tidy check
tidy: Skipping binary file check, read-only filesystem
tidy error: `/checkout/src/tools/compiletest/src/load_cfg/test_itemlist.rs:166` contains `#[test]`; unit tests and benchmarks must be placed into separate files or directories named `tests.rs`, `benches.rs`, `tests` or `benches`
tidy error: /checkout/src/tools/compiletest/src/load_cfg/test_prepare.rs: leading newline
tidy error: /checkout/src/tools/compiletest/src/load_cfg/test_prepare.rs: too many trailing newlines (2)
##[error]tidy error: /checkout/src/tools/compiletest/src/load_cfg/prepare.rs:33: TODO is used for tasks that should be done before merging a PR; If you want to leave a message in the codebase use FIXME
##[error]tidy error: /checkout/src/tools/compiletest/src/load_cfg/prepare.rs:139: TODO is used for tasks that should be done before merging a PR; If you want to leave a message in the codebase use FIXME
##[error]tidy error: /checkout/src/tools/compiletest/src/load_cfg/prepare.rs:423: TODO is used for tasks that should be done before merging a PR; If you want to leave a message in the codebase use FIXME
##[error]tidy error: /checkout/src/tools/compiletest/src/load_cfg/prepare.rs:469: TODO is used for tasks that should be done before merging a PR; If you want to leave a message in the codebase use FIXME
##[error]tidy error: /checkout/src/tools/compiletest/src/load_cfg/itemlist.rs:91: TODO is used for tasks that should be done before merging a PR; If you want to leave a message in the codebase use FIXME
##[error]tidy error: /checkout/src/tools/compiletest/src/load_cfg/itemlist.rs:380: trailing whitespace
removing old virtual environment
creating virtual environment at '/checkout/obj/build/venv' using 'python3.10'
Requirement already satisfied: pip in ./build/venv/lib/python3.10/site-packages (24.1)
  Downloading pip-24.1.2-py3-none-any.whl.metadata (3.6 kB)
Downloading pip-24.1.2-py3-none-any.whl (1.8 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.8/1.8 MB 63.1 MB/s eta 0:00:00
Installing collected packages: pip
---
All checks passed!
checking C++ file formatting
some tidy checks failed

Command LD_LIBRARY_PATH="/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-bootstrap-tools/x86_64-unknown-linux-gnu/release/deps:/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/lib" RUSTC="/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/rustc" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/rust-tidy" "/checkout" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "/checkout/obj/build" "4" "--extra-checks=py:lint,cpp:fmt" (failure_mode=DelayFail, stdout_mode=Print, stderr_mode=Print) did not execute successfully.
Created at: src/core/build_steps/tool.rs:1109:23
Executed at: src/core/build_steps/test.rs:1084:29

Build completed unsuccessfully in 0:01:16

@jieyouxu
Copy link
Member

jieyouxu commented Jul 22, 2024

Thank you for taking this on! I've been wanting to do this myself, but never quite gotten around to it. I'll take a closer look at the approach idea tomorrow. I want to do a survey of what compiletest currently does, and look at the problems before I look at your solution sketch. That is, I want to try to come up independently with a potential solution sketch first, and then compare notes with what you have here in case we miss things or have other considerations.

@jieyouxu
Copy link
Member

jieyouxu commented Jul 22, 2024

I have not yet looked at your PR specifics (intentionally as aforementioned), but here are some of the thoughts I had previously (I will check against this when I actually look at your solution sketch):

  • I think we want a clean phase separation:
    • Explanation: we probably want to separate out the various steps involved in eventually running a concrete test. That is, instead of the current entangled logic with test collection / building / running / output processing / status calculation etc. all kinda intermingled together, we may want to "quarantine" each step out. I really want the steps to be testable in isolation (as unit tests) as well as collectively (as smoke tests / integration tests). Currently, there are "early" props (directives) and "late" directives, which is extra confusing to follow. The late props might also only fail when you actually try to run the test, I'm not too sure.
    • Key idea: we want directive handling to be easy to test and easy to follow.
  • I think we probably want test suites (including child test runners) to declaratively register which directives are supported by the test suite.
    • Remark: this may want to have compiletest-generic preset directives, test mode preset directives versus test suite specific directives
  • We want directive handling to be robust, strict and have good error reporting.
    • Remark: I've wanted to change the ignore-*/only-* directives for targets to become ignore-target-*, because it's possible that in ignore-foo foo could be both a target substring and another ignore kind, then you have ambiguity which is resolved by either failing or somehow having a precedence (neither of which are ideal).
    • Remark: some directive syntaxes are not documented at all, have ambiguities, or are not robust (consider revisions).
  • We want to fail-fast and give helpful suggestions where possible.
  • We want directives to be documented: their syntax and behavior.
  • We might not want to use regex where not needed -- or change the syntax such that regex isn't needed -- it's very hard to understand at first glance and is very subtle.
    • Remark: as I understand it, we may have directives which want syntax that are not necessarily regular (in the simple regular expression sense, not PCRE sense) that may warrant improved parsing because we may want to support escapes. A lot of the directives use split-by-whitespace, which does not handle string escapes!
  • I think directive handling includes validation, but probably better as a separate step after parsing the intended syntax of the directive.

@jieyouxu
Copy link
Member

jieyouxu commented Jul 22, 2024

FYI: there has been ideas to eventually migrate to ui_test, but compiletest is sufficiently complex that I think we'll have to implement custom stuff on top of ui_test anyway. In any case, we might be able to take inspiration from ui_test's directive handling (if we feel like it makes sense too for compiletest).

cf. a series of MCPs:

@jieyouxu
Copy link
Member

After I take an initial look, I'll post this PR for discussion to the #t-compiler zulip thread, this could use a few more eyes and brains on a solution sketch and see how we can make this more robust and maintainable (whatever we have now is the opposite of maintainable unfortunately, it's mostly due to organic growth).

@tgross35
Copy link
Contributor Author

tgross35 commented Jul 22, 2024

Just some quick replies:

I have not yet looked at your PR specifics (intentionally as aforementioned)

Well you can look at it 😆 only thing is the code isn't as clean as it could be - functions not in logical ordering, rough namings, TODOs. itemlist.rs (the initial parser) is reasonably complete, prepare.rs (next stage) slightly less so.

but here are some of the thoughts I had previously (I will check against this when I actually look at your solution sketch):

* I think we want a clean **phase separation**:
  
  * _Explanation_: we probably want to separate out the various steps involved in eventually running a concrete test. That is, instead of the current entangled logic with test collection / building / running / output processing / status calculation etc. all kinda intermingled together, we may want to "quarantine" each step out. I really want the steps to be testable in isolation (as unit tests) as well as collectively (as smoke tests / integration tests). Currently, there are "early" props (directives) and "late" directives, which is extra confusing to follow. The late props might also only fail when you actually try to run the test, I'm not too sure.
  * _Key idea_: we want directive handling to be **easy to test** and **easy to follow**.

With respect to this, are you only talking about handling of the directives within compiletest or do you mean changes to the tests themselves? Is this just a separation of Config/Testprops-like structures?

At least some of this can probably be accomplished here as far as making it easier to split things out, but I think I might need an example to see exactly what you mean.

* I think we probably want test suites (including child test runners) to **declaratively register** which directives are supported by the test suite.
  
  * _Remark_: this may want to have **compiletest-generic** preset directives, **test mode preset** directives versus **test suite specific** directives

Totally possible, this can be filtered either at parse time or as a pass at validation time. At its basic level, this can just be a pass that populates test-specific Config from Items, and rejects anything that doesn't apply to that test.

* We want directive handling to be **robust**, **strict** and have **good error reporting**.
  
  * _Remark_: I've wanted to change the `ignore-*`/`only-*` directives for targets to become `ignore-target-*`, because it's possible that in `ignore-foo` `foo` could be both a target substring and another ignore kind, then you have ambiguity which is resolved by either failing or somehow having a precedence (neither of which are ideal).
  * _Remark_: some directive syntaxes are not documented at all, have ambiguities, or are not robust (consider revisions).

* We want to **fail-fast** and give **helpful suggestions** where possible.

I think that the prefix directives could probably use some thought about what should be accepted. Maybe they take some other form like //@ ignore: x86.

However, regarding fail-fast and helpful suggestions: yes, huge goal here! E.g. typing //@ rustc-en will recommend //@ rustc-env, the matcher for UI directives tries to find things that look like like directives but aren't (and otherwise would be skipped), and a list of all syntax errors in the file is reported early.

* We want directives to be **documented**: their syntax and behavior.

As in, autogenerated? I had a few ideas for this but didn't yet implement it.

* We might not want to use **regex** where not needed -- or change the syntax such that regex _isn't_ needed -- it's very hard to understand at first glance and is very subtle.

Here, regex is limited to a few places that have a somewhat wide range of valid inputs:

  • UI directives
  • Filecheck directives
  • Revision name validity checking

Everything else is just standard string operations.

  * _Remark_: as I understand it, we may have directives which want syntax that are not necessarily _regular_ (in the simple regular expression sense, not PCRE sense) that may warrant improved parsing because we may want to support escapes. A lot of the directives use split-by-whitespace, which does not handle string escapes!

Could you give an example of this, assuming there is more than just normalize? Plugging in a separate splitter function sounds easy enough, currently splitting on whitespace is only used for revision names (but I also haven't implemented a lot of the areas that may make use of it).

@tgross35
Copy link
Contributor Author

Relevant compiletest issues:

Not all of these are related to parsing, but I think we may be able to generally handle them better:

Currently, uppercase directives that start with CHECK are accepted, anything in the form // foo: causes an error as something that probably should be a filecheck directive. This was part of my motivation with #128018.

MatchItem::re_global(
|| {
let ok_template = r"
COMMENT \s*
(?P<directive> CHECK[A-Z-]* ) \s*:
(?P<content> .+ )?
";
cty_re_from_template(ok_template, "COMMENT")
},
// Checking for `// .*:` might be too restrictive,
Some(|| cty_re_from_template(r"COMMENT //\s*\S*\s*:", "COMMENT")),
|c| ItemVal::FileCheckDirective {
directive: c.name("directive").unwrap().as_str().trim(),
content: c.name("content").map(|m| m.as_str().trim()),
},
"usage: `//~ LABEL`, `//[revision]~ LABEL`, `//[rev1,rev2]~ LABEL`, ...",
),

(first item is regex that matches an item pattern, second is the fallback for near misses to warn on, third constructs the item from the regex capture)

Maybe we could do something better, unsure what that would be.

The syntax errors/warnings could be addressed here easily enough. I think I have this raise an error if you type //@ foo bar but //@ foo: (colon) is a valid directive (need to double check if I actually did that).

Absolutely a goal here.

Easy enough to add.

I am not sure exactly where this behavior comes from, but seems like it might be fixable now even?

Almost solved - the //@ revisions: header gets its value split by whitespace, but then revision names are checked against a regex. Unfortunately I realize now that my REV_NAME_PAT contains , which is just wrong.

fn pass_extract_revisions(items: &[Item], pcx: &mut PassCtx) -> Result<()> {
let mut iter = items.iter().filter_map(|v| matches!(v.val, ItemVal::Revisions(_)).then_some(v));
let Some(first) = iter.next() else {
// Revisions not specified
pcx.revisions = Some(Vec::new());
return Ok(());
};
assert!(iter.count() == 0, "duplicates should have been checked already");
first.used.set(true);
let ItemVal::Revisions(all_revs) = first.val else {
unreachable!("filtered above");
};
let revs: Vec<_> = all_revs.split_whitespace().map(Rc::from).collect();
for idx in 0..revs.len() {
let name = &revs[idx];
if !REV_NAME_RE.is_match(&name) {
Err(format!("revision '{name}' is not valid. Expected: `{REV_NAME_PAT}`"))?;
}
if revs[..idx].contains(&name) {
Err(format!("revision '{name}' is listed twice"))?;
}
}
pcx.revisions = Some(revs);
Ok(())
}

Should raise an error, but I need to add a test here.

@jieyouxu
Copy link
Member

jieyouxu commented Jul 22, 2024

Yeah, I was just jotting down some thoughts and info/context I had so they existed somewhere, you don't necessarily need to respond :3

Also I would maybe wait a bit on the overall design direction, no need to fix the details just yet

@tgross35
Copy link
Contributor Author

You collecting it together is much appreciated :)

I guess the decision of whether to use ui_test or to try to improve what is in-tree remains the biggest question still.

@jieyouxu
Copy link
Member

jieyouxu commented Jul 22, 2024

You collecting it together is much appreciated :)

I guess the decision of whether to use ui_test or to try to improve what is in-tree remains the biggest question still.

Err sorry let me clarify, ui_test only supports ui tests currently (as in the ui test suite style tests), it wouldn't replace compiletest 1-to-1, so we still want and need compiletest to be maintainable in any forseeable future. So yes, this PR is very valuable. If anything, ui_test might be what compiletest uses under the hood (for ui tests and perhaps more), but compiletest definitely isn't going away.

@jieyouxu
Copy link
Member

I've reposted my previous comment on #t-compiler, and I would move this discussion to the zulip stream for more visiblity to other T-compiler people. Link: https://rust-lang.zulipchat.com/#narrow/stream/131828-t-compiler/topic/Reworking.20compiletest.20directive.20parsing.

const fn directive(self) -> &'static str {
match self {
CommentTy::Slashes => "//@",
CommentTy::Hash => "#@",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remark: unfortunately, Makefiles are still using the same directive prefix # as comments #, because I didn't try to port them when I ported // -> //@ since we're porting Makefiles into rmake.rs anyway.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is an interesting twist - I think it would be easy enough to just ignore unknown directives for specific kinds of tests rather than raising errors. Unless #121876 happens to be done sooner than expected.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that makes sense to me. We don't really need to error for existing Makefile tests, and at least we are guarding the test suite with a tidy check to disallow new Makefiles to make it in.

@jieyouxu
Copy link
Member

It uses a new parser that is pretty simple, with this basic architecture:

  • Try to parse each line as an Item, which is anything we care about: header directives, UI error annotations, LLVM filecheck annotations (so we can validate those).

There might also be rustdoc annotations, but yes that sounds about right to me.

  • If nothing shows up, try to parse as a near miss. This should catch errors in the syntax and things like typos.

I'm not 100% sure you mean by this, do you mean //@ idk-what-this-is would not error if it did
not look sufficiently similar to registered directives?

  • If it didn't match an Item nor its similar parser, ignore the line

If the comment starts with //@ but fails to match any registered directives, we should hard
error.

  • We can then do a handful of passes to either check for errors or perform some behavior. Some important ones:

    • Populate items into a default Config
    • Build revision-specific Config
    • Extract and validate UI directives
    • Extract and validate filecheck directives

The goal at this point is to catch anything that is correct syntax but invalid for whatever reason, and to construct configuration to run tests.

Yes, sounds good.

  • Rather than panic!, errors get raised and tagged with location info (line, col, file) as they bubble up.

I think this should help solve a number of problems:

  • The near-miss parser gives us a way to suggest fixes for typos (using strsim) or to catch things that look like they are supposed to be UI directives but aren't (e.g. //~[revision] instead of //[revision]~)

Yeah, we do want to be careful about how much false positives (looks like directive but are actually intended comments) this detection does.

Yes, that's intended as a short term bandaid.

  • We should be able to identify UI and filecheck directives that aren't being used and warn on them

Yes, definitely. In general, we want to reject directives that are invalid for a test suite, as well
as invalid combinations of directives.

  • ...some other things that I definitely remember talking about but can't remember what they were. (can you remind me?)

I can't immediately recall either, sorry! But e.g. currently //@ compile-flags: -C incremental=...
should use //@ incremental instead in ui tests, the compile-flags incremental misbehaves.

It is pretty early and I haven't started connecting everything into the existing infrastructure, but I just wanted to get your feedback before I get too far along @jieyouxu. So please don't read the code too much, not super clean :)

I do like how this looks a lot.

Copy link
Member

@jieyouxu jieyouxu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some partial reviews (I haven't gotten through the whole PR)

match self {
CommentTy::Slashes => "//@",
CommentTy::Hash => "#@",
CommentTy::Semi => ";@",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question: I don't think I've ported any ; <directive-names> directives over either, are these for like assembly tests?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking LLVM of IR, but duh that's not our job to test 🤦

Comment on lines +62 to +68
#[derive(Debug)]
struct FullError {
msg: Box<str>,
fname: Option<PathBuf>,
pos: Option<LineCol>,
context: Vec<Box<str>>,
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remark: I really like having better context for error messages 👍

Comment on lines +59 to +60
pub type Error = Box<dyn std::error::Error>;
pub type Result<T, E = Error> = std::result::Result<T, E>;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remark: we might even use some kind of anyhow/thiserror in the future.

// TODO: parse revisions into `ItemVal`s.

#[derive(Clone, Debug, PartialEq)]
pub enum ItemVal<'src> {
Copy link
Member

@jieyouxu jieyouxu Jul 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Discussion: so this is what I had in mind when I mentioned test suite specific directives: some of these directives make sense for ui/ui-fulldeps but not for run-make, for example, and I think we might want to separate ItemVal (I would even just call this a Directive?) for specific test suites.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Directive does make sense, my naming isn't as clean as it could be 😄. I think that having all directives listed in one place simplifies things, but I was planning to handle this. roughly:

trait TestKind {
    type Config;
    /// Called within `itemlist` to filter which directives we look for
    fn accepts_directive(m: &MatchItem /* I know, MatchItem is a bad name */) -> bool;

    /// Called within `prepare` to set its own config
    fn visit_build_config(item: &Item, cfg: &mut Self::Config) -> Result<()>;
}

This would be the only test-specific interface needed - just needs to say whether or not it supports a specific directive kind, and then say how to update its config when it gets to relevant items.

(Still mind mapping that bit out, but I think this seems reasonably clean and encapsulated)

Comment on lines +150 to +168
/* Directives that are single keys but have variable suffixes */
/// Ignore something specific about this test.
///
/// `//@ ignore-x86`
Ignore {
what: &'src str,
},
/// Components needed by the test
///
/// `//@ needs-rust-lld`
Needs {
what: &'src str,
},
/// Run only if conditions are met
///
/// `//@ only-linux`
Only {
what: &'src str,
},
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion (not necessarily for this PR): only-* and ignore-* are very tricky (if we look at header/cfg.rs) in that the suffixes are parsed with a sequence of precedence-based matchers. If you specify e.g. ignore-x86, you have to look and reason really hard to realize "oh it's the target_cfg.all_archs that's matching and taking effect". We probably want to eventually straight up ask for a longer prefix to help make it excruciatingly clear what we're trying to match on (this would actually better match ui_test)

//@ only-arch-x86

///
/// `//@ normalize-stdout: foo -> bar`
/// `//@ normalize-stderr-32bit: bar -> baz`
/// `//@ normalize-stderr-test: "h[[:xdigit:]]{16}" -> "h[HASH]"`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Discussion: I don't recall, does normalize-* handle string escapes in regexes correctly?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't yet gotten to the reimplementation yet, but I think it just does a regex match. Meaning if there are multiple -> in the string, results might be surprising.

We should probably figure out some quoting rules and use them everywhere relevant. Do you think it makes sense to do something like shell, '...' for literals and "..." allowing escapes? (and interpolations, but hopefully not relevant here).

///
/// `//@[revname] directive`
RevisionSpecificItems {
revs: &'src str,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Discussion: we probably want revisions to be represented as BTreeSet<&'src str>, maybe. Or better yet, we could try to do something similar to ui_test and have revision-specific directive set:

enum UiRevision {
    None,
    Named(String),
}

/// Revision specific?
struct UiDirectiveSet {
    check_pass: bool,
    edition: String,
    // ...
}

type UiDirectives = BTreeMap<UiRevision, UiDirectiveSet>;

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ui_test form is almost exactly what I was going for, just haven't gotten to yet :) my plan is to parse all config that is not specific to revisions into one Config, then clone it per revision, then update the config with anything that is per revision (output lives around here https://github.com/rust-lang/rust/pull/128070/files#diff-4653d6449a3cc26e6235d71fbf2919348e09fe8eaa72db6fc8e623eaa92ce1f5R38-R43, didn't handle the 0-revision case yet).

Comment on lines +220 to +221
/// True if this is an `//@` directive, i.e. something in a header`
pub fn is_header_directive(&self) -> bool {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Discussion: compiletest mixes the terms directives, headers and cfgs, but AFAICT they are trying to say the same thing. Headers do not, in fact, have to appear in the head of the file (that was a historical restriction AFAIK).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have just been using "header directives" to refer to //@ things, "UI directives" to refer to //~ ERROR things, and "filecheck directives" to refer to // CHECK: things. But I agree this isn't as clear as it could be (nor consistent across documentation) - do you have preferred names? "Config directive" instead of "header directives" maybe.

(I actually currently enforce that config directives are above UI or filecheck directives as kind of a tidy thing, since having config directives later in the file seems easy to miss. I did see on Zulip though that sometimes you want to be able to append directives - maybe we can leave the check as-is but add a --no-check-directive-order flag to opt out. Seems like possible overkill, not sure if that is worth it).

@jieyouxu
Copy link
Member

@jieyouxu hey dude this needs another review pass, don't forgor

@tgross35
Copy link
Contributor Author

Oh I also need to actually work on it more, just haven't gotten back to it yet :)

@jieyouxu
Copy link
Member

jieyouxu commented Aug 16, 2024

I keep meaning to do another review pass but keep forgor-ing. Maybe just wait for me a little bit at least for another review pass.

@jieyouxu jieyouxu added S-experimental Status: Ongoing experiment that does not require reviewing and won't be merged in its current state. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Oct 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-testsuite Area: The testsuite used to check the correctness of rustc S-experimental Status: Ongoing experiment that does not require reviewing and won't be merged in its current state. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants