Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch run-make tests from Makefiles to rust #40713

Open
4 tasks
TimNN opened this issue Mar 21, 2017 · 13 comments
Open
4 tasks

Switch run-make tests from Makefiles to rust #40713

TimNN opened this issue Mar 21, 2017 · 13 comments
Labels
A-compiletest Area: The compiletest test runner A-run-make Area: port run-make Makefiles to rmake.rs A-testsuite Area: The testsuite used to check the correctness of rustc C-enhancement Category: An issue proposing an enhancement or a PR with one. E-hard Call for participation: Hard difficulty. Experience needed to fix: A lot. E-help-wanted Call for participation: Help is requested to fix this issue. E-mentor Call for participation: This issue has a mentor. Use #t-compiler/help on Zulip for discussion. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue.

Comments

@TimNN
Copy link
Contributor

TimNN commented Mar 21, 2017

Rust currently has a suite of run-make tests, which generally test specific rustc invocations or behaviour, or require external tools (eg. grep or nm).

The goal of this issue is to rewrite these run-make tests, which are currently written as Makefiles, in rust to a) get rid of the dependency on external tools such as make and b) make them more accessible to rust contributors by not requiring arcane knowledge of the make tool.

The transition will require at least the following steps:

  • Survey the existing run-make tests with regard to what they actually do / which programs they use (and how). I assume the programs will likely fall into one of three categories:
    • Rust tools (rustc, rustdoc)
    • Utilities easily replaceable by rust code, eg. grep
    • Complex utilities, eg. nm, which cannot be easily rewritten in rust
  • Design and (partially) implement a support library, which makes the actions identified above easily possible. For example I imagine a test should look something like this:
    extern crate support;
    
    fn main() {
        let s = support::init();
        
        let lib = s.rustc().compile("file1.rs").output_rlib();
        
        assert!(s.nm(lib).filter_lines("some_symbol").count() == 2);
    }
  • Add a new mode to compile-test, maybe run-rmake, which runs the new rust-based run-make tests. This will either include compiling the support library, or receiving the support library from a previous build stage.
  • Start porting the actual run-make tests, which may involve adding additional functionality to the support library. At that time this issue, or another, will track the state of all the existing tests and include some detailed instruction to allow people to easily contributing by porting one of the existing tests.

There are some open questions:

  • Should the support library allow execution of arbitrary commands? Limiting commands to only those explicitly added to the support library would mean that there is a single place which lists the external tools we depend on.
  • Is the proposed integration with compiletest the correct choice? For example the support library itself may have external dependencies (maybe regex or gcc-rs) which means we should probably use cargo to compile the actual tests. At which point it may be worth considering if compile-test is needed at all or if cargo is enough (Have the support library in src/lib.rs, the new run-make tests in tests/*.rs and auxiliary files in subdirectories of tests/ named after the main test file).

If anyone wants to get involved in the process, please leave a comment on this issue or ping me on IRC.


Original Issue Description

Based on a short experiment, it looks like rust on msvc only has five build dependencies: Visual Studio, Git, Python, CMake and make, where make is only used for the run-make tests as far as I can tell.

Of those five, the first four are easily installable natively on windows, whereas make wasn't as straight forward to installe when I tried and required msys2 / mingw.

The questions the are:

  • Is there any interest in performing such a conversion?
  • If so, then which language should those tests be migrated to? Python or Rust seem like the logical choices.
  • How should the switch happen? We'll probably want some kind of incremental strategy, since there are quite a few run-make tests.
@TimNN TimNN added A-build A-testsuite Area: The testsuite used to check the correctness of rustc labels Mar 21, 2017
@petrochenkov
Copy link
Contributor

cc #38565 (comment)

@petrochenkov
Copy link
Contributor

rust on msvc only has five build dependencies: Visual Studio, Git, Python, CMake and make

IIRC, diff was a dependency as well, probably replaceable with git diff

@alexcrichton
Copy link
Member

Oh dear I completely forgot about filing an issue to do this!

I would love to have this implementation, I think we should drop make as fast as we can. The test are horribly difficult to write and easy to get wrong whereas I think the equivalent Rust code would be much nicer to read.

I would discourage use of Python for basically the sole reason that "contributors to rust-lang/rust are likely Rust programmers" in the sense that it's far easier for us as a community to maintain Rust code than Python code (empircally this seems true as well).

I agree an incremental strategy is best, and I think that we've got a lot of flexibility in terms of what we move to. In general I think it should look like:

  • All tests are basically a main.rs script that's the makefile today.
  • Tests can run arbitrary code (this is a crucial part of run-make vs other test suites)
  • Tests are given tons of inputs about the compiler and whatnot
  • I think that we'll want a support library (e.g. librun_make) or something like that with utility functions (sort of like tools.mk) that all the scripts can link to as well if they'd like to.

Given all that I'd imagine this would be best implemented by extending compiletest slightly to compile a librun_make library and then compile scripts with rustc and then execute them (passing variables as appropriate). That way we could add something like src/test/run-make2 and just slowly start transitioning tests over there.

(note that this'd make a fantastic quest issue to migrate over these tests)

@bstrie
Copy link
Contributor

bstrie commented Mar 23, 2017

@alexcrichton Define "quest issue"? Do you mean something like what we did for #35233 ?

@alexcrichton
Copy link
Member

Yep! Precisely like that

@Mark-Simulacrum Mark-Simulacrum added I-nominated T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue. labels Jun 27, 2017
@TimNN TimNN changed the title Interest in switching run-make to python or rust Switch run-make tests from make to rust Jul 8, 2017
@TimNN TimNN changed the title Switch run-make tests from make to rust Switch run-make tests from Makefiles to rust Jul 8, 2017
@TimNN
Copy link
Contributor Author

TimNN commented Jul 8, 2017

I just updated the original issue with a somewhat more detailed plan on going forward, as well as some open questions. Any feedback / thoughts are greatly appreciated.

I will personally start to work on the steps mentioned, although that will have to wait for about two weeks until I finish my bachelor's thesis.

@Mark-Simulacrum Mark-Simulacrum added the C-enhancement Category: An issue proposing an enhancement or a PR with one. label Jul 27, 2017
@kennytm
Copy link
Member

kennytm commented Jul 27, 2017

Some of the run-make tests like crate-data-smoke, codegen-options-parsing and allow-non-lint-warnings-cmdline could be rewritten as ui tests.

Some must still rely on make, mainly dep-info and dep-info-spaces.

bors added a commit that referenced this issue Nov 28, 2017
Replace most call to grep in run-make by a script that cat the input.

Introduced a new `src/etc/cat-and-grep.sh` script (called in run-make as `$(CGREP)`), which prints the input and do a grep simultaneously. This is mainly used to debug spurious failures in run-make, such as the spurious error in #45810, as well as real errors such as #46126.

(cc #40713)

Some `grep` still remains, mainly the `grep -c` calls that count the number of matches and print the result to stdout.
@jonas-schievink jonas-schievink added T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) and removed T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) A-build labels Apr 21, 2019
@matklad
Copy link
Member

matklad commented Jun 26, 2023

Rather than writing our own shell library it might be nice to use https://docs.rs/xshell/latest/xshell/.

Yeah, xshell has the right scope here. It aims to be essentially an "in-process sh + coreutils", exactly for the purpose of doing shell-ly stuff without having any extra deps besides rust-std.

At the same time, currently xshell is more of a prototype and "works for me", rather than an 1.0 grade project. If we want to move to xshell, it'd be best if someone from t-compiler/contributors (except me, that is :-) ) has commit access to the crate and is ready to play a role of an active maintainer.

matthiaskrgr added a commit to matthiaskrgr/rust that referenced this issue Feb 8, 2024
Introduce `run-make` V2 infrastructure, a `run_make_support` library and port over 2 tests as example

See [issue rust-lang#40713: Switch run-make tests from Makefiles to rust](rust-lang#40713) for more context.

### Basic Description of `run-make` V2

`run-make` V2 aims to eliminate the dependency on `make` and `Makefile`s for building `run-make`-style tests. Makefiles are replaced by *recipes* (`rmake.rs`). The current implementation runs `run-make` V2 tests in 3 steps:

1. We build the support library `run_make_support` which the `rmake.rs` recipes depend on as a tool dylib.
2. We build the recipe `rmake.rs` and link in the support library.
3. We run the recipe to build and run the tests.

`rmake.rs` is basically a replacement for `Makefile`, and allows running arbitrary Rust code.

### Planned Changes

- [x] Get rid of the builder style patterns in `rmake_support` and instead use something like

    ```rust
    let output = rustc!(scx, "--cfg x -Cprefer-dynamic -Csymbol-mangling-version=legacy -
    Zunstable-options");
    ```
    as per Nils' suggestion. This can probably use something like `xshell`.
- [x] Make `run_make_support` into a proper crate so it can have external dependencies like `xshell`.
- [x] Instead of having an entire alternative directory `run-make-v2`, change how V2 tests are collected based on presence of `rmake.rs` recipe file. This should ease migration and prevent git history from being messed up by big moves.

### Disclaimer

The current implementation is very much a **very very rough prototype** just to get the 2 example tests working. I would appreciate any feedback on the design and implementation.
matthiaskrgr added a commit to matthiaskrgr/rust that referenced this issue Feb 8, 2024
Introduce `run-make` V2 infrastructure, a `run_make_support` library and port over 2 tests as example

See [issue rust-lang#40713: Switch run-make tests from Makefiles to rust](rust-lang#40713) for more context.

### Basic Description of `run-make` V2

`run-make` V2 aims to eliminate the dependency on `make` and `Makefile`s for building `run-make`-style tests. Makefiles are replaced by *recipes* (`rmake.rs`). The current implementation runs `run-make` V2 tests in 3 steps:

1. We build the support library `run_make_support` which the `rmake.rs` recipes depend on as a tool dylib.
2. We build the recipe `rmake.rs` and link in the support library.
3. We run the recipe to build and run the tests.

`rmake.rs` is basically a replacement for `Makefile`, and allows running arbitrary Rust code.

### Planned Changes

- [x] Get rid of the builder style patterns in `rmake_support` and instead use something like

    ```rust
    let output = rustc!(scx, "--cfg x -Cprefer-dynamic -Csymbol-mangling-version=legacy -
    Zunstable-options");
    ```
    as per Nils' suggestion. This can probably use something like `xshell`.
- [x] Make `run_make_support` into a proper crate so it can have external dependencies like `xshell`.
- [x] Instead of having an entire alternative directory `run-make-v2`, change how V2 tests are collected based on presence of `rmake.rs` recipe file. This should ease migration and prevent git history from being messed up by big moves.

### Disclaimer

The current implementation is very much a **very very rough prototype** just to get the 2 example tests working. I would appreciate any feedback on the design and implementation.
matthiaskrgr added a commit to matthiaskrgr/rust that referenced this issue Feb 8, 2024
Introduce `run-make` V2 infrastructure, a `run_make_support` library and port over 2 tests as example

See [issue rust-lang#40713: Switch run-make tests from Makefiles to rust](rust-lang#40713) for more context.

### Basic Description of `run-make` V2

`run-make` V2 aims to eliminate the dependency on `make` and `Makefile`s for building `run-make`-style tests. Makefiles are replaced by *recipes* (`rmake.rs`). The current implementation runs `run-make` V2 tests in 3 steps:

1. We build the support library `run_make_support` which the `rmake.rs` recipes depend on as a tool dylib.
2. We build the recipe `rmake.rs` and link in the support library.
3. We run the recipe to build and run the tests.

`rmake.rs` is basically a replacement for `Makefile`, and allows running arbitrary Rust code.

### Planned Changes

- [x] Get rid of the builder style patterns in `rmake_support` and instead use something like

    ```rust
    let output = rustc!(scx, "--cfg x -Cprefer-dynamic -Csymbol-mangling-version=legacy -
    Zunstable-options");
    ```
    as per Nils' suggestion. This can probably use something like `xshell`.
- [x] Make `run_make_support` into a proper crate so it can have external dependencies like `xshell`.
- [x] Instead of having an entire alternative directory `run-make-v2`, change how V2 tests are collected based on presence of `rmake.rs` recipe file. This should ease migration and prevent git history from being messed up by big moves.

### Disclaimer

The current implementation is very much a **very very rough prototype** just to get the 2 example tests working. I would appreciate any feedback on the design and implementation.
matthiaskrgr added a commit to matthiaskrgr/rust that referenced this issue Feb 8, 2024
Introduce `run-make` V2 infrastructure, a `run_make_support` library and port over 2 tests as example

See [issue rust-lang#40713: Switch run-make tests from Makefiles to rust](rust-lang#40713) for more context.

### Basic Description of `run-make` V2

`run-make` V2 aims to eliminate the dependency on `make` and `Makefile`s for building `run-make`-style tests. Makefiles are replaced by *recipes* (`rmake.rs`). The current implementation runs `run-make` V2 tests in 3 steps:

1. We build the support library `run_make_support` which the `rmake.rs` recipes depend on as a tool dylib.
2. We build the recipe `rmake.rs` and link in the support library.
3. We run the recipe to build and run the tests.

`rmake.rs` is basically a replacement for `Makefile`, and allows running arbitrary Rust code.

### Planned Changes

- [x] Get rid of the builder style patterns in `rmake_support` and instead use something like

    ```rust
    let output = rustc!(scx, "--cfg x -Cprefer-dynamic -Csymbol-mangling-version=legacy -
    Zunstable-options");
    ```
    as per Nils' suggestion. This can probably use something like `xshell`.
- [x] Make `run_make_support` into a proper crate so it can have external dependencies like `xshell`.
- [x] Instead of having an entire alternative directory `run-make-v2`, change how V2 tests are collected based on presence of `rmake.rs` recipe file. This should ease migration and prevent git history from being messed up by big moves.

### Disclaimer

The current implementation is very much a **very very rough prototype** just to get the 2 example tests working. I would appreciate any feedback on the design and implementation.
matthiaskrgr added a commit to matthiaskrgr/rust that referenced this issue Feb 9, 2024
Introduce `run-make` V2 infrastructure, a `run_make_support` library and port over 2 tests as example

See [issue rust-lang#40713: Switch run-make tests from Makefiles to rust](rust-lang#40713) for more context.

### Basic Description of `run-make` V2

`run-make` V2 aims to eliminate the dependency on `make` and `Makefile`s for building `run-make`-style tests. Makefiles are replaced by *recipes* (`rmake.rs`). The current implementation runs `run-make` V2 tests in 3 steps:

1. We build the support library `run_make_support` which the `rmake.rs` recipes depend on as a tool dylib.
2. We build the recipe `rmake.rs` and link in the support library.
3. We run the recipe to build and run the tests.

`rmake.rs` is basically a replacement for `Makefile`, and allows running arbitrary Rust code.

### Planned Changes

- [x] Get rid of the builder style patterns in `rmake_support` and instead use something like

    ```rust
    let output = rustc!(scx, "--cfg x -Cprefer-dynamic -Csymbol-mangling-version=legacy -
    Zunstable-options");
    ```
    as per Nils' suggestion. This can probably use something like `xshell`.
- [x] Make `run_make_support` into a proper crate so it can have external dependencies like `xshell`.
- [x] Instead of having an entire alternative directory `run-make-v2`, change how V2 tests are collected based on presence of `rmake.rs` recipe file. This should ease migration and prevent git history from being messed up by big moves.

### Disclaimer

The current implementation is very much a **very very rough prototype** just to get the 2 example tests working. I would appreciate any feedback on the design and implementation.
bors added a commit to rust-lang-ci/rust that referenced this issue Feb 9, 2024
Introduce `run-make` V2 infrastructure, a `run_make_support` library and port over 2 tests as example

See [issue rust-lang#40713: Switch run-make tests from Makefiles to rust](rust-lang#40713) for more context.

### Basic Description of `run-make` V2

`run-make` V2 aims to eliminate the dependency on `make` and `Makefile`s for building `run-make`-style tests. Makefiles are replaced by *recipes* (`rmake.rs`). The current implementation runs `run-make` V2 tests in 3 steps:

1. We build the support library `run_make_support` which the `rmake.rs` recipes depend on as a tool dylib.
2. We build the recipe `rmake.rs` and link in the support library.
3. We run the recipe to build and run the tests.

`rmake.rs` is basically a replacement for `Makefile`, and allows running arbitrary Rust code.

### Planned Changes

- [x] Get rid of the builder style patterns in `rmake_support` and instead use something like

    ```rust
    let output = rustc!(scx, "--cfg x -Cprefer-dynamic -Csymbol-mangling-version=legacy -
    Zunstable-options");
    ```
    as per Nils' suggestion. This can probably use something like `xshell`.
- [x] Make `run_make_support` into a proper crate so it can have external dependencies like `xshell`.
- [x] Instead of having an entire alternative directory `run-make-v2`, change how V2 tests are collected based on presence of `rmake.rs` recipe file. This should ease migration and prevent git history from being messed up by big moves.

### Disclaimer

The current implementation is very much a **very very rough prototype** just to get the 2 example tests working. I would appreciate any feedback on the design and implementation.
bors added a commit to rust-lang-ci/rust that referenced this issue Feb 10, 2024
Introduce `run-make` V2 infrastructure, a `run_make_support` library and port over 2 tests as example

See [issue rust-lang#40713: Switch run-make tests from Makefiles to rust](rust-lang#40713) for more context.

### Basic Description of `run-make` V2

`run-make` V2 aims to eliminate the dependency on `make` and `Makefile`s for building `run-make`-style tests. Makefiles are replaced by *recipes* (`rmake.rs`). The current implementation runs `run-make` V2 tests in 3 steps:

1. We build the support library `run_make_support` which the `rmake.rs` recipes depend on as a tool dylib.
2. We build the recipe `rmake.rs` and link in the support library.
3. We run the recipe to build and run the tests.

`rmake.rs` is basically a replacement for `Makefile`, and allows running arbitrary Rust code.

### Planned Changes

- [x] Get rid of the builder style patterns in `rmake_support` and instead use something like

    ```rust
    let output = rustc!(scx, "--cfg x -Cprefer-dynamic -Csymbol-mangling-version=legacy -
    Zunstable-options");
    ```
    as per Nils' suggestion. This can probably use something like `xshell`.
- [x] Make `run_make_support` into a proper crate so it can have external dependencies like `xshell`.
- [x] Instead of having an entire alternative directory `run-make-v2`, change how V2 tests are collected based on presence of `rmake.rs` recipe file. This should ease migration and prevent git history from being messed up by big moves.

### Disclaimer

The current implementation is very much a **very very rough prototype** just to get the 2 example tests working. I would appreciate any feedback on the design and implementation.
bors added a commit to rust-lang-ci/rust that referenced this issue Feb 12, 2024
Introduce `run-make` V2 infrastructure, a `run_make_support` library and port over 2 tests as example

See [issue rust-lang#40713: Switch run-make tests from Makefiles to rust](rust-lang#40713) for more context.

### Basic Description of `run-make` V2

`run-make` V2 aims to eliminate the dependency on `make` and `Makefile`s for building `run-make`-style tests. Makefiles are replaced by *recipes* (`rmake.rs`). The current implementation runs `run-make` V2 tests in 3 steps:

1. We build the support library `run_make_support` which the `rmake.rs` recipes depend on as a tool dylib.
2. We build the recipe `rmake.rs` and link in the support library.
3. We run the recipe to build and run the tests.

`rmake.rs` is basically a replacement for `Makefile`, and allows running arbitrary Rust code.

### Planned Changes

- [x] Get rid of the builder style patterns in `rmake_support` and instead use something like

    ```rust
    let output = rustc!(scx, "--cfg x -Cprefer-dynamic -Csymbol-mangling-version=legacy -
    Zunstable-options");
    ```
    as per Nils' suggestion. This can probably use something like `xshell`.
- [x] Make `run_make_support` into a proper crate so it can have external dependencies like `xshell`.
- [x] Instead of having an entire alternative directory `run-make-v2`, change how V2 tests are collected based on presence of `rmake.rs` recipe file. This should ease migration and prevent git history from being messed up by big moves.

### Disclaimer

The current implementation is very much a **very very rough prototype** just to get the 2 example tests working. I would appreciate any feedback on the design and implementation.
bors added a commit to rust-lang-ci/rust that referenced this issue Feb 12, 2024
Introduce `run-make` V2 infrastructure, a `run_make_support` library and port over 2 tests as example

See [issue rust-lang#40713: Switch run-make tests from Makefiles to rust](rust-lang#40713) for more context.

### Basic Description of `run-make` V2

`run-make` V2 aims to eliminate the dependency on `make` and `Makefile`s for building `run-make`-style tests. Makefiles are replaced by *recipes* (`rmake.rs`). The current implementation runs `run-make` V2 tests in 3 steps:

1. We build the support library `run_make_support` which the `rmake.rs` recipes depend on as a tool dylib.
2. We build the recipe `rmake.rs` and link in the support library.
3. We run the recipe to build and run the tests.

`rmake.rs` is basically a replacement for `Makefile`, and allows running arbitrary Rust code.

### Planned Changes

- [x] Get rid of the builder style patterns in `rmake_support` and instead use something like

    ```rust
    let output = rustc!(scx, "--cfg x -Cprefer-dynamic -Csymbol-mangling-version=legacy -
    Zunstable-options");
    ```
    as per Nils' suggestion. This can probably use something like `xshell`.
- [x] Make `run_make_support` into a proper crate so it can have external dependencies like `xshell`.
- [x] Instead of having an entire alternative directory `run-make-v2`, change how V2 tests are collected based on presence of `rmake.rs` recipe file. This should ease migration and prevent git history from being messed up by big moves.

### Disclaimer

The current implementation is very much a **very very rough prototype** just to get the 2 example tests working. I would appreciate any feedback on the design and implementation.
bors added a commit to rust-lang-ci/rust that referenced this issue Feb 13, 2024
Introduce `run-make` V2 infrastructure, a `run_make_support` library and port over 2 tests as example

See [issue rust-lang#40713: Switch run-make tests from Makefiles to rust](rust-lang#40713) for more context.

### Basic Description of `run-make` V2

`run-make` V2 aims to eliminate the dependency on `make` and `Makefile`s for building `run-make`-style tests. Makefiles are replaced by *recipes* (`rmake.rs`). The current implementation runs `run-make` V2 tests in 3 steps:

1. We build the support library `run_make_support` which the `rmake.rs` recipes depend on as a tool dylib.
2. We build the recipe `rmake.rs` and link in the support library.
3. We run the recipe to build and run the tests.

`rmake.rs` is basically a replacement for `Makefile`, and allows running arbitrary Rust code.

### Planned Changes

- [x] Get rid of the builder style patterns in `rmake_support` and instead use something like

    ```rust
    let output = rustc!(scx, "--cfg x -Cprefer-dynamic -Csymbol-mangling-version=legacy -
    Zunstable-options");
    ```
    as per Nils' suggestion. This can probably use something like `xshell`.
- [x] Make `run_make_support` into a proper crate so it can have external dependencies like `xshell`.
- [x] Instead of having an entire alternative directory `run-make-v2`, change how V2 tests are collected based on presence of `rmake.rs` recipe file. This should ease migration and prevent git history from being messed up by big moves.

### Disclaimer

The current implementation is very much a **very very rough prototype** just to get the 2 example tests working. I would appreciate any feedback on the design and implementation.
bors added a commit to rust-lang-ci/rust that referenced this issue Feb 13, 2024
Introduce `run-make` V2 infrastructure, a `run_make_support` library and port over 2 tests as example

See [issue rust-lang#40713: Switch run-make tests from Makefiles to rust](rust-lang#40713) for more context.

### Basic Description of `run-make` V2

`run-make` V2 aims to eliminate the dependency on `make` and `Makefile`s for building `run-make`-style tests. Makefiles are replaced by *recipes* (`rmake.rs`). The current implementation runs `run-make` V2 tests in 3 steps:

1. We build the support library `run_make_support` which the `rmake.rs` recipes depend on as a tool dylib.
2. We build the recipe `rmake.rs` and link in the support library.
3. We run the recipe to build and run the tests.

`rmake.rs` is basically a replacement for `Makefile`, and allows running arbitrary Rust code.

### Planned Changes

- [x] Get rid of the builder style patterns in `rmake_support` and instead use something like

    ```rust
    let output = rustc!(scx, "--cfg x -Cprefer-dynamic -Csymbol-mangling-version=legacy -
    Zunstable-options");
    ```
    as per Nils' suggestion. This can probably use something like `xshell`.
- [x] Make `run_make_support` into a proper crate so it can have external dependencies like `xshell`.
- [x] Instead of having an entire alternative directory `run-make-v2`, change how V2 tests are collected based on presence of `rmake.rs` recipe file. This should ease migration and prevent git history from being messed up by big moves.

### Disclaimer

The current implementation is very much a **very very rough prototype** just to get the 2 example tests working. I would appreciate any feedback on the design and implementation.
bors added a commit to rust-lang-ci/rust that referenced this issue Feb 13, 2024
Introduce `run-make` V2 infrastructure, a `run_make_support` library and port over 2 tests as example

See [issue rust-lang#40713: Switch run-make tests from Makefiles to rust](rust-lang#40713) for more context.

### Basic Description of `run-make` V2

`run-make` V2 aims to eliminate the dependency on `make` and `Makefile`s for building `run-make`-style tests. Makefiles are replaced by *recipes* (`rmake.rs`). The current implementation runs `run-make` V2 tests in 3 steps:

1. We build the support library `run_make_support` which the `rmake.rs` recipes depend on as a tool dylib.
2. We build the recipe `rmake.rs` and link in the support library.
3. We run the recipe to build and run the tests.

`rmake.rs` is basically a replacement for `Makefile`, and allows running arbitrary Rust code.

### Planned Changes

- [x] Get rid of the builder style patterns in `rmake_support` and instead use something like

    ```rust
    let output = rustc!(scx, "--cfg x -Cprefer-dynamic -Csymbol-mangling-version=legacy -
    Zunstable-options");
    ```
    as per Nils' suggestion. This can probably use something like `xshell`.
- [x] Make `run_make_support` into a proper crate so it can have external dependencies like `xshell`.
- [x] Instead of having an entire alternative directory `run-make-v2`, change how V2 tests are collected based on presence of `rmake.rs` recipe file. This should ease migration and prevent git history from being messed up by big moves.

### Disclaimer

The current implementation is very much a **very very rough prototype** just to get the 2 example tests working. I would appreciate any feedback on the design and implementation.
bors added a commit to rust-lang-ci/rust that referenced this issue Feb 14, 2024
Introduce `run-make` V2 infrastructure, a `run_make_support` library and port over 2 tests as example

See [issue rust-lang#40713: Switch run-make tests from Makefiles to rust](rust-lang#40713) for more context.

### Basic Description of `run-make` V2

`run-make` V2 aims to eliminate the dependency on `make` and `Makefile`s for building `run-make`-style tests. Makefiles are replaced by *recipes* (`rmake.rs`). The current implementation runs `run-make` V2 tests in 3 steps:

1. We build the support library `run_make_support` which the `rmake.rs` recipes depend on as a tool dylib.
2. We build the recipe `rmake.rs` and link in the support library.
3. We run the recipe to build and run the tests.

`rmake.rs` is basically a replacement for `Makefile`, and allows running arbitrary Rust code.

### Planned Changes

- [x] Get rid of the builder style patterns in `rmake_support` and instead use something like

    ```rust
    let output = rustc!(scx, "--cfg x -Cprefer-dynamic -Csymbol-mangling-version=legacy -
    Zunstable-options");
    ```
    as per Nils' suggestion. This can probably use something like `xshell`.
- [x] Make `run_make_support` into a proper crate so it can have external dependencies like `xshell`.
- [x] Instead of having an entire alternative directory `run-make-v2`, change how V2 tests are collected based on presence of `rmake.rs` recipe file. This should ease migration and prevent git history from being messed up by big moves.

### Disclaimer

The current implementation is very much a **very very rough prototype** just to get the 2 example tests working. I would appreciate any feedback on the design and implementation.
bors added a commit to rust-lang-ci/rust that referenced this issue Feb 14, 2024
Introduce `run-make` V2 infrastructure, a `run_make_support` library and port over 2 tests as example

See [issue rust-lang#40713: Switch run-make tests from Makefiles to rust](rust-lang#40713) for more context.

### Basic Description of `run-make` V2

`run-make` V2 aims to eliminate the dependency on `make` and `Makefile`s for building `run-make`-style tests. Makefiles are replaced by *recipes* (`rmake.rs`). The current implementation runs `run-make` V2 tests in 3 steps:

1. We build the support library `run_make_support` which the `rmake.rs` recipes depend on as a tool dylib.
2. We build the recipe `rmake.rs` and link in the support library.
3. We run the recipe to build and run the tests.

`rmake.rs` is basically a replacement for `Makefile`, and allows running arbitrary Rust code.

### Planned Changes

- [x] Get rid of the builder style patterns in `rmake_support` and instead use something like

    ```rust
    let output = rustc!(scx, "--cfg x -Cprefer-dynamic -Csymbol-mangling-version=legacy -
    Zunstable-options");
    ```
    as per Nils' suggestion. This can probably use something like `xshell`.
- [x] Make `run_make_support` into a proper crate so it can have external dependencies like `xshell`.
- [x] Instead of having an entire alternative directory `run-make-v2`, change how V2 tests are collected based on presence of `rmake.rs` recipe file. This should ease migration and prevent git history from being messed up by big moves.

### Disclaimer

The current implementation is very much a **very very rough prototype** just to get the 2 example tests working. I would appreciate any feedback on the design and implementation.
bors added a commit to rust-lang-ci/rust that referenced this issue Feb 21, 2024
Introduce `run-make` V2 infrastructure, a `run_make_support` library and port over 2 tests as example

See [issue rust-lang#40713: Switch run-make tests from Makefiles to rust](rust-lang#40713) for more context.

### Basic Description of `run-make` V2

`run-make` V2 aims to eliminate the dependency on `make` and `Makefile`s for building `run-make`-style tests. Makefiles are replaced by *recipes* (`rmake.rs`). The current implementation runs `run-make` V2 tests in 3 steps:

1. We build the support library `run_make_support` which the `rmake.rs` recipes depend on as a tool lib.
2. We build the recipe `rmake.rs` and link in the support library.
3. We run the recipe to build and run the tests.

`rmake.rs` is basically a replacement for `Makefile`, and allows running arbitrary Rust code. The support library is built using cargo, and so can depend on external crates if desired.

The infrastructure implemented by this PR is very barebones, and is the minimally required infrastructure needed to build, run and pass the two example `run-make` tests ported over to the new infrastructure.

### Example `run-make` V2 test

```rs
// ignore-tidy-linelength

extern crate run_make_support;

use std::path::PathBuf;

use run_make_support::{aux_build, rustc};

fn main() {
    aux_build()
        .arg("--emit=metadata")
        .arg("stable.rs")
        .run();
    let mut stable_path = PathBuf::from(env!("TMPDIR"));
    stable_path.push("libstable.rmeta");
    let output = rustc()
        .arg("--emit=metadata")
        .arg("--extern")
        .arg(&format!("stable={}", &stable_path.to_string_lossy()))
        .arg("main.rs")
        .run();

    let stderr = String::from_utf8_lossy(&output.stderr);
    let version = include_str!(concat!(env!("S"), "/src/version"));
    let expected_string = format!("stable since {}", version.trim());
    assert!(stderr.contains(&expected_string));
}
```
bors added a commit to rust-lang-ci/rust that referenced this issue Feb 28, 2024
Introduce `run-make` V2 infrastructure, a `run_make_support` library and port over 2 tests as example

## Preface

See [issue rust-lang#40713: Switch run-make tests from Makefiles to rust](rust-lang#40713) for more context.

## Basic Description of `run-make` V2

`run-make` V2 aims to eliminate the dependency on `make` and `Makefile`s for building `run-make`-style tests. Makefiles are replaced by *recipes* (`rmake.rs`). The current implementation runs `run-make` V2 tests in 3 steps:

1. We build the support library `run_make_support` which the `rmake.rs` recipes depend on as a tool lib.
2. We build the recipe `rmake.rs` and link in the support library.
3. We run the recipe to build and run the tests.

`rmake.rs` is basically a replacement for `Makefile`, and allows running arbitrary Rust code. The support library is built using cargo, and so can depend on external crates if desired.

The infrastructure implemented by this PR is very barebones, and is the minimally required infrastructure needed to build, run and pass the two example `run-make` tests ported over to the new infrastructure.

### Example `run-make` V2 test

```rs
// ignore-tidy-linelength

extern crate run_make_support;

use std::path::PathBuf;

use run_make_support::{aux_build, rustc};

fn main() {
    aux_build()
        .arg("--emit=metadata")
        .arg("stable.rs")
        .run();
    let mut stable_path = PathBuf::from(env!("TMPDIR"));
    stable_path.push("libstable.rmeta");
    let output = rustc()
        .arg("--emit=metadata")
        .arg("--extern")
        .arg(&format!("stable={}", &stable_path.to_string_lossy()))
        .arg("main.rs")
        .run();

    let stderr = String::from_utf8_lossy(&output.stderr);
    let version = include_str!(concat!(env!("S"), "/src/version"));
    let expected_string = format!("stable since {}", version.trim());
    assert!(stderr.contains(&expected_string));
}
```

## Follow Up Work

- [ ] Adjust rustc-dev-guide docs
bors added a commit to rust-lang-ci/rust that referenced this issue Mar 1, 2024
Introduce `run-make` V2 infrastructure, a `run_make_support` library and port over 2 tests as example

## Preface

See [issue rust-lang#40713: Switch run-make tests from Makefiles to rust](rust-lang#40713) for more context.

## Basic Description of `run-make` V2

`run-make` V2 aims to eliminate the dependency on `make` and `Makefile`s for building `run-make`-style tests. Makefiles are replaced by *recipes* (`rmake.rs`). The current implementation runs `run-make` V2 tests in 3 steps:

1. We build the support library `run_make_support` which the `rmake.rs` recipes depend on as a tool lib.
2. We build the recipe `rmake.rs` and link in the support library.
3. We run the recipe to build and run the tests.

`rmake.rs` is basically a replacement for `Makefile`, and allows running arbitrary Rust code. The support library is built using cargo, and so can depend on external crates if desired.

The infrastructure implemented by this PR is very barebones, and is the minimally required infrastructure needed to build, run and pass the two example `run-make` tests ported over to the new infrastructure.

### Example `run-make` V2 test

```rs
// ignore-tidy-linelength

extern crate run_make_support;

use std::path::PathBuf;

use run_make_support::{aux_build, rustc};

fn main() {
    aux_build()
        .arg("--emit=metadata")
        .arg("stable.rs")
        .run();
    let mut stable_path = PathBuf::from(env!("TMPDIR"));
    stable_path.push("libstable.rmeta");
    let output = rustc()
        .arg("--emit=metadata")
        .arg("--extern")
        .arg(&format!("stable={}", &stable_path.to_string_lossy()))
        .arg("main.rs")
        .run();

    let stderr = String::from_utf8_lossy(&output.stderr);
    let version = include_str!(concat!(env!("S"), "/src/version"));
    let expected_string = format!("stable since {}", version.trim());
    assert!(stderr.contains(&expected_string));
}
```

## Follow Up Work

- [ ] Adjust rustc-dev-guide docs
@jieyouxu
Copy link
Member

jieyouxu commented Mar 1, 2024

There's now very basic infrastructure support for writing run-make tests in Rust instead of Makefiles, now that #113026 is merged!

#121876 is the tracking issue for porting over all of the remaining run-make tests to use Rust.

github-actions bot pushed a commit to rust-lang/miri that referenced this issue Mar 2, 2024
Introduce `run-make` V2 infrastructure, a `run_make_support` library and port over 2 tests as example

## Preface

See [issue #40713: Switch run-make tests from Makefiles to rust](rust-lang/rust#40713) for more context.

## Basic Description of `run-make` V2

`run-make` V2 aims to eliminate the dependency on `make` and `Makefile`s for building `run-make`-style tests. Makefiles are replaced by *recipes* (`rmake.rs`). The current implementation runs `run-make` V2 tests in 3 steps:

1. We build the support library `run_make_support` which the `rmake.rs` recipes depend on as a tool lib.
2. We build the recipe `rmake.rs` and link in the support library.
3. We run the recipe to build and run the tests.

`rmake.rs` is basically a replacement for `Makefile`, and allows running arbitrary Rust code. The support library is built using cargo, and so can depend on external crates if desired.

The infrastructure implemented by this PR is very barebones, and is the minimally required infrastructure needed to build, run and pass the two example `run-make` tests ported over to the new infrastructure.

### Example `run-make` V2 test

```rs
// ignore-tidy-linelength

extern crate run_make_support;

use std::path::PathBuf;

use run_make_support::{aux_build, rustc};

fn main() {
    aux_build()
        .arg("--emit=metadata")
        .arg("stable.rs")
        .run();
    let mut stable_path = PathBuf::from(env!("TMPDIR"));
    stable_path.push("libstable.rmeta");
    let output = rustc()
        .arg("--emit=metadata")
        .arg("--extern")
        .arg(&format!("stable={}", &stable_path.to_string_lossy()))
        .arg("main.rs")
        .run();

    let stderr = String::from_utf8_lossy(&output.stderr);
    let version = include_str!(concat!(env!("S"), "/src/version"));
    let expected_string = format!("stable since {}", version.trim());
    assert!(stderr.contains(&expected_string));
}
```

## Follow Up Work

- [ ] Adjust rustc-dev-guide docs
@WaffleLapkin
Copy link
Member

@jieyouxu is there any reason why you are still using "make" terminology? (rmake.rs, run_make_support)

I think since this does not involve make, it would be much less confusing to have something else. Like test-recipe.rs + test_recipe_support, since you used "recipe" terminology already.

@jieyouxu
Copy link
Member

jieyouxu commented Mar 2, 2024

@jieyouxu is there any reason why you are still using "make" terminology? (rmake.rs, run_make_support)

I think since this does not involve make, it would be much less confusing to have something else. Like test-recipe.rs + test_recipe_support, since you used "recipe" terminology already.

Absolutely zero reason, it's still "make" because I didn't bother coming up with new names. Feel free to change the actual thing lol. I do note however, the entire test suite is called "run-make" so shrug.

@WaffleLapkin
Copy link
Member

@jieyouxu I don't think I have time for that rn 😅

we could also rename the test suite 👀

@jieyouxu jieyouxu added the A-compiletest Area: The compiletest test runner label May 31, 2024
@jieyouxu jieyouxu added the A-run-make Area: port run-make Makefiles to rmake.rs label Jun 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-compiletest Area: The compiletest test runner A-run-make Area: port run-make Makefiles to rmake.rs A-testsuite Area: The testsuite used to check the correctness of rustc C-enhancement Category: An issue proposing an enhancement or a PR with one. E-hard Call for participation: Hard difficulty. Experience needed to fix: A lot. E-help-wanted Call for participation: Help is requested to fix this issue. E-mentor Call for participation: This issue has a mentor. Use #t-compiler/help on Zulip for discussion. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue.
Projects
Status: In progress
Development

No branches or pull requests