Add a chapter on the test harness. #82350
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
(rust-highfive has picked a reviewer for you, use r? to override) |
rust-highfive
added
the
S-waiting-on-review
This comment has been minimized.
This comment has been minimized.
camelid
reviewed
Feb 20, 2021
library/test/src/cli.rs
Outdated
| "Write logs to the specified file instead \ | ||
| of stdout", | ||
| "Write logs to the specified file", |
There was a problem hiding this comment.
I think it is incorrect. Normal output continues to go to stdout with --logfile. I discovered some of the help text was incorrect while writing the chapter.
There was a problem hiding this comment.
Normal output continues to go to stdout with
--logfile.
What is "normal output"? Do you meant that with --logfile logs still go to stdout, but they also go to the log file?
There was a problem hiding this comment.
Normal output is the output the harness normally displays, such as:
test it_works ... ok
With --logfile, it writes a specific format to the given file, which looks like:
ok it_works
I was reluctant to document the specific format, since I don't know if it is intended to be stable or relied upon. It was added without much detail in #2127, and I'm not sure anyone actually uses it (I haven't seen it used before).
|
Looks good to me. If an english-native could double-check for the spelling, it'd be great. cc @rust-lang/docs |
I’ve flagged a couple of typos but otherwise the spelling and grammar looks fine. |
jyn514
reviewed
Feb 22, 2021
Comment on lines
+33
to
+35
| * Enables building of functions annotated with the [`test`][attribute-test] | ||
| and [`bench`](#benchmarks) attributes, which will be run by the test | ||
| harness. |
There was a problem hiding this comment.
Oh huh, TIL that #[test] functions are omitted even with --cfg test if you don't pass --test.
$ cat src/main.rs
#[test]
pub fn it_works() {
}
$ cargo rustc -q -- --cfg test
$ nm /home/joshua/src/rust/test-rustdoc/hello-world/target/debug/hello-world | grep works | wc -l
0
jyn514
added
A-libtest
|
Pushed an update, thanks @GuillaumeGomez, @theotherphil, and @jyn514 for taking a look! |
jyn514
added
S-waiting-on-review
|
@bors r+ rollup |
|
📌 Commit 7d99d6d has been approved by |
bors
added
S-waiting-on-bors
|
☀️ Test successful - checks-actions |
epage
added a commit
to epage/rust
that referenced
this pull request
Dec 13, 2024
rust-lang/testing-devex-team#9 proposed changing the behavior of `--logfile`. The given reasons were: (1) Bazel can't programmatically process stdout. This seems like a limitation in Bazel and we recommend focusing on that. If we look at the wider Rust ecosystem, Rustc and Cargo don't support any such mechanism and the Cargo team rejected having one. Expecting this in libtest when its not supported elsewhere seems too specialized. (2) Tests that leak out non-programmatic output that intermixes with programmatic output. We acknowledge this is a problem to be evaluated but we need to make sure we are stepping back and gathering requirements, rather than assuming `--logfile` will fit the needs. Independent of the motive, regarding using or changing `--logfile` (1) Most ways to do it would be a breaking change, like if we respect any stable `--format`. As suggested above, we could specialize this to new `--format` values but that would be confusing for some values to apply but not others. (2) Other ways of solving this add new features to lib`test` when we are instead wanting to limit the feature set it has to minimize the compatibility surface that has to be maintained and the burden it would put on third party harnesses which are a focus area. Examples include `--format compact` or a `--log-format` flag (3) The existence of `--logfile` dates back quite a ways (rust-lang@5cc050b, rust-lang#2127) and the history gives the impression this more of slipped through rather than being an intended feature (see also rust-lang#82350 (comment)). Deprecation would better match to how it has been treated. By deprecating this, we do not expect custom test harnesses (rust-lang/testing-devex-team#2) to implement this. T-testing-devex held an FCP for deprecating in rust-lang/testing-devex-team#9 though according to [RFC rust-lang#3455](https://rust-lang.github.io/rfcs/3455-t-test.html), this is still subject to final approval from T-libs-api.
epage
added a commit
to epage/rust
that referenced
this pull request
Dec 13, 2024
rust-lang/testing-devex-team#9 proposed changing the behavior of `--logfile`. The given reasons were: (1) Bazel can't programmatically process stdout. This seems like a limitation in Bazel and we recommend focusing on that. If we look at the wider Rust ecosystem, Rustc and Cargo don't support any such mechanism and the Cargo team rejected having one. Expecting this in libtest when its not supported elsewhere seems too specialized. (2) Tests that leak out non-programmatic output that intermixes with programmatic output. We acknowledge this is a problem to be evaluated but we need to make sure we are stepping back and gathering requirements, rather than assuming `--logfile` will fit the needs. Independent of the motive, regarding using or changing `--logfile` (1) Most ways to do it would be a breaking change, like if we respect any stable `--format`. As suggested above, we could specialize this to new `--format` values but that would be confusing for some values to apply but not others. (2) Other ways of solving this add new features to lib`test` when we are instead wanting to limit the feature set it has to minimize the compatibility surface that has to be maintained and the burden it would put on third party harnesses which are a focus area. Examples include `--format compact` or a `--log-format` flag (3) The existence of `--logfile` dates back quite a ways (rust-lang@5cc050b, rust-lang#2127) and the history gives the impression this more of slipped through rather than being an intended feature (see also rust-lang#82350 (comment)). Deprecation would better match to how it has been treated. By deprecating this, we do not expect custom test harnesses (rust-lang/testing-devex-team#2) to implement this. T-testing-devex held an FCP for deprecating in rust-lang/testing-devex-team#9 though according to [RFC rust-lang#3455](https://rust-lang.github.io/rfcs/3455-t-test.html), this is still subject to final approval from T-libs-api.
epage
added a commit
to epage/rust
that referenced
this pull request
Jan 7, 2025
rust-lang/testing-devex-team#9 proposed changing the behavior of `--logfile`. The given reasons were: (1) Bazel can't programmatically process stdout. This seems like a limitation in Bazel and we recommend focusing on that. If we look at the wider Rust ecosystem, Rustc and Cargo don't support any such mechanism and the Cargo team rejected having one. Expecting this in libtest when its not supported elsewhere seems too specialized. (2) Tests that leak out non-programmatic output that intermixes with programmatic output. We acknowledge this is a problem to be evaluated but we need to make sure we are stepping back and gathering requirements, rather than assuming `--logfile` will fit the needs. Independent of the motive, regarding using or changing `--logfile` (1) Most ways to do it would be a breaking change, like if we respect any stable `--format`. As suggested above, we could specialize this to new `--format` values but that would be confusing for some values to apply but not others. (2) Other ways of solving this add new features to lib`test` when we are instead wanting to limit the feature set it has to minimize the compatibility surface that has to be maintained and the burden it would put on third party harnesses which are a focus area. Examples include `--format compact` or a `--log-format` flag (3) The existence of `--logfile` dates back quite a ways (rust-lang@5cc050b, rust-lang#2127) and the history gives the impression this more of slipped through rather than being an intended feature (see also rust-lang#82350 (comment)). Deprecation would better match to how it has been treated. By deprecating this, we do not expect custom test harnesses (rust-lang/testing-devex-team#2) to implement this. T-testing-devex held an FCP for deprecating in rust-lang/testing-devex-team#9 though according to [RFC rust-lang#3455](https://rust-lang.github.io/rfcs/3455-t-test.html), this is still subject to final approval from T-libs-api.
epage
added a commit
to epage/rust
that referenced
this pull request
Jan 7, 2025
rust-lang/testing-devex-team#9 proposed changing the behavior of `--logfile`. The given reasons were: (1) Bazel can't programmatically process stdout. This seems like a limitation in Bazel and we recommend focusing on that. If we look at the wider Rust ecosystem, Rustc and Cargo don't support any such mechanism and the Cargo team rejected having one. Expecting this in libtest when its not supported elsewhere seems too specialized. (2) Tests that leak out non-programmatic output that intermixes with programmatic output. We acknowledge this is a problem to be evaluated but we need to make sure we are stepping back and gathering requirements, rather than assuming `--logfile` will fit the needs. Independent of the motive, regarding using or changing `--logfile` (1) Most ways to do it would be a breaking change, like if we respect any stable `--format`. As suggested above, we could specialize this to new `--format` values but that would be confusing for some values to apply but not others. (2) Other ways of solving this add new features to lib`test` when we are instead wanting to limit the feature set it has to minimize the compatibility surface that has to be maintained and the burden it would put on third party harnesses which are a focus area. Examples include `--format compact` or a `--log-format` flag (3) The existence of `--logfile` dates back quite a ways (rust-lang@5cc050b, rust-lang#2127) and the history gives the impression this more of slipped through rather than being an intended feature (see also rust-lang#82350 (comment)). Deprecation would better match to how it has been treated. By deprecating this, we do not expect custom test harnesses (rust-lang/testing-devex-team#2) to implement this. T-testing-devex held an FCP for deprecating in rust-lang/testing-devex-team#9 though according to [RFC rust-lang#3455](https://rust-lang.github.io/rfcs/3455-t-test.html), this is still subject to final approval from T-libs-api.
epage
added a commit
to epage/rust
that referenced
this pull request
Jan 24, 2025
rust-lang/testing-devex-team#9 proposed changing the behavior of `--logfile`. The given reasons were: (1) Bazel can't programmatically process stdout. This seems like a limitation in Bazel and we recommend focusing on that. If we look at the wider Rust ecosystem, Rustc and Cargo don't support any such mechanism and the Cargo team rejected having one. Expecting this in libtest when its not supported elsewhere seems too specialized. (2) Tests that leak out non-programmatic output that intermixes with programmatic output. We acknowledge this is a problem to be evaluated but we need to make sure we are stepping back and gathering requirements, rather than assuming `--logfile` will fit the needs. Independent of the motive, regarding using or changing `--logfile` (1) Most ways to do it would be a breaking change, like if we respect any stable `--format`. As suggested above, we could specialize this to new `--format` values but that would be confusing for some values to apply but not others. (2) Other ways of solving this add new features to lib`test` when we are instead wanting to limit the feature set it has to minimize the compatibility surface that has to be maintained and the burden it would put on third party harnesses which are a focus area. Examples include `--format compact` or a `--log-format` flag (3) The existence of `--logfile` dates back quite a ways (rust-lang@5cc050b, rust-lang#2127) and the history gives the impression this more of slipped through rather than being an intended feature (see also rust-lang#82350 (comment)). Deprecation would better match to how it has been treated. By deprecating this, we do not expect custom test harnesses (rust-lang/testing-devex-team#2) to implement this. T-testing-devex held an FCP for deprecating in rust-lang/testing-devex-team#9 though according to [RFC rust-lang#3455](https://rust-lang.github.io/rfcs/3455-t-test.html), this is still subject to final approval from T-libs-api.
epage
added a commit
to epage/rust
that referenced
this pull request
Jan 24, 2025
rust-lang/testing-devex-team#9 proposed changing the behavior of `--logfile`. The given reasons were: (1) Bazel can't programmatically process stdout. This seems like a limitation in Bazel and we recommend focusing on that. If we look at the wider Rust ecosystem, Rustc and Cargo don't support any such mechanism and the Cargo team rejected having one. Expecting this in libtest when its not supported elsewhere seems too specialized. (2) Tests that leak out non-programmatic output that intermixes with programmatic output. We acknowledge this is a problem to be evaluated but we need to make sure we are stepping back and gathering requirements, rather than assuming `--logfile` will fit the needs. Independent of the motive, regarding using or changing `--logfile` (1) Most ways to do it would be a breaking change, like if we respect any stable `--format`. As suggested above, we could specialize this to new `--format` values but that would be confusing for some values to apply but not others. (2) Other ways of solving this add new features to lib`test` when we are instead wanting to limit the feature set it has to minimize the compatibility surface that has to be maintained and the burden it would put on third party harnesses which are a focus area. Examples include `--format compact` or a `--log-format` flag (3) The existence of `--logfile` dates back quite a ways (rust-lang@5cc050b, rust-lang#2127) and the history gives the impression this more of slipped through rather than being an intended feature (see also rust-lang#82350 (comment)). Deprecation would better match to how it has been treated. By deprecating this, we do not expect custom test harnesses (rust-lang/testing-devex-team#2) to implement this. T-testing-devex held an FCP for deprecating in rust-lang/testing-devex-team#9 though according to [RFC rust-lang#3455](https://rust-lang.github.io/rfcs/3455-t-test.html), this is still subject to final approval from T-libs-api.
epage
added a commit
to epage/rust
that referenced
this pull request
Jan 24, 2025
rust-lang/testing-devex-team#9 proposed changing the behavior of `--logfile`. The given reasons were: (1) Bazel can't programmatically process stdout. This seems like a limitation in Bazel and we recommend focusing on that. If we look at the wider Rust ecosystem, Rustc and Cargo don't support any such mechanism and the Cargo team rejected having one. Expecting this in libtest when its not supported elsewhere seems too specialized. (2) Tests that leak out non-programmatic output that intermixes with programmatic output. We acknowledge this is a problem to be evaluated but we need to make sure we are stepping back and gathering requirements, rather than assuming `--logfile` will fit the needs. Independent of the motive, regarding using or changing `--logfile` (1) Most ways to do it would be a breaking change, like if we respect any stable `--format`. As suggested above, we could specialize this to new `--format` values but that would be confusing for some values to apply but not others. (2) Other ways of solving this add new features to lib`test` when we are instead wanting to limit the feature set it has to minimize the compatibility surface that has to be maintained and the burden it would put on third party harnesses which are a focus area. Examples include `--format compact` or a `--log-format` flag (3) The existence of `--logfile` dates back quite a ways (rust-lang@5cc050b, rust-lang#2127) and the history gives the impression this more of slipped through rather than being an intended feature (see also rust-lang#82350 (comment)). Deprecation would better match to how it has been treated. By deprecating this, we do not expect custom test harnesses (rust-lang/testing-devex-team#2) to implement this. T-testing-devex held an FCP for deprecating in rust-lang/testing-devex-team#9 though according to [RFC rust-lang#3455](https://rust-lang.github.io/rfcs/3455-t-test.html), this is still subject to final approval from T-libs-api.
matthiaskrgr
added a commit
to matthiaskrgr/rust
that referenced
this pull request
Jan 25, 2025
fix(libtest): Deprecate '--logfile' rust-lang/testing-devex-team#9 proposed changing the behavior of `--logfile`. The given reasons were: (1) Bazel can't programmatically process stdout. This seems like a limitation in Bazel and we recommend focusing on that. If we look at the wider Rust ecosystem, Rustc and Cargo don't support any such mechanism and the Cargo team rejected having one. Expecting this in libtest when its not supported elsewhere seems too specialized. (2) Tests that leak out non-programmatic output that intermixes with programmatic output. We acknowledge this is a problem to be evaluated but we need to make sure we are stepping back and gathering requirements, rather than assuming `--logfile` will fit the needs. Independent of the motive, regarding using or changing `--logfile` (1) Most ways to do it would be a breaking change, like if we respect any stable `--format`. As suggested above, we could specialize this to new `--format` values but that would be confusing for some values to apply but not others. (2) Other ways of solving this add new features to lib`test` when we are instead wanting to limit the feature set it has to minimize the compatibility surface that has to be maintained and the burden it would put on third party harnesses which are a focus area. Examples include `--format compact` or a `--log-format` flag (3) The existence of `--logfile` dates back quite a ways (rust-lang@5cc050b, rust-lang#2127) and the history gives the impression this more of slipped through rather than being an intended feature (see also rust-lang#82350 (comment)). Deprecation would better match to how it has been treated. By deprecating this, we do not expect custom test harnesses (rust-lang/testing-devex-team#2) to implement this. T-testing-devex held an FCP for deprecating in rust-lang/testing-devex-team#9 though according to [RFC rust-lang#3455](https://rust-lang.github.io/rfcs/3455-t-test.html), this is still subject to final approval from T-libs-api. Closes rust-lang/testing-devex-team#9
matthiaskrgr
added a commit
to matthiaskrgr/rust
that referenced
this pull request
Jan 26, 2025
fix(libtest): Deprecate '--logfile' rust-lang/testing-devex-team#9 proposed changing the behavior of `--logfile`. The given reasons were: (1) Bazel can't programmatically process stdout. This seems like a limitation in Bazel and we recommend focusing on that. If we look at the wider Rust ecosystem, Rustc and Cargo don't support any such mechanism and the Cargo team rejected having one. Expecting this in libtest when its not supported elsewhere seems too specialized. (2) Tests that leak out non-programmatic output that intermixes with programmatic output. We acknowledge this is a problem to be evaluated but we need to make sure we are stepping back and gathering requirements, rather than assuming `--logfile` will fit the needs. Independent of the motive, regarding using or changing `--logfile` (1) Most ways to do it would be a breaking change, like if we respect any stable `--format`. As suggested above, we could specialize this to new `--format` values but that would be confusing for some values to apply but not others. (2) Other ways of solving this add new features to lib`test` when we are instead wanting to limit the feature set it has to minimize the compatibility surface that has to be maintained and the burden it would put on third party harnesses which are a focus area. Examples include `--format compact` or a `--log-format` flag (3) The existence of `--logfile` dates back quite a ways (rust-lang@5cc050b, rust-lang#2127) and the history gives the impression this more of slipped through rather than being an intended feature (see also rust-lang#82350 (comment)). Deprecation would better match to how it has been treated. By deprecating this, we do not expect custom test harnesses (rust-lang/testing-devex-team#2) to implement this. T-testing-devex held an FCP for deprecating in rust-lang/testing-devex-team#9 though according to [RFC rust-lang#3455](https://rust-lang.github.io/rfcs/3455-t-test.html), this is still subject to final approval from T-libs-api. Closes rust-lang/testing-devex-team#9
jhpratt
added a commit
to jhpratt/rust
that referenced
this pull request
Jan 26, 2025
fix(libtest): Deprecate '--logfile' rust-lang/testing-devex-team#9 proposed changing the behavior of `--logfile`. The given reasons were: (1) Bazel can't programmatically process stdout. This seems like a limitation in Bazel and we recommend focusing on that. If we look at the wider Rust ecosystem, Rustc and Cargo don't support any such mechanism and the Cargo team rejected having one. Expecting this in libtest when its not supported elsewhere seems too specialized. (2) Tests that leak out non-programmatic output that intermixes with programmatic output. We acknowledge this is a problem to be evaluated but we need to make sure we are stepping back and gathering requirements, rather than assuming `--logfile` will fit the needs. Independent of the motive, regarding using or changing `--logfile` (1) Most ways to do it would be a breaking change, like if we respect any stable `--format`. As suggested above, we could specialize this to new `--format` values but that would be confusing for some values to apply but not others. (2) Other ways of solving this add new features to lib`test` when we are instead wanting to limit the feature set it has to minimize the compatibility surface that has to be maintained and the burden it would put on third party harnesses which are a focus area. Examples include `--format compact` or a `--log-format` flag (3) The existence of `--logfile` dates back quite a ways (rust-lang@5cc050b, rust-lang#2127) and the history gives the impression this more of slipped through rather than being an intended feature (see also rust-lang#82350 (comment)). Deprecation would better match to how it has been treated. By deprecating this, we do not expect custom test harnesses (rust-lang/testing-devex-team#2) to implement this. T-testing-devex held an FCP for deprecating in rust-lang/testing-devex-team#9 though according to [RFC rust-lang#3455](https://rust-lang.github.io/rfcs/3455-t-test.html), this is still subject to final approval from T-libs-api. Closes rust-lang/testing-devex-team#9
rust-timer
added a commit
to rust-lang-ci/rust
that referenced
this pull request
Jan 26, 2025
Rollup merge of rust-lang#134283 - epage:logfile, r=Amanieu fix(libtest): Deprecate '--logfile' rust-lang/testing-devex-team#9 proposed changing the behavior of `--logfile`. The given reasons were: (1) Bazel can't programmatically process stdout. This seems like a limitation in Bazel and we recommend focusing on that. If we look at the wider Rust ecosystem, Rustc and Cargo don't support any such mechanism and the Cargo team rejected having one. Expecting this in libtest when its not supported elsewhere seems too specialized. (2) Tests that leak out non-programmatic output that intermixes with programmatic output. We acknowledge this is a problem to be evaluated but we need to make sure we are stepping back and gathering requirements, rather than assuming `--logfile` will fit the needs. Independent of the motive, regarding using or changing `--logfile` (1) Most ways to do it would be a breaking change, like if we respect any stable `--format`. As suggested above, we could specialize this to new `--format` values but that would be confusing for some values to apply but not others. (2) Other ways of solving this add new features to lib`test` when we are instead wanting to limit the feature set it has to minimize the compatibility surface that has to be maintained and the burden it would put on third party harnesses which are a focus area. Examples include `--format compact` or a `--log-format` flag (3) The existence of `--logfile` dates back quite a ways (rust-lang@5cc050b, rust-lang#2127) and the history gives the impression this more of slipped through rather than being an intended feature (see also rust-lang#82350 (comment)). Deprecation would better match to how it has been treated. By deprecating this, we do not expect custom test harnesses (rust-lang/testing-devex-team#2) to implement this. T-testing-devex held an FCP for deprecating in rust-lang/testing-devex-team#9 though according to [RFC rust-lang#3455](https://rust-lang.github.io/rfcs/3455-t-test.html), this is still subject to final approval from T-libs-api. Closes rust-lang/testing-devex-team#9
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There isn't really any online documentation on the test harness, so this adds a chapter to the rustc book which provides information on how the harness works and details on the command-line options.