-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
test: Auto-redact ... after last build at ...
; Migrate freshness
to Snapbox
#14161
test: Auto-redact ... after last build at ...
; Migrate freshness
to Snapbox
#14161
Conversation
4e3688f
to
3f80722
Compare
freshness
to Snapbox
// Following 3 subs redact: | ||
// "1719325877.527949100s, 61549498ns after last build at 1719325877.466399602s" | ||
// "1719503592.218193216s, 1h 1s after last build at 1719499991.982681034s" | ||
// into "[DIRTY_REASON_NEW_TIME], [DIRTY_REASON_DIFF] after last build at [DIRTY_REASON_OLD_TIME]" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@weihanglo what are your thoughts on these redactions?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we instead reduct everything in ()
that contains "after last build"?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am fine with the current redaction. Would could remove/change it when needed.
freshness
to Snapbox... after last build a ...
; Migrate freshness
to Snapbox
... after last build a ...
; Migrate freshness
to Snapbox... after last build at ...
; Migrate freshness
to Snapbox
Could you squash your "fixup" commits into their appropriate commit? |
cbf4b1b
to
56ca35f
Compare
tests/testsuite/freshness.rs
Outdated
.with_stderr_data( | ||
str![[r#" | ||
[LOCKING] 3 packages to latest compatible versions | ||
[COMPILING] bar [..] | ||
[RUNNING] `rustc --crate-name bar --edition=2015 bar/src/lib.rs [..]--crate-type lib --emit=[..]link[..] | ||
[RUNNING] `rustc --crate-name bar --edition=2015 bar/src/lib.rs [..]--crate-type lib --emit=[..]link -C panic=abort[..]-C debuginfo=2 [..] | ||
[COMPILING] somepm [..] | ||
[RUNNING] `rustc --crate-name somepm [..] | ||
[COMPILING] foo [..] | ||
[RUNNING] `rustc --crate-name foo --edition=2015 src/lib.rs [..]-C panic=abort[..] | ||
[FINISHED] [..] | ||
", | ||
[COMPILING] bar v0.0.1 ([ROOT]/foo/bar) | ||
[RUNNING] `rustc --crate-name bar [..] bar/src/lib.rs [..]--crate-type lib --emit=[..]link[..] | ||
[RUNNING] `rustc --crate-name bar [..] bar/src/lib.rs [..]--crate-type lib --emit=[..]link -C panic=abort[..]-C debuginfo=2 [..] | ||
[COMPILING] somepm v0.0.1 ([ROOT]/foo/somepm) | ||
[RUNNING] `rustc --crate-name somepm [..]` | ||
[COMPILING] foo v0.0.1 ([ROOT]/foo) | ||
[RUNNING] `rustc --crate-name foo [..]-C panic=abort[..]` | ||
[FINISHED] `dev` profile [unoptimized + debuginfo] target(s) in [ELAPSED]s | ||
|
||
"#]] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm trying to fix this reuse_panic_pm
CI failure, which passed on Tests on macOS aarch64 stable
but failed on Test on macOS x86_64 stable
with below:
---- expected: tests/testsuite/freshness.rs:1616:13
++++ actual: stderr
1 1 | [LOCKING] 3 packages to latest compatible versions
2 2 | [COMPILING] bar v0.0.1 ([ROOT]/foo/bar)
3 3 | [RUNNING] `rustc --crate-name bar [..] bar/src/lib.rs [..]--crate-type lib --emit=[..]link[..]
- 4 - [RUNNING] `rustc --crate-name bar [..] bar/src/lib.rs [..]--crate-type lib --emit=[..]link -C panic=abort[..]-C debuginfo=2 [..]
5 4 | [COMPILING] somepm v0.0.1 ([ROOT]/foo/somepm)
6 5 | [RUNNING] `rustc --crate-name somepm [..]`
7 6 | [COMPILING] foo v0.0.1 ([ROOT]/foo)
8 7 | [RUNNING] `rustc --crate-name foo [..]-C panic=abort[..]`
9 8 | [FINISHED] `dev` profile [unoptimized + debuginfo] target(s) in [ELAPSED]s
+ 9 + [RUNNING] `rustc --crate-name bar --edition=2015 bar/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no --check-cfg 'cfg(docsrs)' --check-cfg 'cfg(feature, values())' -C metadata=9229e8e86d572a0d -C extra-filename=-9229e8e86d572a0d --out-dir [ROOT]/foo/target/debug/deps -L dependency=[ROOT]/foo/target/debug/deps`
Some thought & findings so far:
Since the test logic was unchanged, and it passed on Test on macOS x86_64 stable
in next rerun (CI), I think the arch difference can be ruled out first.
Me next suspect is flakiness, which leads me to this with_stderr_unordered
caveat from its nice comment:
/// Be careful when using patterns such as `[..]`, because you may end up /// with multiple lines that might match, and this is not smart enough to /// do anything like longest-match. For example, avoid something like: /// /// ```text /// [RUNNING] `rustc [..] /// [RUNNING] `rustc --crate-name foo [..] /// ``` /// /// This will randomly fail if the other crate name is `bar`, and the /// order changes.
The original L1531-L1532 seems to already match this scenario (A line matching L1532 could also match L1531). However, I assume reuse_panic_pm
has been stable until this something introduced in my PR.
According to the comment form test:
bar is built once without panic (for proc-macro) and once with (for the normal dependency).
If the order of two bar
build is non-deterministic this could be a problem; In contrast, if the order is deterministic, that makes me wonder why using unordered
in the first place.
In fact, I removed .unordered
, reran and passed reuse_panic_pm
a few times locally, so let my try this first 👉 a453efd
I also tried to reproduce any randomness introduce by Snapbox unordered
,
but no finding:
use snapbox::prelude::*;
use snapbox::str;
use snapbox::assert_data_eq;
#[cargo_test]
fn test_snapbox_unorder_randomness() {
let actual = str![[r#"
rustc --crate-name foo --a=b --ccc=ddd
rustc --crate-name foo --a=b --panic=abort --ccc=ddd
eee fff
"#]]
.unordered();
let expected = str![[r#"
rustc --crate-name foo [..]
rustc --crate-name foo [..] --panic=abort [..]
eee fff
"#]]
.unordered();
assert_data_eq!(actual, expected); // Always pass. Not flaky.
}
P.s. For reference, the full output from my local runs are like:
[LOCKING] 3 packages to latest compatible versions
[COMPILING] bar v0.0.1 ([ROOT]/foo/bar)
[RUNNING] `rustc --crate-name bar --edition=2015 bar/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no --check-cfg 'cfg(docsrs)' --check-cfg 'cfg(feature, values())' -C metadata=3eb56625d2059feb -C extra-filename=-3eb56625d2059feb --out-dir [ROOT]/foo/target/debug/deps -L dependency=[ROOT]/foo/target/debug/deps`
[RUNNING] `rustc --crate-name bar --edition=2015 bar/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C panic=abort -C embed-bitcode=no -C debuginfo=2 -C split-debuginfo=unpacked --check-cfg 'cfg(docsrs)' --check-cfg 'cfg(feature, values())' -C metadata=e34f62847e3b5fb0 -C extra-filename=-e34f62847e3b5fb0 --out-dir [ROOT]/foo/target/debug/deps -L dependency=[ROOT]/foo/target/debug/deps`
[COMPILING] somepm v0.0.1 ([ROOT]/foo/somepm)
[RUNNING] `rustc --crate-name somepm --edition=2015 somepm/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C embed-bitcode=no --check-cfg 'cfg(docsrs)' --check-cfg 'cfg(feature, values())' -C metadata=a7ef9925b27e01b6 -C extra-filename=-a7ef9925b27e01b6 --out-dir [ROOT]/foo/target/debug/deps -L dependency=[ROOT]/foo/target/debug/deps --extern bar=[ROOT]/foo/target/debug/deps/libbar-[HASH].rlib --extern proc_macro`
[COMPILING] foo v0.0.1 ([ROOT]/foo)
[RUNNING] `rustc --crate-name foo --edition=2015 src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C panic=abort -C embed-bitcode=no -C debuginfo=2 -C split-debuginfo=unpacked --check-cfg 'cfg(docsrs)' --check-cfg 'cfg(feature, values())' -C metadata=ebe9fb0a09bbfc5f -C extra-filename=-ebe9fb0a09bbfc5f --out-dir [ROOT]/foo/target/debug/deps -L dependency=[ROOT]/foo/target/debug/deps --extern bar=[ROOT]/foo/target/debug/deps/libbar-[HASH].rmeta --extern somepm=[ROOT]/foo/target/debug/deps/libsomepm-[HASH].dylib`
[FINISHED] `dev` profile [unoptimized + debuginfo] target(s) in [ELAPSED]s
I guess some of my questions are:
- Just in case, was there any similar
reuse_panic_pm
flakiness you recall seeing prior to this PR? - Does
reuse_panic_pm
really needwith_stderr_unordered
or.unordered()
? - Anything else I should look into?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Update: a453efd CI passes reuse_panic_pm
on all stable pipeline: linux, mac x86, mac aarch64, Windows
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Update: The same error happened again after a453efd on Tests Windows x86_64 gnu nightly
, so a453efd didn't fix the problem 😞
CI
thread 'freshness::reuse_panic_pm' panicked at tests\testsuite\freshness.rs:1608:7:
---- expected: tests\testsuite\freshness.rs:1597:42
++++ actual: stderr
1 1 | [LOCKING] 3 packages to latest compatible versions
2 2 | [COMPILING] bar v0.0.1 ([ROOT]/foo/bar)
3 3 | [RUNNING] `rustc --crate-name bar [..] bar/src/lib.rs [..]--crate-type lib --emit=[..]link[..]
- 4 - [RUNNING] `rustc --crate-name bar [..] bar/src/lib.rs [..]--crate-type lib --emit=[..]link -C panic=abort[..]-C debuginfo=2 [..]
+ 4 + [RUNNING] `rustc --crate-name bar --edition=2015 bar/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no --check-cfg cfg(docsrs) --check-cfg "cfg(feature, values())" -C metadata=7b256d6ddbd506b7 -C extra-filename=-7b256d6ddbd506b7 --out-dir [ROOT]/foo/target/debug/deps -L dependency=[ROOT]/foo/target/debug/deps`
5 5 | [COMPILING] somepm v0.0.1 ([ROOT]/foo/somepm)
6 6 | [RUNNING] `rustc --crate-name somepm [..]`
7 7 | [COMPILING] foo v0.0.1 ([ROOT]/foo)
8 8 | [RUNNING] `rustc --crate-name foo [..]-C panic=abort[..]`
9 9 | [FINISHED] `dev` profile [unoptimized + debuginfo] target(s) in [ELAPSED]s
---- expected: tests/testsuite/freshness.rs:1620:13
++++ actual: stderr
1 1 | [LOCKING] 3 packages to latest compatible versions
2 2 | [COMPILING] bar v0.0.1 ([ROOT]/foo/bar)
3 3 | [RUNNING] `rustc --crate-name bar [..] bar/src/lib.rs [..]--crate-type lib --emit=[..]link[..]
- 4 - [RUNNING] `rustc --crate-name bar [..] bar/src/lib.rs [..]--crate-type lib --emit=[..]link -C panic=abort[..]-C debuginfo=2 [..]
5 4 | [COMPILING] somepm v0.0.1 ([ROOT]/foo/somepm)
6 5 | [RUNNING] `rustc --crate-name somepm [..]`
7 6 | [COMPILING] foo v0.0.1 ([ROOT]/foo)
8 7 | [RUNNING] `rustc --crate-name foo [..]-C panic=abort[..]`
9 8 | [FINISHED] `dev` profile [unoptimized + debuginfo] target(s) in [ELAPSED]s
+ 9 + [RUNNING] `rustc --crate-name bar --edition=2015 bar/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no --check-cfg 'cfg(docsrs)' --check-cfg 'cfg(feature, values())' -C metadata=9229e8e86d572a0d -C extra-filename=-9229e8e86d572a0d --out-dir [ROOT]/foo/target/debug/deps -L dependency=[ROOT]/foo/target/debug/deps`
This seems like a flaky error either already exist or somehow related to my change on reuse_panic_pm
.
Dropping a453efd and revert my change to reuse_panic_pm
for now.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Might need a deeper investigation of it. Fine with holding off for now.
afb4e9c
to
14cbe9e
Compare
c826531
to
133ea7e
Compare
133ea7e
to
b8a62da
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's not block this. Thank you!
// Following 3 subs redact: | ||
// "1719325877.527949100s, 61549498ns after last build at 1719325877.466399602s" | ||
// "1719503592.218193216s, 1h 1s after last build at 1719499991.982681034s" | ||
// into "[DIRTY_REASON_NEW_TIME], [DIRTY_REASON_DIFF] after last build at [DIRTY_REASON_OLD_TIME]" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am fine with the current redaction. Would could remove/change it when needed.
tests/testsuite/freshness.rs
Outdated
.with_stderr_data( | ||
str![[r#" | ||
[LOCKING] 3 packages to latest compatible versions | ||
[COMPILING] bar [..] | ||
[RUNNING] `rustc --crate-name bar --edition=2015 bar/src/lib.rs [..]--crate-type lib --emit=[..]link[..] | ||
[RUNNING] `rustc --crate-name bar --edition=2015 bar/src/lib.rs [..]--crate-type lib --emit=[..]link -C panic=abort[..]-C debuginfo=2 [..] | ||
[COMPILING] somepm [..] | ||
[RUNNING] `rustc --crate-name somepm [..] | ||
[COMPILING] foo [..] | ||
[RUNNING] `rustc --crate-name foo --edition=2015 src/lib.rs [..]-C panic=abort[..] | ||
[FINISHED] [..] | ||
", | ||
[COMPILING] bar v0.0.1 ([ROOT]/foo/bar) | ||
[RUNNING] `rustc --crate-name bar [..] bar/src/lib.rs [..]--crate-type lib --emit=[..]link[..] | ||
[RUNNING] `rustc --crate-name bar [..] bar/src/lib.rs [..]--crate-type lib --emit=[..]link -C panic=abort[..]-C debuginfo=2 [..] | ||
[COMPILING] somepm v0.0.1 ([ROOT]/foo/somepm) | ||
[RUNNING] `rustc --crate-name somepm [..]` | ||
[COMPILING] foo v0.0.1 ([ROOT]/foo) | ||
[RUNNING] `rustc --crate-name foo [..]-C panic=abort[..]` | ||
[FINISHED] `dev` profile [unoptimized + debuginfo] target(s) in [ELAPSED]s | ||
|
||
"#]] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Might need a deeper investigation of it. Fine with holding off for now.
@bors r+ |
☀️ Test successful - checks-actions |
Update cargo 13 commits in a515d463427b3912ec0365d106791f88c1c14e1b..f86ce56396168edf5d0e1c412ddda0b2edba7d46 2024-07-02 20:53:36 +0000 to 2024-07-05 17:52:05 +0000 - test: Migrate jobserver to snapbox (rust-lang/cargo#14191) - chore(deps): update msrv (3 versions) to v1.77 (rust-lang/cargo#14186) - test: migrate build_plan and build_script to snapbox (rust-lang/cargo#14193) - test: migrate cfg and check to snapbox (rust-lang/cargo#14185) - test: migrate install* and inheritable_workspace_fields to snapbox (rust-lang/cargo#14170) - Pass rustflags to artifacts built with implicit targets when using target-applies-to-host (rust-lang/cargo#13900) - test: Migrate network tests to snapbox (rust-lang/cargo#14187) - test: migrate some files to snapbox (rust-lang/cargo#14113) - test: Auto-redact `... after last build at ...`; Migrate `freshness` to Snapbox (rust-lang/cargo#14161) - chore: fix some typos (rust-lang/cargo#14182) - fix: improve message for inactive weak optional feature with edition2024 through unused dep collection (rust-lang/cargo#14026) - test:migrate `doc/directory/docscrape` to snapbox (rust-lang/cargo#14171) - test: Migrate git_auth to snapbox (rust-lang/cargo#14172)
Update cargo 20 commits in a515d463427b3912ec0365d106791f88c1c14e1b..154fdac39ae9629954e19e9986fd2cf2cdd8d964 2024-07-02 20:53:36 +0000 to 2024-07-07 01:28:23 +0000 - test: relax redactions for rust-lang/rust (rust-lang/cargo#14203) - use "bootstrap" instead of "rustbuild" (rust-lang/cargo#14207) - test: migrate serveral files to snapbox (rust-lang/cargo#14180) - Add rustdocflags to Unit's Debug impl (rust-lang/cargo#14201) - Allow enabling `config-include` feature in config (rust-lang/cargo#14196) - fix(test): Restore `does_not_contain` for check (rust-lang/cargo#14198) - test: migrate patch, pkgid, proc_macro and progress to snapbox (rust-lang/cargo#14181) - test: Migrate jobserver to snapbox (rust-lang/cargo#14191) - chore(deps): update msrv (3 versions) to v1.77 (rust-lang/cargo#14186) - test: migrate build_plan and build_script to snapbox (rust-lang/cargo#14193) - test: migrate cfg and check to snapbox (rust-lang/cargo#14185) - test: migrate install* and inheritable_workspace_fields to snapbox (rust-lang/cargo#14170) - Pass rustflags to artifacts built with implicit targets when using target-applies-to-host (rust-lang/cargo#13900) - test: Migrate network tests to snapbox (rust-lang/cargo#14187) - test: migrate some files to snapbox (rust-lang/cargo#14113) - test: Auto-redact `... after last build at ...`; Migrate `freshness` to Snapbox (rust-lang/cargo#14161) - chore: fix some typos (rust-lang/cargo#14182) - fix: improve message for inactive weak optional feature with edition2024 through unused dep collection (rust-lang/cargo#14026) - test:migrate `doc/directory/docscrape` to snapbox (rust-lang/cargo#14171) - test: Migrate git_auth to snapbox (rust-lang/cargo#14172)
test: Update some emaining unordered tests to snapbox ### What does this PR try to resolve? This is part of #14039 This leaves `global_cache_tracker.rs` as it requires some more thinking. As for the flakiness in `freshness.rs` that was seen in #14161, `compare.rs` would prioritize expected lines according to their length (assuming its more specific). Currently, snapbox prioritizes according to the line order. So we just need to put the proc-macro line before the other one to ensure it gets precedence. ### How should we test and review this PR? ### Additional information
test: Update some emaining unordered tests to snapbox ### What does this PR try to resolve? This is part of #14039 This leaves `global_cache_tracker.rs` as it requires some more thinking. As for the flakiness in `freshness.rs` that was seen in #14161, `compare.rs` would prioritize expected lines according to their length (assuming its more specific). Currently, snapbox prioritizes according to the line order. So we just need to put the proc-macro line before the other one to ensure it gets precedence. ### How should we test and review this PR? ### Additional information
What does this PR try to resolve?
Part of #14039.
Migrate
tests/testsuite/freshness.rs
to snapbox.How should we test and review this PR?
I followed #14039. The
#![allow(deprecated)]
was removed andtests/testsuite/freshness.rs
is still passing.Additional information
The redaction for "dirty reason" was a bit tricky
cargo/src/cargo/core/compiler/fingerprint/dirty_reason.rs
Line 131 in 7dcf764
Current implementation would redact
1719325877.527949100s, 61549498ns after last build at 1719325877.466399602s
1719503592.218193216s, 1h 1s after last build at 1719499991.982681034s
into
[DIRTY_REASON_NEW_TIME], [DIRTY_REASON_DIFF] after last build at [DIRTY_REASON_OLD_TIME]
Please let me know if that seems right.