Reduce rustdoc GUI tests flakyness #126436

GuillaumeGomez · 2024-06-13T20:33:13Z

lqd · 2024-06-13T20:48:10Z

With this great documentation, the fix looks simple to review. If you don't absolutely need notriddle's r+, you can have my r+ once the x86_64-gnu-tools builder is green.

GuillaumeGomez · 2024-06-13T20:50:58Z

This is very nice of you. 😉 Then let's go!

@bors r=lqd rollup

bors · 2024-06-13T20:51:00Z

📌 Commit 3de874a has been approved by lqd

It is now in the queue for this repository.

…akyness, r=lqd Reduce rustdoc GUI tests flakyness Fixes rust-lang#126430. r? `@notriddle`

…iaskrgr Rollup of 9 pull requests Successful merges: - rust-lang#121216 (Always emit `native-static-libs` note, even if it is empty) - rust-lang#123962 (change method resolution to constrain hidden types instead of rejecting method candidates) - rust-lang#126285 (`UniqueRc`: support allocators and `T: ?Sized`.) - rust-lang#126315 (Add pub struct with allow(dead_code) into worklist) - rust-lang#126320 (Avoid ICES after reporting errors on erroneous patterns) - rust-lang#126343 (Remove some msys2 utils) - rust-lang#126351 (std::unix::fs::link using direct linkat call for Solaris.) - rust-lang#126399 (extend the check for LLVM build) - rust-lang#126436 (Reduce rustdoc GUI tests flakyness) r? `@ghost` `@rustbot` modify labels: rollup

lqd · 2024-06-13T21:30:41Z

:/

GuillaumeGomez · 2024-06-13T21:31:46Z

That's interesting!

@bors r-

workingjubilee · 2024-06-13T21:48:03Z

should go next whenever it clears CI
@bors p=14

lqd · 2024-06-13T21:48:34Z

r=me when x86_64-gnu-tools is green.

matthiaskrgr · 2024-06-13T22:36:12Z

@bors treeclosed=13

GuillaumeGomez · 2024-06-13T22:47:28Z

I don't understand why the code used in literally all GUI doctests is failing only in this one test, especially since we didn't update the test nor the rustdoc output in quite a long time.

I added some debug information, let's see if it's helpful...

GuillaumeGomez · 2024-06-13T22:50:20Z

Until I figure out what's wrong with this test, I opened #126445 to remove it so the rust's CI is unblocked.

rust-log-analyzer · 2024-06-13T23:09:59Z

The job x86_64-gnu-llvm-17 failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)

#16 exporting to docker image format
#16 sending tarball 25.5s done
#16 DONE 28.4s
##[endgroup]
Setting extra environment values for docker:  --env ENABLE_GCC_CODEGEN=1 --env GCC_EXEC_PREFIX=/usr/lib/gcc/
[CI_JOB_NAME=x86_64-gnu-llvm-17]
---
sccache: Starting the server...
##[group]Configure the build
configure: processing command line
configure: 
configure: build.configure-args := ['--build=x86_64-unknown-linux-gnu', '--llvm-root=/usr/lib/llvm-17', '--enable-llvm-link-shared', '--set', 'rust.thin-lto-import-instr-limit=10', '--set', 'change-id=99999999', '--enable-verbose-configure', '--enable-sccache', '--disable-manage-submodules', '--enable-locked-deps', '--enable-cargo-native-static', '--set', 'rust.codegen-units-std=1', '--set', 'dist.compression-profile=balanced', '--dist-compression-formats=xz', '--disable-dist-src', '--release-channel=nightly', '--enable-debug-assertions', '--enable-overflow-checks', '--enable-llvm-assertions', '--set', 'rust.verify-llvm-ir', '--set', 'rust.codegen-backends=llvm,cranelift,gcc', '--set', 'llvm.static-libstdcpp', '--enable-new-symbol-mangling']
configure: target.x86_64-unknown-linux-gnu.llvm-config := /usr/lib/llvm-17/bin/llvm-config
configure: llvm.link-shared     := True
configure: rust.thin-lto-import-instr-limit := 10
configure: change-id            := 99999999

Remove failing GUI test to stop blocking CI until it is fixed Until rust-lang#126436 is solved, let's unblock rust CI. r? `@lqd`

fmease · 2024-06-14T02:42:54Z

@bors treeclosed-

bors · 2024-06-14T03:36:48Z

☔ The latest upstream changes (presumably #126445) made this pull request unmergeable. Please resolve the merge conflicts.

workingjubilee · 2024-06-14T03:40:58Z

no shit, really?

rust-log-analyzer · 2024-06-14T12:10:26Z

The job x86_64-gnu-tools failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 25 filtered out; finished in 0.00s

     Running tests/ui.rs (obj/build/x86_64-unknown-linux-gnu/stage1-tools/x86_64-unknown-linux-gnu/release/deps/ui-ff8e5f7317669711)
## Running ui tests in tests/pass for aarch64-unknown-linux-gnu
   Compiler: "MIRI_ENV_VAR_TEST"="0" "MIRI_TEMP"="/tmp/miri-uitest-qCqHOo" "RUST_BACKTRACE"="1" /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/bin/miri "--error-format=json" "--sysroot=/checkout/obj/build/x86_64-unknown-linux-gnu/miri-sysroot" "-Dwarnings" "-Dunused" "-Ainternal_features" "-Zui-testing" "--target" "aarch64-unknown-linux-gnu" "--out-dir" OUT_DIR
test result: ok. 339 passed; 14 ignored;

## Running ui tests in tests/pass-dep for aarch64-unknown-linux-gnu
## Running ui tests in tests/pass-dep for aarch64-unknown-linux-gnu
   Compiler: "MIRI_ENV_VAR_TEST"="0" "MIRI_TEMP"="/tmp/miri-uitest-qCqHOo" "RUST_BACKTRACE"="1" /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/bin/miri "--error-format=json" "--sysroot=/checkout/obj/build/x86_64-unknown-linux-gnu/miri-sysroot" "-Dwarnings" "-Dunused" "-Ainternal_features" "-Zui-testing" "--target" "aarch64-unknown-linux-gnu" "--out-dir" OUT_DIR
test result: ok. 27 passed; 5 ignored;

## Running ui tests in tests/panic for aarch64-unknown-linux-gnu
## Running ui tests in tests/panic for aarch64-unknown-linux-gnu
   Compiler: "MIRI_ENV_VAR_TEST"="0" "MIRI_TEMP"="/tmp/miri-uitest-qCqHOo" "RUST_BACKTRACE"="1" /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/bin/miri "--error-format=json" "--sysroot=/checkout/obj/build/x86_64-unknown-linux-gnu/miri-sysroot" "-Dwarnings" "-Dunused" "-Ainternal_features" "-Zui-testing" "--target" "aarch64-unknown-linux-gnu" "--out-dir" OUT_DIR
test result: ok.

## Running ui tests in tests/fail for aarch64-unknown-linux-gnu
## Running ui tests in tests/fail for aarch64-unknown-linux-gnu
   Compiler: "MIRI_ENV_VAR_TEST"="0" "MIRI_TEMP"="/tmp/miri-uitest-qCqHOo" "RUST_BACKTRACE"="1" /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/bin/miri "--error-format=json" "--sysroot=/checkout/obj/build/x86_64-unknown-linux-gnu/miri-sysroot" "-Dwarnings" "-Dunused" "-Ainternal_features" "-Zui-testing" "--target" "aarch64-unknown-linux-gnu" "--out-dir" OUT_DIR
test result: ok. 9 passed;

## Running ui tests in tests/fail-dep for aarch64-unknown-linux-gnu
## Running ui tests in tests/fail-dep for aarch64-unknown-linux-gnu
   Compiler: "MIRI_ENV_VAR_TEST"="0" "MIRI_TEMP"="/tmp/miri-uitest-qCqHOo" "RUST_BACKTRACE"="1" /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/bin/miri "--error-format=json" "--sysroot=/checkout/obj/build/x86_64-unknown-linux-gnu/miri-sysroot" "-Dwarnings" "-Dunused" "-Ainternal_features" "-Zui-testing" "--target" "aarch64-unknown-linux-gnu" "--out-dir" OUT_DIR
test result: ok.

## Running ui tests in tests/native-lib/pass for aarch64-unknown-linux-gnu
## Running ui tests in tests/native-lib/pass for aarch64-unknown-linux-gnu
   Compiler: "MIRI_ENV_VAR_TEST"="0" "MIRI_TEMP"="/tmp/miri-uitest-qCqHOo" "RUST_BACKTRACE"="1" /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/bin/miri "--error-format=json" "--sysroot=/checkout/obj/build/x86_64-unknown-linux-gnu/miri-sysroot" "-Dwarnings" "-Dunused" "-Ainternal_features" "-Zui-testing" "--target" "aarch64-unknown-linux-gnu" "-Zmiri-native-lib=/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-tools/miri-native-lib/libtestlib.so" "--out-dir" OUT_DIR

test result: ok. 1 ignored;


   Compiler: "MIRI_ENV_VAR_TEST"="0" "MIRI_TEMP"="/tmp/miri-uitest-qCqHOo" "RUST_BACKTRACE"="1" /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/bin/miri "--error-format=json" "--sysroot=/checkout/obj/build/x86_64-unknown-linux-gnu/miri-sysroot" "-Dwarnings" "-Dunused" "-Ainternal_features" "-Zui-testing" "--target" "aarch64-unknown-linux-gnu" "-Zmiri-native-lib=/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-tools/miri-native-lib/libtestlib.so" "--out-dir" OUT_DIR
test result: ok.

 finished in 38.232 seconds
##[endgroup]
---
.......... (120/124)
...F       (124/124)


/checkout/tests/rustdoc-gui/help-page.goml help-page... 
[DEBUG] Done!
[DEBUG] EXECUTING (line 8
    from `tests/rustdoc-gui/help-page.goml` line 24
    from `tests/rustdoc-gui/help-page.goml` line 33) "let parseClickVar = await page.$("#settings-menu");
if (parseClickVar === null) { throw '"#settings-menu" not found'; }
await parseClickVar.click();"
[DEBUG] Done!
[DEBUG] EXECUTING (line 10
    from `tests/rustdoc-gui/help-page.goml` line 24
    from `tests/rustdoc-gui/help-page.goml` line 33) "const timeLimit = page.getDefaultTimeout();
const timeAdd = 50;
let allTime = 0;
let parseWaitFor = null;
while (true) {
    parseWaitFor = await page.$("#settings");
    if (parseWaitFor !== null) {
    }
    }
    await new Promise(r => setTimeout(r, timeAdd));
    if (timeLimit === 0) {
    }
    }
    allTime += timeAdd;
    if (allTime >= timeLimit) {
        throw new Error("The following CSS selector \"#settings\" was not found");
}"
[DEBUG] Done!
[DEBUG] EXECUTING (line 12
[DEBUG] EXECUTING (line 12
    from `tests/rustdoc-gui/help-page.goml` line 24
    from `tests/rustdoc-gui/help-page.goml` line 33) "let parseClickVar = await page.$("#theme-ayu");
if (parseClickVar === null) { throw '"#theme-ayu" not found'; }
await parseClickVar.click();"
[DEBUG] Done!
[DEBUG] EXECUTING (line 14
    from `tests/rustdoc-gui/help-page.goml` line 24
    from `tests/rustdoc-gui/help-page.goml` line 33) "const timeLimit = page.getDefaultTimeout();
const timeAdd = 50;
let allTime = 0;
let parseWaitFor = null;
while (true) {
    parseWaitFor = await page.$("#theme-ayu:checked");
    if (parseWaitFor !== null) {
    }
    }
    await new Promise(r => setTimeout(r, timeAdd));
    if (timeLimit === 0) {
    }
    }
    allTime += timeAdd;
    if (allTime >= timeLimit) {
        throw new Error("The following CSS selector \"#theme-ayu:checked\" was not found");
}"
[DEBUG] Done!
[DEBUG] EXECUTING (line 16
[DEBUG] EXECUTING (line 16
    from `tests/rustdoc-gui/help-page.goml` line 24
    from `tests/rustdoc-gui/help-page.goml` line 33) "let parseClickVar = await page.$("#settings-menu");
if (parseClickVar === null) { throw '"#settings-menu" not found'; }
await parseClickVar.click();"
[DEBUG] Done!
[DEBUG] EXECUTING (line 18
    from `tests/rustdoc-gui/help-page.goml` line 24
    from `tests/rustdoc-gui/help-page.goml` line 33) "const timeLimit = page.getDefaultTimeout();
const timeAdd = 50;
let allTime = 0;
let parseWaitForLocalStorage = null;
while (true) {
    parseWaitForLocalStorage = await page.evaluate(() => {
        const errors = [];
        const parseWaitForLocalStorageDict = {
            "rustdoc-theme": "ayu"
        };
        for (const [parseWaitForLocalStorageKey, parseWaitForLocalStorageValue] of Object.entries(parseWaitForLocalStorageDict)) {
            let parseWaitForLocalStorage = window.localStorage.getItem(parseWaitForLocalStorageKey);
            if (parseWaitForLocalStorage != parseWaitForLocalStorageValue) {
                errors.push("localStorage item \"" + parseWaitForLocalStorageKey + "\" (of value \"" + parseWaitForLocalStorageValue + "\") != \"" + parseWaitForLocalStorage + "\"");
        }
        return errors;
    });
    });
    if (parseWaitForLocalStorage.length === 0) {
    }
    }
    await new Promise(r => setTimeout(r, timeAdd));
    if (timeLimit === 0) {
    }
    }
    allTime += timeAdd;
    if (allTime >= timeLimit) {
        const errs = parseWaitForLocalStorage.join(", ");
        throw new Error("The following local storage entries still don't match: [" + errs + "]");
}"
[DEBUG] Done!
[DEBUG] EXECUTING (line 25
[DEBUG] EXECUTING (line 25
    from `tests/rustdoc-gui/help-page.goml` line 33) "if (!arg.showText) {
    throw "`show-text: true` needs to be used before checking for `color` (otherwise the browser doesn't compute it)";
[DEBUG] Done!
[DEBUG] EXECUTING (line 25
[DEBUG] EXECUTING (line 25
    from `tests/rustdoc-gui/help-page.goml` line 33) "const { checkCssProperty } = require('command-helpers.js');

async function checkElem(elem) {
    const nonMatchingProps = [];
    const jsHandle = await elem.evaluateHandle(e => {
        const parseAssertElemCssDict = ["color","background-color","box-shadow"];
        const assertComputedStyle = window.getComputedStyle(e);
        const simple = [];
        const computed = [];
        const keys = [];

        for (const entry of parseAssertElemCssDict) {
            simple.push(e.style[entry]);
            computed.push(assertComputedStyle[entry]);
            keys.push(entry);
        }
        return [keys, simple, computed];
    });
    const [keys, simple, computed] = await jsHandle.jsonValue();
    const values = ["#c5c5c5","#314559","#5c6773 0px -1px 0px 0px inset"];

    for (const [i, key] of keys.entries()) {
        const localErr = [];
        checkCssProperty(key, values[i], simple[i], computed[i], localErr);
            nonMatchingProps.push(...localErr);
    }
    if (nonMatchingProps.length !== 0) {
        const props = nonMatchingProps.join("; ");
        throw "The following errors happened (for selector `#help kbd`): [" + props + "]";
}
}
let parseAssertElemCss = await page.$$("#help kbd");
if (parseAssertElemCss.length === 0) { throw '"#help kbd" not found'; }
for (const elem of parseAssertElemCss) {
    await checkElem(elem);
[DEBUG] Done!
[DEBUG] EXECUTING (line 23
[DEBUG] EXECUTING (line 23
    from `tests/rustdoc-gui/help-page.goml` line 39) "if (arg && arg.debug_log && arg.debug_log.setDebugEnabled) {
    arg.debug_log.setDebugEnabled(true);
} else {
    throw "`debug` command needs an object with a `debug_log` field of `Debug` type!";
[DEBUG] Done!
[DEBUG] EXECUTING (line 8
[DEBUG] EXECUTING (line 8
    from `tests/rustdoc-gui/help-page.goml` line 24
    from `tests/rustdoc-gui/help-page.goml` line 39) "let parseClickVar = await page.$("#settings-menu");
if (parseClickVar === null) { throw '"#settings-menu" not found'; }
await parseClickVar.click();"
[DEBUG] Done!
[DEBUG] EXECUTING (line 10
    from `tests/rustdoc-gui/help-page.goml` line 24
    from `tests/rustdoc-gui/help-page.goml` line 39) "const timeLimit = page.getDefaultTimeout();
const timeAdd = 50;
let allTime = 0;
let parseWaitFor = null;
while (true) {
    parseWaitFor = await page.$("#settings");
    if (parseWaitFor !== null) {
    }
    }
    await new Promise(r => setTimeout(r, timeAdd));
    if (timeLimit === 0) {
    }
    }
    allTime += timeAdd;
    if (allTime >= timeLimit) {
        throw new Error("The following CSS selector \"#settings\" was not found");
}"
[DEBUG] Done!
[DEBUG] EXECUTING (line 12
[DEBUG] EXECUTING (line 12
    from `tests/rustdoc-gui/help-page.goml` line 24
    from `tests/rustdoc-gui/help-page.goml` line 39) "let parseClickVar = await page.$("#theme-dark");
if (parseClickVar === null) { throw '"#theme-dark" not found'; }
await parseClickVar.click();"
[DEBUG] Done!
[DEBUG] EXECUTING (line 14
    from `tests/rustdoc-gui/help-page.goml` line 24
    from `tests/rustdoc-gui/help-page.goml` line 39) "const timeLimit = page.getDefaultTimeout();
const timeAdd = 50;
let allTime = 0;
let parseWaitFor = null;
while (true) {
    parseWaitFor = await page.$("#theme-dark:checked");
    if (parseWaitFor !== null) {
    }
    }
    await new Promise(r => setTimeout(r, timeAdd));
    if (timeLimit === 0) {
    }
    }
    allTime += timeAdd;
    if (allTime >= timeLimit) {
        throw new Error("The following CSS selector \"#theme-dark:checked\" was not found");
}"
[DEBUG] Done!
[DEBUG] EXECUTING (line 16
[DEBUG] EXECUTING (line 16
    from `tests/rustdoc-gui/help-page.goml` line 24
    from `tests/rustdoc-gui/help-page.goml` line 39) "let parseClickVar = await page.$("#settings-menu");
if (parseClickVar === null) { throw '"#settings-menu" not found'; }
await parseClickVar.click();"
[DEBUG] Done!
[DEBUG] EXECUTING (line 18
    from `tests/rustdoc-gui/help-page.goml` line 24
    from `tests/rustdoc-gui/help-page.goml` line 39) "const timeLimit = page.getDefaultTimeout();
const timeAdd = 50;
let allTime = 0;
let parseWaitForLocalStorage = null;
while (true) {
    parseWaitForLocalStorage = await page.evaluate(() => {
        const errors = [];
        const parseWaitForLocalStorageDict = {
            "rustdoc-theme": "dark"
        };
        for (const [parseWaitForLocalStorageKey, parseWaitForLocalStorageValue] of Object.entries(parseWaitForLocalStorageDict)) {
            let parseWaitForLocalStorage = window.localStorage.getItem(parseWaitForLocalStorageKey);
            if (parseWaitForLocalStorage != parseWaitForLocalStorageValue) {
                errors.push("localStorage item \"" + parseWaitForLocalStorageKey + "\" (of value \"" + parseWaitForLocalStorageValue + "\") != \"" + parseWaitForLocalStorage + "\"");
        }
        return errors;
    });
    });
    if (parseWaitForLocalStorage.length === 0) {
    }
    }
    await new Promise(r => setTimeout(r, timeAdd));
    if (timeLimit === 0) {
    }
    }
    allTime += timeAdd;
    if (allTime >= timeLimit) {
        const errs = parseWaitForLocalStorage.join(", ");
        throw new Error("The following local storage entries still don't match: [" + errs + "]");
}"
[DEBUG] Done!
[DEBUG] EXECUTING (line 25
[DEBUG] EXECUTING (line 25
    from `tests/rustdoc-gui/help-page.goml` line 39) "if (!arg.showText) {
    throw "`show-text: true` needs to be used before checking for `color` (otherwise the browser doesn't compute it)";
[DEBUG] Done!
[DEBUG] EXECUTING (line 25
[DEBUG] EXECUTING (line 25
    from `tests/rustdoc-gui/help-page.goml` line 39) "const { checkCssProperty } = require('command-helpers.js');

async function checkElem(elem) {
    const nonMatchingProps = [];
    const jsHandle = await elem.evaluateHandle(e => {
        const parseAssertElemCssDict = ["color","background-color","box-shadow"];
        const assertComputedStyle = window.getComputedStyle(e);
        const simple = [];
        const computed = [];
        const keys = [];

        for (const entry of parseAssertElemCssDict) {
            simple.push(e.style[entry]);
            computed.push(assertComputedStyle[entry]);
            keys.push(entry);
        }
        return [keys, simple, computed];
    });
    const [keys, simple, computed] = await jsHandle.jsonValue();
    const values = ["#000","#fafbfc","#c6cbd1 0px -1px 0px 0px inset"];

    for (const [i, key] of keys.entries()) {
        const localErr = [];
        checkCssProperty(key, values[i], simple[i], computed[i], localErr);
            nonMatchingProps.push(...localErr);
    }
    if (nonMatchingProps.length !== 0) {
        const props = nonMatchingProps.join("; ");
        throw "The following errors happened (for selector `#help kbd`): [" + props + "]";
}
}
let parseAssertElemCss = await page.$$("#help kbd");
if (parseAssertElemCss.length === 0) { throw '"#help kbd" not found'; }
for (const elem of parseAssertElemCss) {
    await checkElem(elem);
[DEBUG] Done!
[DEBUG] EXECUTING (line 23
[DEBUG] EXECUTING (line 23
    from `tests/rustdoc-gui/help-page.goml` line 45) "if (arg && arg.debug_log && arg.debug_log.setDebugEnabled) {
    arg.debug_log.setDebugEnabled(true);
} else {
    throw "`debug` command needs an object with a `debug_log` field of `Debug` type!";
[DEBUG] Done!
[DEBUG] EXECUTING (line 8
[DEBUG] EXECUTING (line 8
    from `tests/rustdoc-gui/help-page.goml` line 24
    from `tests/rustdoc-gui/help-page.goml` line 45) "let parseClickVar = await page.$("#settings-menu");
if (parseClickVar === null) { throw '"#settings-menu" not found'; }
await parseClickVar.click();"
[DEBUG] Done!
[DEBUG] EXECUTING (line 10
    from `tests/rustdoc-gui/help-page.goml` line 24
    from `tests/rustdoc-gui/help-page.goml` line 45) "const timeLimit = page.getDefaultTimeout();
const timeAdd = 50;
let allTime = 0;
let parseWaitFor = null;
while (true) {
    parseWaitFor = await page.$("#settings");
    if (parseWaitFor !== null) {
    }
    }
    await new Promise(r => setTimeout(r, timeAdd));
    if (timeLimit === 0) {
    }
    }
    allTime += timeAdd;
    if (allTime >= timeLimit) {
        throw new Error("The following CSS selector \"#settings\" was not found");
}"
[DEBUG] Done!
[DEBUG] EXECUTING (line 12
[DEBUG] EXECUTING (line 12
    from `tests/rustdoc-gui/help-page.goml` line 24
    from `tests/rustdoc-gui/help-page.goml` line 45) "let parseClickVar = await page.$("#theme-light");
if (parseClickVar === null) { throw '"#theme-light" not found'; }
await parseClickVar.click();"
[DEBUG] Done!
[DEBUG] EXECUTING (line 14
    from `tests/rustdoc-gui/help-page.goml` line 24
    from `tests/rustdoc-gui/help-page.goml` line 45) "const timeLimit = page.getDefaultTimeout();
const timeAdd = 50;
let allTime = 0;
let parseWaitFor = null;
while (true) {
    parseWaitFor = await page.$("#theme-light:checked");
    if (parseWaitFor !== null) {
    }
    }
    await new Promise(r => setTimeout(r, timeAdd));
    if (timeLimit === 0) {
    }
    }
    allTime += timeAdd;
    if (allTime >= timeLimit) {
        throw new Error("The following CSS selector \"#theme-light:checked\" was not found");
}"
[DEBUG] Done!
[DEBUG] EXECUTING (line 16
[DEBUG] EXECUTING (line 16
    from `tests/rustdoc-gui/help-page.goml` line 24
    from `tests/rustdoc-gui/help-page.goml` line 45) "let parseClickVar = await page.$("#settings-menu");
if (parseClickVar === null) { throw '"#settings-menu" not found'; }
await parseClickVar.click();"
[DEBUG] Done!
[DEBUG] EXECUTING (line 18
    from `tests/rustdoc-gui/help-page.goml` line 24
    from `tests/rustdoc-gui/help-page.goml` line 45) "const timeLimit = page.getDefaultTimeout();
const timeAdd = 50;
let allTime = 0;
let parseWaitForLocalStorage = null;
while (true) {
    parseWaitForLocalStorage = await page.evaluate(() => {
        const errors = [];
        const parseWaitForLocalStorageDict = {
            "rustdoc-theme": "light"
        };
        for (const [parseWaitForLocalStorageKey, parseWaitForLocalStorageValue] of Object.entries(parseWaitForLocalStorageDict)) {
            let parseWaitForLocalStorage = window.localStorage.getItem(parseWaitForLocalStorageKey);
            if (parseWaitForLocalStorage != parseWaitForLocalStorageValue) {
                errors.push("localStorage item \"" + parseWaitForLocalStorageKey + "\" (of value \"" + parseWaitForLocalStorageValue + "\") != \"" + parseWaitForLocalStorage + "\"");
        }
        return errors;
    });
    });
    if (parseWaitForLocalStorage.length === 0) {
    }
    }
    await new Promise(r => setTimeout(r, timeAdd));
    if (timeLimit === 0) {
    }
    }
    allTime += timeAdd;
    if (allTime >= timeLimit) {
        const errs = parseWaitForLocalStorage.join(", ");
        throw new Error("The following local storage entries still don't match: [" + errs + "]");
}"
[DEBUG] Done!
[DEBUG] EXECUTING (line 25
[DEBUG] EXECUTING (line 25
    from `tests/rustdoc-gui/help-page.goml` line 45) "if (!arg.showText) {
    throw "`show-text: true` needs to be used before checking for `color` (otherwise the browser doesn't compute it)";
[DEBUG] Done!
[DEBUG] EXECUTING (line 25
[DEBUG] EXECUTING (line 25
    from `tests/rustdoc-gui/help-page.goml` line 45) "const { checkCssProperty } = require('command-helpers.js');

async function checkElem(elem) {
    const nonMatchingProps = [];
    const jsHandle = await elem.evaluateHandle(e => {
        const parseAssertElemCssDict = ["color","background-color","box-shadow"];
        const assertComputedStyle = window.getComputedStyle(e);
        const simple = [];
        const computed = [];
        const keys = [];

        for (const entry of parseAssertElemCssDict) {
            simple.push(e.style[entry]);
            computed.push(assertComputedStyle[entry]);
            keys.push(entry);
        }
        return [keys, simple, computed];
    });
    const [keys, simple, computed] = await jsHandle.jsonValue();
    const values = ["#000","#fafbfc","#c6cbd1 0px -1px 0px 0px inset"];

    for (const [i, key] of keys.entries()) {
        const localErr = [];
        checkCssProperty(key, values[i], simple[i], computed[i], localErr);
            nonMatchingProps.push(...localErr);
    }
    if (nonMatchingProps.length !== 0) {
        const props = nonMatchingProps.join("; ");
        throw "The following errors happened (for selector `#help kbd`): [" + props + "]";
}
}
let parseAssertElemCss = await page.$$("#help kbd");
if (parseAssertElemCss.length === 0) { throw '"#help kbd" not found'; }
for (const elem of parseAssertElemCss) {
   Error: ()
 await checkElem(elem);
[DEBUG] Done!
[DEBUG] Done!
[DEBUG] EXECUTING (line 53) "const url = "file:///checkout/obj/build/x86_64-unknown-linux-gnu/test/rustdoc-gui/doc/test_docs/index.html";
try {
    await page.goto(url);
} catch(exc) {
    if (exc instanceof arg.puppeteer.ProtocolError) {
        throw "Cannot navigate to invalid URL `" + url + "`";
        throw exc;
    }
}"
[DEBUG] Done!
[DEBUG] Done!
[DEBUG] EXECUTING (line 53) "await arg.browser.overridePermissions(page.url(), arg.permissions);"
[DEBUG] Done!
[DEBUG] EXECUTING (line 54) "const viewport = page.viewport();
viewport.width = 1000;
viewport.height = 1000;
await page.setViewport(viewport);"
[DEBUG] Done!
[DEBUG] EXECUTING (line 55) "try {
if ((await page.$("#help")) === null) { throw '"#help" not found'; }
} catch(e) { return; } throw "assert didn't fail";"
[DEBUG] Done!
[DEBUG] EXECUTING (line 56) "let parseClickVar = await page.$("#help-button > a");
if (parseClickVar === null) { throw '"#help-button > a" not found'; }
await parseClickVar.click();"
[DEBUG] Done!
[DEBUG] EXECUTING (line 57) "const { checkCssProperty } = require('command-helpers.js');

async function checkElem(elem) {
    const nonMatchingProps = [];
    const jsHandle = await elem.evaluateHandle(e => {
        const parseAssertElemCssDict = ["display"];
        const assertComputedStyle = window.getComputedStyle(e);
        const simple = [];
        const computed = [];
        const keys = [];

        for (const entry of parseAssertElemCssDict) {
            simple.push(e.style[entry]);
            computed.push(assertComputedStyle[entry]);
            keys.push(entry);
        }
        return [keys, simple, computed];
    });
    const [keys, simple, computed] = await jsHandle.jsonValue();
    const values = ["block"];

    for (const [i, key] of keys.entries()) {
        const localErr = [];
        checkCssProperty(key, values[i], simple[i], computed[i], localErr);
            nonMatchingProps.push(...localErr);
    }
    if (nonMatchingProps.length !== 0) {
        const props = nonMatchingProps.join("; ");
        throw "The following errors happened (for selector `#help`): [" + props + "]";
}
}
let parseAssertElemCss = await page.$("#help");
if (parseAssertElemCss === null) { throw '"#help" not found'; }
await checkElem(parseAssertElemCss);"
[DEBUG] Done!
[DEBUG] EXECUTING (line 58) "const { checkCssProperty } = require('command-helpers.js');

async function checkElem(elem) {
    const nonMatchingProps = [];
    const jsHandle = await elem.evaluateHandle(e => {
        const parseAssertElemCssDict = ["font-size"];
        const assertComputedStyle = window.getComputedStyle(e);

GuillaumeGomez · 2024-06-14T12:25:51Z

Yep, that was completely useless. All test commands are run as expected, but the rustdoc JS seems to not be run for some reason... I'll try to see if maybe something changed on the CI side.

…=notriddle Update browser-ui-test version to `0.18.0` Should help with rust-lang#126436. r? `@notriddle`

…=notriddle Update browser-ui-test version to `0.18.0` Should help with rust-lang#126436. r? ``@notriddle``

…=notriddle Update browser-ui-test version to `0.18.0` Should help with rust-lang#126436. r? ```@notriddle```

…otriddle Update browser-ui-test version to `0.18.0` Should help with rust-lang#126436. r? `@notriddle`

rustbot assigned notriddle Jun 13, 2024

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-rustdoc Relevant to the rustdoc team, which will review and decide on the PR/issue. labels Jun 13, 2024

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jun 13, 2024

matthiaskrgr added a commit to matthiaskrgr/rust that referenced this pull request Jun 13, 2024

Rollup merge of rust-lang#126436 - GuillaumeGomez:reduce-gui-tests-fl…

6483e65

…akyness, r=lqd Reduce rustdoc GUI tests flakyness Fixes rust-lang#126430. r? `@notriddle`

matthiaskrgr mentioned this pull request Jun 13, 2024

Rollup of 9 pull requests #126438

Closed

This comment has been minimized.

Sign in to view

bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Jun 13, 2024

Reduce rustdoc GUI tests flakyness

d78746c

GuillaumeGomez force-pushed the reduce-gui-tests-flakyness branch from 3de874a to d78746c Compare June 13, 2024 21:44

matthiaskrgr mentioned this pull request Jun 13, 2024

Rollup of 10 pull requests #126439

Merged

This comment has been minimized.

Sign in to view

This was referenced Jun 13, 2024

Indicate in non_local_defs lint that the macro needs to change #125722

Merged

place explicit lifetime bound after generic param #124884

Merged

TO REMOVE: add debug in failing gui test

ba8a955

GuillaumeGomez mentioned this pull request Jun 13, 2024

Remove failing GUI test to stop blocking CI until it is fixed #126445

Merged

bors mentioned this pull request Jun 13, 2024

Rollup of 8 pull requests #126446

Closed

bors added a commit to rust-lang-ci/rust that referenced this pull request Jun 14, 2024

Auto merge of rust-lang#126445 - GuillaumeGomez:rm-help-page, r=lqd

0ef0dd2

Remove failing GUI test to stop blocking CI until it is fixed Until rust-lang#126436 is solved, let's unblock rust CI. r? `@lqd`

This was referenced Jun 14, 2024

Fill out missing Windows support information #126449

Merged

[beta] backports and stage0 bump #126453

Merged

fmease mentioned this pull request Jun 14, 2024

Tweak output of import suggestions #126371

Closed

GuillaumeGomez mentioned this pull request Jun 26, 2024

Update browser-ui-test version to 0.18.0 #127010

Merged

GuillaumeGomez added a commit to GuillaumeGomez/rust that referenced this pull request Jun 27, 2024

Rollup merge of rust-lang#127010 - GuillaumeGomez:update-puppeteer, r…

0df1058

…=notriddle Update browser-ui-test version to `0.18.0` Should help with rust-lang#126436. r? `@notriddle`

matthiaskrgr added a commit to matthiaskrgr/rust that referenced this pull request Jun 27, 2024

Rollup merge of rust-lang#127010 - GuillaumeGomez:update-puppeteer, r…

8a4c44a

…=notriddle Update browser-ui-test version to `0.18.0` Should help with rust-lang#126436. r? ``@notriddle``

jhpratt added a commit to jhpratt/rust that referenced this pull request Jun 28, 2024

Rollup merge of rust-lang#127010 - GuillaumeGomez:update-puppeteer, r…

d19478b

…=notriddle Update browser-ui-test version to `0.18.0` Should help with rust-lang#126436. r? ```@notriddle```

bors added a commit to rust-lang-ci/rust that referenced this pull request Jun 28, 2024

Auto merge of rust-lang#127010 - GuillaumeGomez:update-puppeteer, r=n…

42add88

…otriddle Update browser-ui-test version to `0.18.0` Should help with rust-lang#126436. r? `@notriddle`

GuillaumeGomez closed this Jun 28, 2024

GuillaumeGomez deleted the reduce-gui-tests-flakyness branch June 28, 2024 11:38

GuillaumeGomez mentioned this pull request Jun 28, 2024

Add back help-page.goml rustdoc GUI test #127082

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reduce rustdoc GUI tests flakyness #126436

Reduce rustdoc GUI tests flakyness #126436

GuillaumeGomez commented Jun 13, 2024

lqd commented Jun 13, 2024

GuillaumeGomez commented Jun 13, 2024

bors commented Jun 13, 2024

This comment has been minimized.

lqd commented Jun 13, 2024

GuillaumeGomez commented Jun 13, 2024

workingjubilee commented Jun 13, 2024

lqd commented Jun 13, 2024

This comment has been minimized.

matthiaskrgr commented Jun 13, 2024

GuillaumeGomez commented Jun 13, 2024 •

edited

Loading

GuillaumeGomez commented Jun 13, 2024

rust-log-analyzer commented Jun 13, 2024

fmease commented Jun 14, 2024

bors commented Jun 14, 2024

workingjubilee commented Jun 14, 2024

rust-log-analyzer commented Jun 14, 2024

GuillaumeGomez commented Jun 14, 2024

Reduce rustdoc GUI tests flakyness #126436

Reduce rustdoc GUI tests flakyness #126436

Conversation

GuillaumeGomez commented Jun 13, 2024

lqd commented Jun 13, 2024

GuillaumeGomez commented Jun 13, 2024

bors commented Jun 13, 2024

This comment has been minimized.

lqd commented Jun 13, 2024

GuillaumeGomez commented Jun 13, 2024

workingjubilee commented Jun 13, 2024

lqd commented Jun 13, 2024

This comment has been minimized.

matthiaskrgr commented Jun 13, 2024

GuillaumeGomez commented Jun 13, 2024 • edited Loading

GuillaumeGomez commented Jun 13, 2024

rust-log-analyzer commented Jun 13, 2024

fmease commented Jun 14, 2024

bors commented Jun 14, 2024

workingjubilee commented Jun 14, 2024

rust-log-analyzer commented Jun 14, 2024

GuillaumeGomez commented Jun 14, 2024

GuillaumeGomez commented Jun 13, 2024 •

edited

Loading