Skip to content

Comments

dfx.nix: test ic-http-agent without parallelism#440

Merged
mergify[bot] merged 2 commits intomasterfrom
basvandijk/test-ic-http-agent-without-parallelism
Mar 11, 2020
Merged

dfx.nix: test ic-http-agent without parallelism#440
mergify[bot] merged 2 commits intomasterfrom
basvandijk/test-ic-http-agent-without-parallelism

Conversation

@basvandijk
Copy link
Contributor

The ic-http-agent test suite has proven to be flaky when tests are run in parallel.

nix/sources.json Outdated
},
"common": {
"ref": "master",
"branch": "basvandijk/allow-overriding-cargoBuildCommands",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When do you expect this to be merged in master?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As soon as this PR succeeds.

@eftychis
Copy link
Contributor

Oh I see. Nevermind, the ping I made in the other PR @basvandijk

"^.cargo/config$"
];
cargoTestCommands = _: [
''cargo $cargo_options test $cargo_test_options --workspace --exclude ic-http-agent''
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we do test --all (that is add --all in the options?) I am not sure it is set.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that --all is a deprecated alias for --workspace which we already pass.

@eftychis
Copy link
Contributor

So this branch seems to still fail. What's the buzz? I am guessing other tests run in parallel?

@basvandijk
Copy link
Contributor Author

So this branch seems to still fail. What's the buzz? I am guessing other tests run in parallel?

Posting the error here since I'll be restarting the build to see if this could be flakiness.

Note that I'm disabling parallelism (RUST_TEST_THREADS=1) so I'm not sure why it's failing.

[2020-03-09 22:54:21] RUST_TEST_THREADS=1 cargo -Z unstable-options test $cargo_release -j "$NIX_BUILD_CORES" --all-targets -p ic-http-agent
[2020-03-09 22:54:21]    Compiling ic-http-agent v0.1.0 (/build/dfinity-sdk-rust-src/src/ic_http_agent)
[2020-03-09 22:54:25]     Finished release [optimized] target(s) in 4.39s
[2020-03-09 22:54:25]      Running target/x86_64-unknown-linux-musl/release/deps/ic_http_agent-83a1e2d2e0b8b8e5
[2020-03-09 22:54:25]
[2020-03-09 22:54:25] running 16 tests
[2020-03-09 22:54:25] test agent::agent_test::call ... ok
[2020-03-09 22:54:25] test agent::agent_test::call_error ... ok
[2020-03-09 22:54:25] test agent::agent_test::call_rejected ... FAILED
[2020-03-09 22:54:25] test agent::agent_test::install ... FAILED
[2020-03-09 22:54:25] test agent::agent_test::ping ... ok
[2020-03-09 22:54:25] test agent::agent_test::ping_error ... ok
[2020-03-09 22:54:25] test agent::agent_test::ping_okay ... ok
[2020-03-09 22:54:25] test agent::agent_test::query ... ok
[2020-03-09 22:54:25] test agent::agent_test::query_error ... ok
[2020-03-09 22:54:25] test agent::agent_test::query_rejected ... ok
[2020-03-09 22:54:25] test agent::signer::test::request_id_dummy_signer ... ok
[2020-03-09 22:54:25] test types::canister_id::tests::check_serialize_deserialize ... ok
[2020-03-09 22:54:25] test types::canister_id::tests::text_form ... ok
[2020-03-09 22:54:25] test types::canister_id::tests::text_form_matches_public_spec ... ok
[2020-03-09 22:54:25] test types::request_id::tests::public_spec_example ... ok
[2020-03-09 22:54:25] test types::request_id::tests::public_spec_example_api_client ... ok
[2020-03-09 22:54:25]
[2020-03-09 22:54:25] failures:
[2020-03-09 22:54:25]
[2020-03-09 22:54:25] ---- agent::agent_test::call_rejected stdout ----
[2020-03-09 22:54:25] thread 'main' panicked at 'internal error: entered unreachable code: Err(ReqwestError(reqwest::Error { kind: Request, url: "http://127.0.0.1:1234/api/v1/read", source: hyper::Error(IncompleteMessage) }))', src/ic_http_agent/src/agent/agent_test.rs:194:19
[2020-03-09 22:54:25] stack backtrace:
[2020-03-09 22:54:25]    0: <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt
[2020-03-09 22:54:25]    1: core::fmt::write
[2020-03-09 22:54:25]    2: std::io::Write::write_fmt
[2020-03-09 22:54:25]    3: std::io::impls::<impl std::io::Write for alloc::boxed::Box<W>>::write_fmt
[2020-03-09 22:54:25]    4: std::panicking::default_hook::{{closure}}
[2020-03-09 22:54:25]    5: std::panicking::default_hook
[2020-03-09 22:54:25]    6: std::panicking::rust_panic_with_hook
[2020-03-09 22:54:25]    7: rust_begin_unwind
[2020-03-09 22:54:25]    8: std::panicking::begin_panic_fmt
[2020-03-09 22:54:25]    9: ic_http_agent::agent::agent_test::call_rejected
[2020-03-09 22:54:25]   10: core::ops::function::FnOnce::call_once
[2020-03-09 22:54:25]   11: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once
[2020-03-09 22:54:25]   12: __rust_maybe_catch_panic
[2020-03-09 22:54:25]   13: test::run_test_in_process
[2020-03-09 22:54:25]   14: test::run_test::run_test_inner
[2020-03-09 22:54:25]   15: test::run_test
[2020-03-09 22:54:25]   16: test::run_tests
[2020-03-09 22:54:25]   17: test::console::run_tests_console
[2020-03-09 22:54:25]   18: test::test_main
[2020-03-09 22:54:25]   19: test::test_main_static
[2020-03-09 22:54:25]   20: std::rt::lang_start::{{closure}}
[2020-03-09 22:54:25]   21: std::panicking::try::do_call
[2020-03-09 22:54:25]   22: __rust_maybe_catch_panic
[2020-03-09 22:54:25]   23: std::rt::lang_start_internal
[2020-03-09 22:54:25]   24: main
[2020-03-09 22:54:25]   25: libc_start_main_stage2
[2020-03-09 22:54:25] note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
[2020-03-09 22:54:25]
[2020-03-09 22:54:25] ---- agent::agent_test::install stdout ----
[2020-03-09 22:54:25] thread 'main' panicked at 'assertion failed: `(left == right)`
[2020-03-09 22:54:25]   left: `1`,
[2020-03-09 22:54:25]  right: `0`:
[2020-03-09 22:54:25] > Expected 1 request(s) to:
[2020-03-09 22:54:25]
[2020-03-09 22:54:25] POST /api/v1/read?(any)
[2020-03-09 22:54:25]
[2020-03-09 22:54:25] ...but received 0
[2020-03-09 22:54:25]
[2020-03-09 22:54:25] ', /sources/mockito-0.20.0/src/lib.rs:928:44
[2020-03-09 22:54:25] stack backtrace:
[2020-03-09 22:54:25]    0: <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt
[2020-03-09 22:54:25]    1: core::fmt::write
[2020-03-09 22:54:25]    2: std::io::Write::write_fmt
[2020-03-09 22:54:25]    3: std::io::impls::<impl std::io::Write for alloc::boxed::Box<W>>::write_fmt
[2020-03-09 22:54:25]    4: std::panicking::default_hook::{{closure}}
[2020-03-09 22:54:25]    5: std::panicking::default_hook
[2020-03-09 22:54:25]    6: std::panicking::rust_panic_with_hook
[2020-03-09 22:54:25]    7: rust_begin_unwind
[2020-03-09 22:54:25]    8: std::panicking::begin_panic_fmt
[2020-03-09 22:54:25]    9: mockito::Mock::assert
[2020-03-09 22:54:25]   10: ic_http_agent::agent::agent_test::install
[2020-03-09 22:54:25]   11: core::ops::function::FnOnce::call_once
[2020-03-09 22:54:25]   12: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once
[2020-03-09 22:54:25]   13: __rust_maybe_catch_panic
[2020-03-09 22:54:25]   14: test::run_test_in_process
[2020-03-09 22:54:25]   15: test::run_test::run_test_inner
[2020-03-09 22:54:25]   16: test::run_test
[2020-03-09 22:54:25]   17: test::run_tests
[2020-03-09 22:54:25]   18: test::console::run_tests_console
[2020-03-09 22:54:25]   19: test::test_main
[2020-03-09 22:54:25]   20: test::test_main_static
[2020-03-09 22:54:25]   21: std::rt::lang_start::{{closure}}
[2020-03-09 22:54:25]   22: std::panicking::try::do_call
[2020-03-09 22:54:25]   23: __rust_maybe_catch_panic
[2020-03-09 22:54:25]   24: std::rt::lang_start_internal
[2020-03-09 22:54:25]   25: main
[2020-03-09 22:54:25]   26: libc_start_main_stage2
[2020-03-09 22:54:25] note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
[2020-03-09 22:54:25]
[2020-03-09 22:54:25]
[2020-03-09 22:54:25] failures:
[2020-03-09 22:54:25]     agent::agent_test::call_rejected
[2020-03-09 22:54:25]     agent::agent_test::install
[2020-03-09 22:54:25]
[2020-03-09 22:54:25] test result: FAILED. 14 passed; 2 failed; 0 ignored; 0 measured; 0 filtered out
[2020-03-09 22:54:25]
[2020-03-09 22:54:25] error: test failed, to rerun pass '-p ic-http-agent --lib'
builder for '/nix/store/23nc882gsbjbiqirvchr61l6i0f5wnxk-dfinity-sdk-rust-unknown-x86_64-unknown-linux-musl.drv' failed with exit code 1

@basvandijk basvandijk force-pushed the basvandijk/test-ic-http-agent-without-parallelism branch from bafc3e3 to d9e8161 Compare March 11, 2020 08:08
@basvandijk
Copy link
Contributor Author

Restarting the build seemed to have fixed it. So it seems the ic-http-agent test suite is flaky even when run on a single thread.

@hansl
Copy link
Contributor

hansl commented Mar 11, 2020

That worries me and should definitely not be the case. I'd like to get this in in any case as it's recommended testing behaviour by the testing framework we use (see https://docs.rs/mockito/0.7.0/mockito/#run-your-tests). I'll investigate the flakiness on master when it happens.

Let me know when this is ready for review.

The `ic-http-agent` test suite has proven to be flaky when tests are
run in parallel.
@basvandijk basvandijk force-pushed the basvandijk/test-ic-http-agent-without-parallelism branch from d9e8161 to cad2e74 Compare March 11, 2020 16:50
@basvandijk basvandijk marked this pull request as ready for review March 11, 2020 16:50
@basvandijk basvandijk requested a review from a team March 11, 2020 16:50
@mergify mergify bot merged commit be9bd14 into master Mar 11, 2020
@mergify mergify bot deleted the basvandijk/test-ic-http-agent-without-parallelism branch March 11, 2020 17:46
dfinity-bot added a commit that referenced this pull request Oct 26, 2020
## Changelog for advisory-db:
Branch: master
Commits: [rustsec/advisory-db@6e48979d...0ad26bc7](rustsec/advisory-db@6e48979...0ad26bc)

* [`5751a5f4`](rustsec/advisory-db@5751a5f) CI: bump rustsec-admin to v0.3.0-pre2 ([RustSec/advisory-db⁠#438](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/438))
* [`65441901`](rustsec/advisory-db@6544190) Add unmaintained crate advisory for stdweb ([RustSec/advisory-db⁠#439](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/439))
* [`acc73d95`](rustsec/advisory-db@acc73d9) CI: bump rustsec-admin to v0.3.0-pre3 ([RustSec/advisory-db⁠#440](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/440))
* [`691a7504`](rustsec/advisory-db@691a750) Assigned RUSTSEC-2020-0056 to stdweb ([RustSec/advisory-db⁠#441](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/441))
* [`8505c957`](rustsec/advisory-db@8505c95) Add unmaintained crate advisory for `block-cipher` ([RustSec/advisory-db⁠#442](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/442))
* [`fa616899`](rustsec/advisory-db@fa61689) Assigned RUSTSEC-2020-0057 to block-cipher ([RustSec/advisory-db⁠#443](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/443))
* [`5c9ebbfa`](rustsec/advisory-db@5c9ebbf) Add unmaintained crate advisory for `stream-cipher` ([RustSec/advisory-db⁠#444](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/444))
* [`218de91a`](rustsec/advisory-db@218de91) Assigned RUSTSEC-2020-0058 to stream-cipher ([RustSec/advisory-db⁠#445](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/445))
* [`681a2040`](rustsec/advisory-db@681a204) Add advisory for notable UB fix in libpulse-binding v2.6.0 ([RustSec/advisory-db⁠#435](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/435))
* [`1e48ac39`](rustsec/advisory-db@1e48ac3) Assigned RUSTSEC-2019-0038 to libpulse-binding ([RustSec/advisory-db⁠#446](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/446))
* [`59bee556`](rustsec/advisory-db@59bee55) Add advisory for use-after-frees fixed in libpulse-binding v1.2.1 ([RustSec/advisory-db⁠#433](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/433))
* [`146de2d8`](rustsec/advisory-db@146de2d) Assigned RUSTSEC-2018-0021 to libpulse-binding ([RustSec/advisory-db⁠#447](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/447))
* [`3b1f08f2`](rustsec/advisory-db@3b1f08f) Unyank RUSTSEC-2020-0011 ([RustSec/advisory-db⁠#448](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/448))
* [`3796cc00`](rustsec/advisory-db@3796cc0) README.md: point chat badge to Zulip ([RustSec/advisory-db⁠#449](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/449))
* [`1bf68e0d`](rustsec/advisory-db@1bf68e0) RUSTSEC-2020-0015: use wildcards in version req ([RustSec/advisory-db⁠#450](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/450))
* [`67a2144b`](rustsec/advisory-db@67a2144) RUSTSEC-2020-0015: remove wildcards ([RustSec/advisory-db⁠#451](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/451))
* [`0ad26bc7`](rustsec/advisory-db@0ad26bc) Bump `rustsec-admin` to v0.3.0 ([RustSec/advisory-db⁠#452](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/452))
mergify bot pushed a commit that referenced this pull request Oct 26, 2020
## Changelog for advisory-db:
Branch: master
Commits: [rustsec/advisory-db@6e48979d...0ad26bc7](rustsec/advisory-db@6e48979...0ad26bc)

* [`5751a5f4`](rustsec/advisory-db@5751a5f) CI: bump rustsec-admin to v0.3.0-pre2 ([RustSec/advisory-db⁠#438](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/438))
* [`65441901`](rustsec/advisory-db@6544190) Add unmaintained crate advisory for stdweb ([RustSec/advisory-db⁠#439](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/439))
* [`acc73d95`](rustsec/advisory-db@acc73d9) CI: bump rustsec-admin to v0.3.0-pre3 ([RustSec/advisory-db⁠#440](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/440))
* [`691a7504`](rustsec/advisory-db@691a750) Assigned RUSTSEC-2020-0056 to stdweb ([RustSec/advisory-db⁠#441](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/441))
* [`8505c957`](rustsec/advisory-db@8505c95) Add unmaintained crate advisory for `block-cipher` ([RustSec/advisory-db⁠#442](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/442))
* [`fa616899`](rustsec/advisory-db@fa61689) Assigned RUSTSEC-2020-0057 to block-cipher ([RustSec/advisory-db⁠#443](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/443))
* [`5c9ebbfa`](rustsec/advisory-db@5c9ebbf) Add unmaintained crate advisory for `stream-cipher` ([RustSec/advisory-db⁠#444](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/444))
* [`218de91a`](rustsec/advisory-db@218de91) Assigned RUSTSEC-2020-0058 to stream-cipher ([RustSec/advisory-db⁠#445](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/445))
* [`681a2040`](rustsec/advisory-db@681a204) Add advisory for notable UB fix in libpulse-binding v2.6.0 ([RustSec/advisory-db⁠#435](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/435))
* [`1e48ac39`](rustsec/advisory-db@1e48ac3) Assigned RUSTSEC-2019-0038 to libpulse-binding ([RustSec/advisory-db⁠#446](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/446))
* [`59bee556`](rustsec/advisory-db@59bee55) Add advisory for use-after-frees fixed in libpulse-binding v1.2.1 ([RustSec/advisory-db⁠#433](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/433))
* [`146de2d8`](rustsec/advisory-db@146de2d) Assigned RUSTSEC-2018-0021 to libpulse-binding ([RustSec/advisory-db⁠#447](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/447))
* [`3b1f08f2`](rustsec/advisory-db@3b1f08f) Unyank RUSTSEC-2020-0011 ([RustSec/advisory-db⁠#448](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/448))
* [`3796cc00`](rustsec/advisory-db@3796cc0) README.md: point chat badge to Zulip ([RustSec/advisory-db⁠#449](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/449))
* [`1bf68e0d`](rustsec/advisory-db@1bf68e0) RUSTSEC-2020-0015: use wildcards in version req ([RustSec/advisory-db⁠#450](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/450))
* [`67a2144b`](rustsec/advisory-db@67a2144) RUSTSEC-2020-0015: remove wildcards ([RustSec/advisory-db⁠#451](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/451))
* [`0ad26bc7`](rustsec/advisory-db@0ad26bc) Bump `rustsec-admin` to v0.3.0 ([RustSec/advisory-db⁠#452](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/452))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants