test failed in CI: test_cannot_provision_instance_beyond_ram_capacity #9461
Description
This test failed on a CI run on #9456:
https://github.com/oxidecomputer/omicron/runs/56934380290
Log showing the specific test failure:
Excerpt from the log showing the failure:
7931 2025-12-02T18:58:08.973Z stdout ───
7932 2025-12-02T18:58:08.973Z
7933 2025-12-02T18:58:08.973Z running 1 test
7934 2025-12-02T18:58:08.973Z test integration_tests::instances::test_cannot_provision_instance_beyond_ram_capacity ... FAILED
7935 2025-12-02T18:58:08.973Z
7936 2025-12-02T18:58:08.973Z failures:
7937 2025-12-02T18:58:08.973Z
7938 2025-12-02T18:58:08.973Z failures:
7939 2025-12-02T18:58:08.973Z integration_tests::instances::test_cannot_provision_instance_beyond_ram_capacity
7940 2025-12-02T18:58:08.973Z
7941 2025-12-02T18:58:08.973Z test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 553 filtered out; finished in 38.49s
7942 2025-12-02T18:58:08.973Z
7943 2025-12-02T18:58:08.973Z stderr ───
7944 2025-12-02T18:58:08.973Z log file: /var/tmp/omicron_tmp/test_all-8f1857917b85dfe0-test_cannot_provision_instance_beyond_ram_capacity.27158.0.log
7945 2025-12-02T18:58:08.973Z note: configured to log to "/var/tmp/omicron_tmp/test_all-8f1857917b85dfe0-test_cannot_provision_instance_beyond_ram_capacity.27158.0.log"
7946 2025-12-02T18:58:08.973Z DB URL: postgresql://root@[::1]:65065/omicron?sslmode=disable
7947 2025-12-02T18:58:08.973Z DB address: [::1]:65065
7948 2025-12-02T18:58:08.973Z log file: /var/tmp/omicron_tmp/test_all-8f1857917b85dfe0-test_cannot_provision_instance_beyond_ram_capacity.27158.2.log
7949 2025-12-02T18:58:08.973Z note: configured to log to "/var/tmp/omicron_tmp/test_all-8f1857917b85dfe0-test_cannot_provision_instance_beyond_ram_capacity.27158.2.log"
7950 2025-12-02T18:58:08.973Z
7951 2025-12-02T18:58:08.973Z thread 'integration_tests::instances::test_cannot_provision_instance_beyond_ram_capacity' panicked at sled-agent/src/sim/collection.rs:274:65:
7952 2025-12-02T18:58:08.973Z called `Result::unwrap()` on an `Err` value: InternalError { internal_message: "Communication Error: error sending request for url (http://127.0.0.1:46986/vmms/e93daf90-0cdd-406d-8a01-53a4df737fb0)" }
7953 2025-12-02T18:58:08.973Z stack backtrace:
7954 2025-12-02T18:58:08.973Z 0: __rustc::rust_begin_unwind
7955 2025-12-02T18:58:08.973Z at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/std/src/panicking.rs:697:5
7956 2025-12-02T18:58:08.973Z 1: core::panicking::panic_fmt
7957 2025-12-02T18:58:08.973Z at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/panicking.rs:75:14
7958 2025-12-02T18:58:08.973Z 2: core::result::unwrap_failed
7959 2025-12-02T18:58:08.973Z at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/result.rs:1761:5
7960 2025-12-02T18:58:08.973Z 3: unwrap<(), omicron_common::api::external::error::Error>
7961 2025-12-02T18:58:08.973Z at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/result.rs:1167:23
7962 2025-12-02T18:58:08.973Z 4: {async_fn#0}<omicron_sled_agent::sim::instance::SimInstance>
7963 2025-12-02T18:58:08.973Z at /work/oxidecomputer/omicron/sled-agent/src/sim/collection.rs:274:65
7964 2025-12-02T18:58:08.973Z 5: {async_fn#0}
7965 2025-12-02T18:58:08.973Z at /work/oxidecomputer/omicron/sled-agent/src/sim/sled_agent.rs:553:58
7966 2025-12-02T18:58:08.973Z 6: {async_fn#0}
7967 2025-12-02T18:58:08.973Z at /work/oxidecomputer/omicron/sled-agent/src/sim/http_entrypoints.rs:934:38
7968 2025-12-02T18:58:08.985Z 7: {async_block#0}<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>, fn(dropshot::handler::RequestContext<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>>, dropshot::extractor::path::Path<sled_agent_api::VmmPathParam>) -> omicron_sled_agent::sim::http_entrypoints::{impl#1}::from::instance_poke_post::{async_fn_env#0}, omicron_sled_agent::sim::http_entrypoints::{impl#1}::from::instance_poke_post::{async_fn_env#0}, dropshot::handler::HttpResponseUpdatedNoContent, dropshot::error::HttpError, dropshot::extractor::path::Path<sled_agent_api::VmmPathParam>>
7969 2025-12-02T18:58:08.985Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.4/src/handler.rs:664:78
7970 2025-12-02T18:58:08.989Z 8: poll<alloc::boxed::Box<(dyn core::future::future::Future<Output=core::result::Result<http::response::Response<dropshot::body::Body>, dropshot::handler::HandlerError>> + core::marker::Send), alloc::alloc::Global>>
7971 2025-12-02T18:58:08.989Z at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/future/future.rs:124:9
7972 2025-12-02T18:58:08.989Z 9: {async_block#0}<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>, fn(dropshot::handler::RequestContext<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>>, dropshot::extractor::path::Path<sled_agent_api::VmmPathParam>) -> omicron_sled_agent::sim::http_entrypoints::{impl#1}::from::instance_poke_post::{async_fn_env#0}, (dropshot::extractor::path::Path<sled_agent_api::VmmPathParam>), dropshot::handler::HttpResponseUpdatedNoContent>
7973 2025-12-02T18:58:08.989Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.4/src/handler.rs:793:16
7974 2025-12-02T18:58:08.989Z 10: poll<alloc::boxed::Box<(dyn core::future::future::Future<Output=core::result::Result<http::response::Response<dropshot::body::Body>, dropshot::handler::HandlerError>> + core::marker::Send), alloc::alloc::Global>>
7975 2025-12-02T18:58:08.989Z at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/future/future.rs:124:9
7976 2025-12-02T18:58:08.989Z 11: {async_block#0}<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>>
7977 2025-12-02T18:58:08.989Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.4/src/server.rs:937:69
7978 2025-12-02T18:58:08.989Z 12: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
7979 2025-12-02T18:58:08.989Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/task/core.rs:365:24
7980 2025-12-02T18:58:08.989Z 13: with_mut<tokio::runtime::task::core::Stage<dropshot::server::http_request_handle::{async_fn#0}::{async_block_env#0}<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>>>, core::task::poll::Poll<()>, tokio::runtime::task::core::{impl#6}::poll::{closure_env#0}<dropshot::server::http_request_handle::{async_fn#0}::{async_block_env#0}<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>>, alloc::sync::Arc<tokio::runtime::scheduler::current_thread::Handle, alloc::alloc::Global>>>
7981 2025-12-02T18:58:08.989Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/loom/std/unsafe_cell.rs:16:9
7982 2025-12-02T18:58:08.989Z 14: poll<dropshot::server::http_request_handle::{async_fn#0}::{async_block_env#0}<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>>, alloc::sync::Arc<tokio::runtime::scheduler::current_thread::Handle, alloc::alloc::Global>>
7983 2025-12-02T18:58:08.990Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/task/core.rs:354:30
7984 2025-12-02T18:58:08.990Z 15: tokio::runtime::task::harness::poll_future::{{closure}}
7985 2025-12-02T18:58:08.990Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/task/harness.rs:535:30
7986 2025-12-02T18:58:08.990Z 16: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
7987 2025-12-02T18:58:08.990Z at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/panic/unwind_safe.rs:272:9
7988 2025-12-02T18:58:08.990Z 17: do_call<core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<dropshot::server::http_request_handle::{async_fn#0}::{async_block_env#0}<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>>, alloc::sync::Arc<tokio::runtime::scheduler::current_thread::Handle, alloc::alloc::Global>>>, core::task::poll::Poll<()>>
7989 2025-12-02T18:58:08.990Z at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/std/src/panicking.rs:589:40
7990 2025-12-02T18:58:08.990Z 18: __rust_try
7991 2025-12-02T18:58:08.990Z 19: catch_unwind<core::task::poll::Poll<()>, core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<dropshot::server::http_request_handle::{async_fn#0}::{async_block_env#0}<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>>, alloc::sync::Arc<tokio::runtime::scheduler::current_thread::Handle, alloc::alloc::Global>>>>
7992 2025-12-02T18:58:08.990Z at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/std/src/panicking.rs:552:19
7993 2025-12-02T18:58:08.991Z 20: catch_unwind<core::panic::unwind_safe::AssertUnwindSafe<tokio::runtime::task::harness::poll_future::{closure_env#0}<dropshot::server::http_request_handle::{async_fn#0}::{async_block_env#0}<alloc::sync::Arc<omicron_sled_agent::sim::sled_agent::SledAgent, alloc::alloc::Global>>, alloc::sync::Arc<tokio::runtime::scheduler::current_thread::Handle, alloc::alloc::Global>>>, core::task::poll::Poll<()>>
7994 2025-12-02T18:58:08.991Z at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/std/src/panic.rs:359:14
7995 2025-12-02T18:58:08.991Z 21: tokio::runtime::task::harness::poll_future
7996 2025-12-02T18:58:08.991Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/task/harness.rs:523:18
7997 2025-12-02T18:58:08.992Z 22: tokio::runtime::task::harness::Harness<T,S>::poll_inner
7998 2025-12-02T18:58:08.992Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/task/harness.rs:210:27
7999 2025-12-02T18:58:08.992Z 23: tokio::runtime::task::harness::Harness<T,S>::poll
8000 2025-12-02T18:58:08.992Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/task/harness.rs:155:20
8001 2025-12-02T18:58:08.992Z 24: tokio::runtime::task::raw::poll
8002 2025-12-02T18:58:08.992Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/task/raw.rs:325:13
8003 2025-12-02T18:58:08.992Z 25: tokio::runtime::task::raw::RawTask::poll
8004 2025-12-02T18:58:08.992Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/task/raw.rs:255:18
8005 2025-12-02T18:58:08.992Z 26: tokio::runtime::task::LocalNotified<S>::run
8006 2025-12-02T18:58:08.992Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/task/mod.rs:509:13
8007 2025-12-02T18:58:08.992Z 27: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}
8008 2025-12-02T18:58:08.992Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/scheduler/current_thread/mod.rs:789:30
8009 2025-12-02T18:58:08.992Z 28: with_budget<(), tokio::runtime::scheduler::current_thread::{impl#8}::block_on::{closure#0}::{closure_env#1}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>>
8010 2025-12-02T18:58:08.993Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/task/coop/mod.rs:167:5
8011 2025-12-02T18:58:08.993Z 29: budget<(), tokio::runtime::scheduler::current_thread::{impl#8}::block_on::{closure#0}::{closure_env#1}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>>
8012 2025-12-02T18:58:08.993Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/task/coop/mod.rs:133:5
8013 2025-12-02T18:58:08.993Z 30: {closure#0}<(), tokio::runtime::scheduler::current_thread::{impl#8}::block_on::{closure#0}::{closure_env#1}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>>
8014 2025-12-02T18:58:08.995Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/scheduler/current_thread/mod.rs:368:43
8015 2025-12-02T18:58:08.995Z 31: tokio::runtime::scheduler::current_thread::Context::enter
8016 2025-12-02T18:58:08.995Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/scheduler/current_thread/mod.rs:432:19
8017 2025-12-02T18:58:08.995Z 32: tokio::runtime::scheduler::current_thread::Context::run_task
8018 2025-12-02T18:58:08.996Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/scheduler/current_thread/mod.rs:368:28
8019 2025-12-02T18:58:08.996Z 33: {closure#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
8020 2025-12-02T18:58:08.996Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/scheduler/current_thread/mod.rs:785:43
8021 2025-12-02T18:58:08.996Z 34: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
8022 2025-12-02T18:58:08.997Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/scheduler/current_thread/mod.rs:829:68
8023 2025-12-02T18:58:08.997Z 35: tokio::runtime::context::scoped::Scoped<T>::set
8024 2025-12-02T18:58:08.997Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/context/scoped.rs:40:9
8025 2025-12-02T18:58:08.997Z 36: tokio::runtime::context::set_scheduler::{{closure}}
8026 2025-12-02T18:58:08.997Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/context.rs:176:38
8027 2025-12-02T18:58:08.997Z 37: try_with<tokio::runtime::context::Context, tokio::runtime::context::set_scheduler::{closure_env#0}<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>), tokio::runtime::scheduler::current_thread::{impl#8}::enter::{closure_env#0}<tokio::runtime::scheduler::current_thread::{impl#8}::block_on::{closure_env#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>, core::option::Option<()>>>, (alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>)>
8028 2025-12-02T18:58:08.997Z at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/std/src/thread/local.rs:315:12
8029 2025-12-02T18:58:08.998Z 38: std::thread::local::LocalKey<T>::with
8030 2025-12-02T18:58:08.998Z at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/std/src/thread/local.rs:279:20
8031 2025-12-02T18:58:08.998Z 39: tokio::runtime::context::set_scheduler
8032 2025-12-02T18:58:08.998Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/context.rs:176:17
8033 2025-12-02T18:58:08.998Z 40: tokio::runtime::scheduler::current_thread::CoreGuard::enter
8034 2025-12-02T18:58:08.998Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/scheduler/current_thread/mod.rs:829:27
8035 2025-12-02T18:58:08.999Z 41: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
8036 2025-12-02T18:58:08.999Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/scheduler/current_thread/mod.rs:729:24
8037 2025-12-02T18:58:08.999Z 42: {closure#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
8038 2025-12-02T18:58:08.999Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/scheduler/current_thread/mod.rs:200:33
8039 2025-12-02T18:58:08.999Z 43: tokio::runtime::context::runtime::enter_runtime
8040 2025-12-02T18:58:08.999Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/context/runtime.rs:65:16
8041 2025-12-02T18:58:08.999Z 44: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
8042 2025-12-02T18:58:08.999Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/scheduler/current_thread/mod.rs:188:9
8043 2025-12-02T18:58:08.999Z 45: tokio::runtime::runtime::Runtime::block_on_inner
8044 2025-12-02T18:58:08.999Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/runtime.rs:368:52
8045 2025-12-02T18:58:08.999Z 46: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
8046 2025-12-02T18:58:08.999Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/runtime.rs:342:18
8047 2025-12-02T18:58:08.999Z 47: test_cannot_provision_instance_beyond_ram_capacity
8048 2025-12-02T18:58:08.999Z at ./tests/integration_tests/instances.rs:6534:1
8049 2025-12-02T18:58:08.999Z 48: test_all::integration_tests::instances::test_cannot_provision_instance_beyond_ram_capacity::{{closure}}
8050 2025-12-02T18:58:08.999Z at ./tests/integration_tests/instances.rs:6537:2
8051 2025-12-02T18:58:08.999Z 49: core::ops::function::FnOnce::call_once
8052 2025-12-02T18:58:08.999Z at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/ops/function.rs:250:5
8053 2025-12-02T18:58:08.999Z 50: core::ops::function::FnOnce::call_once
8054 2025-12-02T18:58:08.999Z at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/ops/function.rs:250:5
8055 2025-12-02T18:58:08.999Z note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
8056 2025-12-02T18:58:08.999Z
8057 2025-12-02T18:58:08.999Z thread 'integration_tests::instances::test_cannot_provision_instance_beyond_ram_capacity' panicked at clients/sled-agent-client/src/lib.rs:382:14:
8058 2025-12-02T18:58:09.000Z instance_finish_transition() failed unexpectedly: reqwest::Error { kind: Request, url: "http://[::1]:39322/vmms/e93daf90-0cdd-406d-8a01-53a4df737fb0/poke", source: hyper_util::client::legacy::Error(SendRequest, hyper::Error(IncompleteMessage)) }
8059 2025-12-02T18:58:09.000Z stack backtrace:
8060 2025-12-02T18:58:09.000Z 0: __rustc::rust_begin_unwind
8061 2025-12-02T18:58:09.000Z at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/std/src/panicking.rs:697:5
8062 2025-12-02T18:58:09.000Z 1: core::panicking::panic_fmt
8063 2025-12-02T18:58:09.000Z at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/panicking.rs:75:14
8064 2025-12-02T18:58:09.000Z 2: core::result::unwrap_failed
8065 2025-12-02T18:58:09.000Z at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/result.rs:1761:5
8066 2025-12-02T18:58:09.000Z 3: core::result::Result<T,E>::expect
8067 2025-12-02T18:58:09.000Z at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/result.rs:1119:23
8068 2025-12-02T18:58:09.005Z 4: {async_block#0}
8069 2025-12-02T18:58:09.005Z at /work/oxidecomputer/omicron/clients/sled-agent-client/src/lib.rs:382:14
8070 2025-12-02T18:58:09.005Z 5: poll<alloc::boxed::Box<(dyn core::future::future::Future<Output=()> + core::marker::Send), alloc::alloc::Global>>
8071 2025-12-02T18:58:09.005Z at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/future/future.rs:124:9
8072 2025-12-02T18:58:09.005Z 6: {async_fn#0}
8073 2025-12-02T18:58:09.006Z at ./tests/integration_tests/instances.rs:8377:72
8074 2025-12-02T18:58:09.006Z 7: {async_fn#0}
8075 2025-12-02T18:58:09.006Z at ./tests/integration_tests/instances.rs:6598:44
8076 2025-12-02T18:58:09.006Z 8: {async_block#0}
8077 2025-12-02T18:58:09.006Z at ./tests/integration_tests/instances.rs:6534:1
8078 2025-12-02T18:58:09.006Z 9: poll<&mut dyn core::future::future::Future<Output=()>>
8079 2025-12-02T18:58:09.006Z at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/future/future.rs:124:9
8080 2025-12-02T18:58:09.006Z 10: poll<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
8081 2025-12-02T18:58:09.006Z at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/future/future.rs:124:9
8082 2025-12-02T18:58:09.006Z 11: {closure#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
8083 2025-12-02T18:58:09.006Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/scheduler/current_thread/mod.rs:742:70
8084 2025-12-02T18:58:09.006Z 12: with_budget<core::task::poll::Poll<()>, tokio::runtime::scheduler::current_thread::{impl#8}::block_on::{closure#0}::{closure#0}::{closure_env#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>>
8085 2025-12-02T18:58:09.006Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/task/coop/mod.rs:167:5
8086 2025-12-02T18:58:09.006Z 13: budget<core::task::poll::Poll<()>, tokio::runtime::scheduler::current_thread::{impl#8}::block_on::{closure#0}::{closure#0}::{closure_env#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>>
8087 2025-12-02T18:58:09.006Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/task/coop/mod.rs:133:5
8088 2025-12-02T18:58:09.007Z 14: {closure#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
8089 2025-12-02T18:58:09.007Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/scheduler/current_thread/mod.rs:742:25
8090 2025-12-02T18:58:09.007Z 15: tokio::runtime::scheduler::current_thread::Context::enter
8091 2025-12-02T18:58:09.007Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/scheduler/current_thread/mod.rs:432:19
8092 2025-12-02T18:58:09.007Z 16: {closure#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
8093 2025-12-02T18:58:09.007Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/scheduler/current_thread/mod.rs:741:44
8094 2025-12-02T18:58:09.007Z 17: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
8095 2025-12-02T18:58:09.007Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/scheduler/current_thread/mod.rs:829:68
8096 2025-12-02T18:58:09.007Z 18: tokio::runtime::context::scoped::Scoped<T>::set
8097 2025-12-02T18:58:09.007Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/context/scoped.rs:40:9
8098 2025-12-02T18:58:09.007Z 19: tokio::runtime::context::set_scheduler::{{closure}}
8099 2025-12-02T18:58:09.007Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/context.rs:176:38
8100 2025-12-02T18:58:09.008Z 20: try_with<tokio::runtime::context::Context, tokio::runtime::context::set_scheduler::{closure_env#0}<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>), tokio::runtime::scheduler::current_thread::{impl#8}::enter::{closure_env#0}<tokio::runtime::scheduler::current_thread::{impl#8}::block_on::{closure_env#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>, core::option::Option<()>>>, (alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>)>
8101 2025-12-02T18:58:09.008Z at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/std/src/thread/local.rs:315:12
8102 2025-12-02T18:58:09.008Z 21: std::thread::local::LocalKey<T>::with
8103 2025-12-02T18:58:09.008Z at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/std/src/thread/local.rs:279:20
8104 2025-12-02T18:58:09.008Z 22: tokio::runtime::context::set_scheduler
8105 2025-12-02T18:58:09.008Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/context.rs:176:17
8106 2025-12-02T18:58:09.008Z 23: tokio::runtime::scheduler::current_thread::CoreGuard::enter
8107 2025-12-02T18:58:09.008Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/scheduler/current_thread/mod.rs:829:27
8108 2025-12-02T18:58:09.008Z 24: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
8109 2025-12-02T18:58:09.008Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/scheduler/current_thread/mod.rs:729:24
8110 2025-12-02T18:58:09.008Z 25: {closure#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
8111 2025-12-02T18:58:09.008Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/scheduler/current_thread/mod.rs:200:33
8112 2025-12-02T18:58:09.008Z 26: tokio::runtime::context::runtime::enter_runtime
8113 2025-12-02T18:58:09.008Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/context/runtime.rs:65:16
8114 2025-12-02T18:58:09.008Z 27: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
8115 2025-12-02T18:58:09.008Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/scheduler/current_thread/mod.rs:188:9
8116 2025-12-02T18:58:09.008Z 28: tokio::runtime::runtime::Runtime::block_on_inner
8117 2025-12-02T18:58:09.008Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/runtime.rs:368:52
8118 2025-12-02T18:58:09.008Z 29: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
8119 2025-12-02T18:58:09.008Z at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.48.0/src/runtime/runtime.rs:342:18
8120 2025-12-02T18:58:09.008Z 30: test_cannot_provision_instance_beyond_ram_capacity
8121 2025-12-02T18:58:09.009Z at ./tests/integration_tests/instances.rs:6534:1
8122 2025-12-02T18:58:09.012Z 31: test_all::integration_tests::instances::test_cannot_provision_instance_beyond_ram_capacity::{{closure}}
8123 2025-12-02T18:58:09.014Z at ./tests/integration_tests/instances.rs:6537:2
8124 2025-12-02T18:58:09.014Z 32: core::ops::function::FnOnce::call_once
8125 2025-12-02T18:58:09.014Z at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/ops/function.rs:250:5
8126 2025-12-02T18:58:09.014Z 33: core::ops::function::FnOnce::call_once
8127 2025-12-02T18:58:09.014Z at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/ops/function.rs:250:5
8128 2025-12-02T18:58:09.014Z note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
8129 2025-12-02T18:58:09.014Z WARN: dropped CockroachInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
8130 2025-12-02T18:58:09.014Z WARN: temporary directory leaked: "/var/tmp/omicron_tmp/.tmpfbAZDF"
8131 2025-12-02T18:58:09.014Z If you would like to access the database for debugging, run the following:
8132 2025-12-02T18:58:09.014Z
8133 2025-12-02T18:58:09.014Z # Run the database
8134 2025-12-02T18:58:09.015Z cargo xtask db-dev run --no-populate --store-dir "/var/tmp/omicron_tmp/.tmpfbAZDF/data"
8135 2025-12-02T18:58:09.015Z # Access the database. Note the port may change if you run multiple databases.
8136 2025-12-02T18:58:09.015Z cockroach sql --host=localhost:32221 --insecure
8137 2025-12-02T18:58:09.015Z WARN: dropped ClickHouse process without cleaning it up first (there may still be a child process running (PID 27180) and a temporary directory leaked, /var/tmp/omicron_tmp/test_all-8f1857917b85dfe0-test_cannot_provision_instance_beyond_ram_capacity.27158.1-clickhouse-g34jP4)
8138 2025-12-02T18:58:09.015Z failed to clean up ClickHouse data dir:
8139 2025-12-02T18:58:09.015Z - /var/tmp/omicron_tmp/test_all-8f1857917b85dfe0-test_cannot_provision_instance_beyond_ram_capacity.27158.1-clickhouse-g34jP4: File exists (os error 17)
8140 2025-12-02T18:58:09.015Z WARN: dropped DendriteInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
8141 2025-12-02T18:58:09.015Z WARN: dendrite temporary directory leaked: /var/tmp/omicron_tmp/.tmprgFZC7
8142 2025-12-02T18:58:09.015Z WARN: dropped MgdInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
8143 2025-12-02T18:58:09.016Z WARN: mgd temporary directory leaked: /var/tmp/omicron_tmp/.tmprQ8I5i