[cp][aptos-release-v1.25] [forge] bump graceful_overload mem threshold temporarily #15527

github-actions · 2024-12-06T18:46:40Z

Backport

This will backport the following commits from main to aptos-release-v1.25:

[forge] bump graceful_overload mem threshold temporarily (#15511)

Questions ?

Please refer to the Backport tool documentation

(cherry picked from commit 3c6e693)

trunk-io · 2024-12-06T18:46:43Z

⏱️ 1h 9m total CI duration on this PR

Slowest 15 Jobs	Cumulative Duration	Recent Runs
rust-unit-tests	25m	🟩
rust-move-tests	13m	🟩
rust-doc-tests	6m	🟩
execution-performance / single-node-performance	6m	🟥
execution-performance / test-target-determinator	5m	🟩
test-target-determinator	5m	🟩
check	3m	🟩
rust-cargo-deny	2m	🟩
fetch-last-released-docker-image-tag	2m	🟩
check-dynamic-deps	39s	🟩
general-lints	32s	🟩
semgrep/ci	21s	🟩
file_change_determinator	10s	🟩
file_change_determinator	9s	🟩
determine-docker-build-metadata	4s	🟩

_{settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io}

github-actions · 2024-12-06T19:14:59Z

✅ Forge suite `compat` success on `082b888013b4d47744289d4f5b7d29dd0854a766` ==> `ab4b83b6af799821e78b0edc80456b6621ebd8bf`

Compatibility test results for 082b888013b4d47744289d4f5b7d29dd0854a766 ==> ab4b83b6af799821e78b0edc80456b6621ebd8bf (PR)
1. Check liveness of validators at old version: 082b888013b4d47744289d4f5b7d29dd0854a766
compatibility::simple-validator-upgrade::liveness-check : committed: 15396.55 txn/s, latency: 2225.19 ms, (p50: 1900 ms, p70: 2100, p90: 2100 ms, p99: 7500 ms), latency samples: 534580
2. Upgrading first Validator to new version: ab4b83b6af799821e78b0edc80456b6621ebd8bf
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 8037.24 txn/s, latency: 3594.93 ms, (p50: 4000 ms, p70: 4100, p90: 4200 ms, p99: 4300 ms), latency samples: 147340
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 8000.56 txn/s, latency: 4053.22 ms, (p50: 4200 ms, p70: 4300, p90: 5400 ms, p99: 6500 ms), latency samples: 272020
3. Upgrading rest of first batch to new version: ab4b83b6af799821e78b0edc80456b6621ebd8bf
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 7821.84 txn/s, latency: 3692.20 ms, (p50: 4100 ms, p70: 4300, p90: 4400 ms, p99: 4500 ms), latency samples: 145040
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 8047.76 txn/s, latency: 4041.85 ms, (p50: 4300 ms, p70: 4400, p90: 4500 ms, p99: 4600 ms), latency samples: 268220
4. upgrading second batch to new version: ab4b83b6af799821e78b0edc80456b6621ebd8bf
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 13997.01 txn/s, latency: 1928.72 ms, (p50: 2000 ms, p70: 2200, p90: 2300 ms, p99: 2400 ms), latency samples: 238480
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 14308.10 txn/s, latency: 2166.60 ms, (p50: 2200 ms, p70: 2300, p90: 2400 ms, p99: 2500 ms), latency samples: 466040
5. check swarm health
Compatibility test for 082b888013b4d47744289d4f5b7d29dd0854a766 ==> ab4b83b6af799821e78b0edc80456b6621ebd8bf passed
Test Ok

github-actions · 2024-12-06T19:15:11Z

❌ Forge suite `realistic_env_max_load` failure on `ab4b83b6af799821e78b0edc80456b6621ebd8bf`

two traffics test: inner traffic : committed: 13637.17 txn/s, submitted: 13734.69 txn/s, expired: 97.52 txn/s, latency: 2432.45 ms, (p50: 2300 ms, p70: 2400, p90: 2500 ms, p99: 3900 ms), latency samples: 5185240
two traffics test : committed: 85.95 txn/s, submitted: 99.99 txn/s, expired: 14.04 txn/s, latency: 3137.40 ms, (p50: 1300 ms, p70: 1400, p90: 1500 ms, p99: 58400 ms), latency samples: 1680
Latency breakdown for phase 0: ["MempoolToBlockCreation: max: 1.192, avg: 0.969", "ConsensusProposalToOrdered: max: 0.351, avg: 0.319", "ConsensusOrderedToCommit: max: 0.395, avg: 0.362", "ConsensusProposalToCommit: max: 0.711, avg: 0.680"]
Test Failed: check for success

Caused by:
    TPS requirement failed. Average TPS 85.95369314207484, minimum TPS requirement 95. Full stats: committed: 85.95 txn/s, submitted: 99.99 txn/s, expired: 14.04 txn/s, latency: 3137.40 ms, (p50: 1300 ms, p70: 1400, p90: 1500 ms, p99: 58400 ms), latency samples: 1680

Stack backtrace:
   0: anyhow::error::<impl anyhow::Error>::msg
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.89/src/error.rs:85:36
   1: aptos_forge::success_criteria::SuccessCriteriaChecker::check_tps
             at ./testsuite/forge/src/success_criteria.rs:456:13
   2: aptos_forge::success_criteria::SuccessCriteriaChecker::check_throughput
             at ./testsuite/forge/src/success_criteria.rs:509:9
   3: aptos_forge::success_criteria::SuccessCriteriaChecker::check_for_success::{{closure}}
             at ./testsuite/forge/src/success_criteria.rs:299:9
   4: aptos_forge::interface::network::NetworkContext::check_for_success::{{closure}}
             at ./testsuite/forge/src/interface/network.rs:112:10
   5: <dyn aptos_testcases::NetworkLoadTest as aptos_forge::interface::network::NetworkTest>::run::{{closure}}
             at ./testsuite/testcases/src/lib.rs:333:14
   6: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
   7: <aptos_testcases::two_traffics_test::TwoTrafficsTest as aptos_forge::interface::network::NetworkTest>::run::{{closure}}
             at ./testsuite/testcases/src/two_traffics_test.rs:77:47
   8: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
   9: <aptos_testcases::CompositeNetworkTest as aptos_forge::interface::network::NetworkTest>::run::{{closure}}
             at ./testsuite/testcases/src/lib.rs:639:37
  10: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
  11: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/park.rs:281:63
  12: tokio::runtime::coop::with_budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs:107:5
  13: tokio::runtime::coop::budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs:73:5
  14: tokio::runtime::park::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/park.rs:281:31
  15: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/blocking.rs:66:9
  16: tokio::runtime::handle::Handle::block_on_inner::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/handle.rs:324:22
  17: tokio::runtime::context::runtime::enter_runtime
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/runtime.rs:65:16
  18: tokio::runtime::handle::Handle::block_on_inner
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/handle.rs:323:9
  19: tokio::runtime::handle::Handle::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/handle.rs:302:18
  20: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:332:50
  21: forge::run_forge_with_changelog
             at ./testsuite/forge-cli/src/main.rs:426:24
  22: forge::main
             at ./testsuite/forge-cli/src/main.rs:329:21
  23: core::ops::function::FnOnce::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:250:5
  24: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/backtrace.rs:155:18
  25: std::rt::lang_start::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:166:18
  26: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:284:13
  27: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  28: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  29: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  30: std::rt::lang_start_internal::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:48
  31: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  32: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  33: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  34: std::rt::lang_start_internal
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:20
  35: main
  36: __libc_start_main
  37: _start
Trailing Log Lines:
  32: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  33: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  34: std::rt::lang_start_internal
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:20
  35: main
  36: __libc_start_main
  37: _start

=== BEGIN JUNIT ===
<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="forge" tests="1" failures="1" errors="0" uuid="b21a250d-2bc6-4d0a-9037-c1a38d9668c7">
    <testsuite name="local" tests="1" disabled="0" errors="0" failures="1">
        <testcase name="CompositeNetworkTest(CpuChaosWrapper(network:multi-region-network-emulation(two traffics test))) with ">
            <failure message="check for success

Caused by:
    TPS requirement failed. Average TPS 85.95369314207484, minimum TPS requirement 95. Full stats: committed: 85.95 txn/s, submitted: 99.99 txn/s, expired: 14.04 txn/s, latency: 3137.40 ms, (p50: 1300 ms, p70: 1400, p90: 1500 ms, p99: 58400 ms), latency samples: 1680

Stack backtrace:
   0: anyhow::error::&lt;impl anyhow::Error&gt;::msg
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.89/src/error.rs:85:36
   1: aptos_forge::success_criteria::SuccessCriteriaChecker::check_tps
             at ./testsuite/forge/src/success_criteria.rs:456:13
   2: aptos_forge::success_criteria::SuccessCriteriaChecker::check_throughput
             at ./testsuite/forge/src/success_criteria.rs:509:9
   3: aptos_forge::success_criteria::SuccessCriteriaChecker::check_for_success::{{closure}}
             at ./testsuite/forge/src/success_criteria.rs:299:9
   4: aptos_forge::interface::network::NetworkContext::check_for_success::{{closure}}
             at ./testsuite/forge/src/interface/network.rs:112:10
   5: &lt;dyn aptos_testcases::NetworkLoadTest as aptos_forge::interface::network::NetworkTest&gt;::run::{{closure}}
             at ./testsuite/testcases/src/lib.rs:333:14
   6: &lt;core::pin::Pin&lt;P&gt; as core::future::future::Future&gt;::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
   7: &lt;aptos_testcases::two_traffics_test::TwoTrafficsTest as aptos_forge::interface::network::NetworkTest&gt;::run::{{closure}}
             at ./testsuite/testcases/src/two_traffics_test.rs:77:47
   8: &lt;core::pin::Pin&lt;P&gt; as core::future::future::Future&gt;::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
   9: &lt;aptos_testcases::CompositeNetworkTest as aptos_forge::interface::network::NetworkTest&gt;::run::{{closure}}
             at ./testsuite/testcases/src/lib.rs:639:37
  10: &lt;core::pin::Pin&lt;P&gt; as core::future::future::Future&gt;::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
  11: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/park.rs:281:63
  12: tokio::runtime::coop::with_budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs:107:5
  13: tokio::runtime::coop::budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs:73:5
  14: tokio::runtime::park::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/park.rs:281:31
  15: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/blocking.rs:66:9
  16: tokio::runtime::handle::Handle::block_on_inner::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/handle.rs:324:22
  17: tokio::runtime::context::runtime::enter_runtime
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/runtime.rs:65:16
  18: tokio::runtime::handle::Handle::block_on_inner
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/handle.rs:323:9
  19: tokio::runtime::handle::Handle::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/handle.rs:302:18
  20: aptos_forge::runner::Forge&lt;F&gt;::run
             at ./testsuite/forge/src/runner.rs:332:50
  21: forge::run_forge_with_changelog
             at ./testsuite/forge-cli/src/main.rs:426:24
  22: forge::main
             at ./testsuite/forge-cli/src/main.rs:329:21
  23: core::ops::function::FnOnce::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:250:5
  24: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/backtrace.rs:155:18
  25: std::rt::lang_start::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:166:18
  26: core::ops::function::impls::&lt;impl core::ops::function::FnOnce&lt;A&gt; for &amp;F&gt;::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:284:13
  27: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  28: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  29: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  30: std::rt::lang_start_internal::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:48
  31: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  32: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  33: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  34: std::rt::lang_start_internal
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:20
  35: main
  36: __libc_start_main
  37: _start"/>
        </testcase>
    </testsuite>
</testsuites>
=== END JUNIT ===

Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:381"},"thread_name":"main","hostname":"forge-e2e-pr-15527-1733511741-ab4b83b6af799821e78b0edc80456b662","timestamp":"2024-12-06T19:15:08.133244Z","message":"Deleting namespace forge-e2e-pr-15527: Some(NamespaceStatus { conditions: None, phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:398"},"thread_name":"main","hostname":"forge-e2e-pr-15527-1733511741-ab4b83b6af799821e78b0edc80456b662","timestamp":"2024-12-06T19:15:08.133263Z","message":"aptos-node resources for Forge removed in namespace: forge-e2e-pr-15527"}
Failed to run tests:
Tests Failed

failures:
    CompositeNetworkTest

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Error: Tests Failed

Stack backtrace:
   0: anyhow::error::<impl anyhow::Error>::msg
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.89/src/error.rs:85:36
   1: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:358:13
   2: forge::run_forge_with_changelog
             at ./testsuite/forge-cli/src/main.rs:426:24
   3: forge::main
             at ./testsuite/forge-cli/src/main.rs:329:21
   4: core::ops::function::FnOnce::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:250:5
   5: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/backtrace.rs:155:18
   6: std::rt::lang_start::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:166:18
   7: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:284:13
   8: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
   9: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  10: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  11: std::rt::lang_start_internal::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:48
  12: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  13: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  14: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  15: std::rt::lang_start_internal
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:20
  16: main
  17: __libc_start_main
  18: _start
Debugging output:
NAME                                   READY   STATUS      RESTARTS   AGE
aptos-node-0-fullnode-eforge84-0       1/1     Running     0          11m
aptos-node-0-validator-0               1/1     Running     0          11m
aptos-node-1-fullnode-eforge84-0       1/1     Running     0          11m
aptos-node-1-validator-0               1/1     Running     0          11m
aptos-node-2-fullnode-eforge84-0       1/1     Running     0          11m
aptos-node-2-validator-0               1/1     Running     0          11m
aptos-node-3-fullnode-eforge84-0       1/1     Running     0          11m
aptos-node-3-validator-0               1/1     Running     0          11m
aptos-node-4-fullnode-eforge84-0       1/1     Running     0          11m
aptos-node-4-validator-0               1/1     Running     0          9m1s
aptos-node-5-validator-0               1/1     Running     0          11m
aptos-node-6-validator-0               1/1     Running     0          11m
forge-testnet-deployer-45drq           0/1     Completed   0          12m
genesis-aptos-genesis-eforge84-8k87v   0/1     Completed   0          11m

github-actions · 2024-12-06T19:39:49Z

❌ Forge suite `framework_upgrade` failure on `082b888013b4d47744289d4f5b7d29dd0854a766` ==> `ab4b83b6af799821e78b0edc80456b6621ebd8bf`

Compatibility test results for 082b888013b4d47744289d4f5b7d29dd0854a766 ==> ab4b83b6af799821e78b0edc80456b6621ebd8bf (PR)
Upgrade the nodes to version: ab4b83b6af799821e78b0edc80456b6621ebd8bf
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1264.82 txn/s, submitted: 1265.95 txn/s, failed submission: 1.13 txn/s, expired: 1.13 txn/s, latency: 2297.97 ms, (p50: 1800 ms, p70: 2100, p90: 4500 ms, p99: 6900 ms), latency samples: 112400
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 19.76 txn/s, submitted: 25.70 txn/s, failed submission: 0.03 txn/s, expired: 5.94 txn/s, latency: 1440.25 ms, (p50: 1200 ms, p70: 1800, p90: 2100 ms, p99: 3500 ms), latency samples: 31337
Test Failed: Waiting for nodes to catch up to target version and epoch (None, Some(29)) timed out after 20 seconds, current status: Ok([("validator-0", 147087, 28), ("validator-1", 147087, 28), ("validator-2", 147087, 28), ("validator-3", 147087, 28)])

Stack backtrace:
   0: anyhow::error::<impl anyhow::Error>::msg
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.89/src/error.rs:85:36
   1: aptos_forge::interface::swarm::wait_for_all_nodes_to_catchup_to_target_version_or_epoch::{{closure}}
             at ./testsuite/forge/src/interface/swarm.rs:463:24
   2: aptos_forge::interface::swarm::wait_for_all_nodes_to_catchup_to_epoch::{{closure}}
             at ./testsuite/forge/src/interface/swarm.rs:396:6
   3: aptos_forge::interface::swarm::SwarmExt::wait_for_all_nodes_to_change_epoch::{{closure}}
             at ./testsuite/forge/src/interface/swarm.rs:289:92
   4: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
   5: aptos_forge::interface::swarm::SwarmExt::fork_check::{{closure}}
             at ./testsuite/forge/src/interface/swarm.rs:200:14
   6: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
   7: <aptos_testcases::framework_upgrade::FrameworkUpgrade as aptos_forge::interface::network::NetworkTest>::run::{{closure}}
             at ./testsuite/testcases/src/framework_upgrade.rs:187:63
   8: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
   9: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/park.rs:281:63
  10: tokio::runtime::coop::with_budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs:107:5
  11: tokio::runtime::coop::budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs:73:5
  12: tokio::runtime::park::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/park.rs:281:31
  13: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/blocking.rs:66:9
  14: tokio::runtime::handle::Handle::block_on_inner::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/handle.rs:324:22
  15: tokio::runtime::context::runtime::enter_runtime
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/runtime.rs:65:16
  16: tokio::runtime::handle::Handle::block_on_inner
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/handle.rs:323:9
  17: tokio::runtime::handle::Handle::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/handle.rs:302:18
  18: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:332:50
  19: forge::run_forge_with_changelog
             at ./testsuite/forge-cli/src/main.rs:426:24
  20: forge::main
             at ./testsuite/forge-cli/src/main.rs:329:21
  21: core::ops::function::FnOnce::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:250:5
  22: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/backtrace.rs:155:18
  23: std::rt::lang_start::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:166:18
  24: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:284:13
  25: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  26: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  27: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  28: std::rt::lang_start_internal::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:48
  29: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  30: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  31: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  32: std::rt::lang_start_internal
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:20
  33: main
  34: __libc_start_main
  35: _start
Trailing Log Lines:
  30: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  31: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  32: std::rt::lang_start_internal
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:20
  33: main
  34: __libc_start_main
  35: _start

=== BEGIN JUNIT ===
<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="forge" tests="1" failures="1" errors="0" uuid="c2fbca4d-f71c-45e2-97cc-8c132d666bbb">
    <testsuite name="local" tests="1" disabled="0" errors="0" failures="1">
        <testcase name="framework_upgrade::framework-upgrade">
            <failure message="Waiting for nodes to catch up to target version and epoch (None, Some(29)) timed out after 20 seconds, current status: Ok([(&quot;validator-0&quot;, 147087, 28), (&quot;validator-1&quot;, 147087, 28), (&quot;validator-2&quot;, 147087, 28), (&quot;validator-3&quot;, 147087, 28)])

Stack backtrace:
   0: anyhow::error::&lt;impl anyhow::Error&gt;::msg
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.89/src/error.rs:85:36
   1: aptos_forge::interface::swarm::wait_for_all_nodes_to_catchup_to_target_version_or_epoch::{{closure}}
             at ./testsuite/forge/src/interface/swarm.rs:463:24
   2: aptos_forge::interface::swarm::wait_for_all_nodes_to_catchup_to_epoch::{{closure}}
             at ./testsuite/forge/src/interface/swarm.rs:396:6
   3: aptos_forge::interface::swarm::SwarmExt::wait_for_all_nodes_to_change_epoch::{{closure}}
             at ./testsuite/forge/src/interface/swarm.rs:289:92
   4: &lt;core::pin::Pin&lt;P&gt; as core::future::future::Future&gt;::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
   5: aptos_forge::interface::swarm::SwarmExt::fork_check::{{closure}}
             at ./testsuite/forge/src/interface/swarm.rs:200:14
   6: &lt;core::pin::Pin&lt;P&gt; as core::future::future::Future&gt;::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
   7: &lt;aptos_testcases::framework_upgrade::FrameworkUpgrade as aptos_forge::interface::network::NetworkTest&gt;::run::{{closure}}
             at ./testsuite/testcases/src/framework_upgrade.rs:187:63
   8: &lt;core::pin::Pin&lt;P&gt; as core::future::future::Future&gt;::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
   9: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/park.rs:281:63
  10: tokio::runtime::coop::with_budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs:107:5
  11: tokio::runtime::coop::budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs:73:5
  12: tokio::runtime::park::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/park.rs:281:31
  13: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/blocking.rs:66:9
  14: tokio::runtime::handle::Handle::block_on_inner::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/handle.rs:324:22
  15: tokio::runtime::context::runtime::enter_runtime
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/runtime.rs:65:16
  16: tokio::runtime::handle::Handle::block_on_inner
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/handle.rs:323:9
  17: tokio::runtime::handle::Handle::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/handle.rs:302:18
  18: aptos_forge::runner::Forge&lt;F&gt;::run
             at ./testsuite/forge/src/runner.rs:332:50
  19: forge::run_forge_with_changelog
             at ./testsuite/forge-cli/src/main.rs:426:24
  20: forge::main
             at ./testsuite/forge-cli/src/main.rs:329:21
  21: core::ops::function::FnOnce::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:250:5
  22: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/backtrace.rs:155:18
  23: std::rt::lang_start::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:166:18
  24: core::ops::function::impls::&lt;impl core::ops::function::FnOnce&lt;A&gt; for &amp;F&gt;::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:284:13
  25: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  26: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  27: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  28: std::rt::lang_start_internal::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:48
  29: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  30: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  31: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  32: std::rt::lang_start_internal
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:20
  33: main
  34: __libc_start_main
  35: _start"/>
        </testcase>
    </testsuite>
</testsuites>
=== END JUNIT ===

Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:381"},"thread_name":"main","hostname":"forge-framework-upgrade-pr-15527-1733511745-082b888013b4d477442","timestamp":"2024-12-06T19:39:45.627690Z","message":"Deleting namespace forge-framework-upgrade-pr-15527: Some(NamespaceStatus { conditions: None, phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:398"},"thread_name":"main","hostname":"forge-framework-upgrade-pr-15527-1733511745-082b888013b4d477442","timestamp":"2024-12-06T19:39:45.627708Z","message":"aptos-node resources for Forge removed in namespace: forge-framework-upgrade-pr-15527"}
Failed to run tests:
Tests Failed

failures:
    framework_upgrade::framework-upgrade

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Error: Tests Failed

Stack backtrace:
   0: anyhow::error::<impl anyhow::Error>::msg
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.89/src/error.rs:85:36
   1: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:358:13
   2: forge::run_forge_with_changelog
             at ./testsuite/forge-cli/src/main.rs:426:24
   3: forge::main
             at ./testsuite/forge-cli/src/main.rs:329:21
   4: core::ops::function::FnOnce::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:250:5
   5: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/backtrace.rs:155:18
   6: std::rt::lang_start::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:166:18
   7: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:284:13
   8: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
   9: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  10: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  11: std::rt::lang_start_internal::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:48
  12: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  13: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  14: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  15: std::rt::lang_start_internal
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:20
  16: main
  17: __libc_start_main
  18: _start
Debugging output:
NAME                                    READY   STATUS      RESTARTS   AGE
aptos-node-0-validator-0                1/1     Running     0          23m
aptos-node-1-validator-0                1/1     Running     0          23m
aptos-node-2-validator-0                1/1     Running     0          33m
aptos-node-3-validator-0                1/1     Running     0          23m
genesis-aptos-genesis-eforge250-89qfb   0/1     Completed   0          36m

grao1991

The PR title and description don't seem to be correct.

github-actions · 2025-01-21T01:59:24Z

This issue is stale because it has been open 45 days with no activity. Remove the stale label, comment or push a commit - otherwise this will be closed in 15 days.

junkil-park and others added 2 commits December 5, 2024 23:58

Event V2 Translation (#14615) (#15515)

17f4b41

[forge] bump graceful_overload mem threshold temporarily (#15511)

ab4b83b

(cherry picked from commit 3c6e693)

github-actions bot assigned thepomeranian Dec 6, 2024

github-actions bot added the cherry-pick label Dec 6, 2024

github-actions bot mentioned this pull request Dec 6, 2024

[forge] bump graceful_overload mem threshold temporarily #15511

Merged

ibalajiarun enabled auto-merge (squash) December 6, 2024 18:47

ibalajiarun approved these changes Dec 6, 2024

View reviewed changes

This comment has been minimized.

Sign in to view

sherry-x force-pushed the aptos-release-v1.25 branch from 17f4b41 to 3c6e693 Compare December 6, 2024 23:24

sherry-x requested review from msmouse, lightmark, grao1991, gregnazario, JoshLind, banool and 0xmaayan as code owners December 6, 2024 23:24

grao1991 reviewed Dec 6, 2024

View reviewed changes

github-actions bot added the Stale label Jan 21, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[cp][aptos-release-v1.25] [forge] bump graceful_overload mem threshold temporarily #15527

[cp][aptos-release-v1.25] [forge] bump graceful_overload mem threshold temporarily #15527

github-actions bot commented Dec 6, 2024

trunk-io bot commented Dec 6, 2024 •

edited

Loading

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

github-actions bot commented Dec 6, 2024

github-actions bot commented Dec 6, 2024

github-actions bot commented Dec 6, 2024

grao1991 left a comment

github-actions bot commented Jan 21, 2025

[cp][aptos-release-v1.25] [forge] bump graceful_overload mem threshold temporarily #15527

Are you sure you want to change the base?

[cp][aptos-release-v1.25] [forge] bump graceful_overload mem threshold temporarily #15527

Conversation

github-actions bot commented Dec 6, 2024

Backport

Questions ?

trunk-io bot commented Dec 6, 2024 • edited Loading

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

github-actions bot commented Dec 6, 2024

✅ Forge suite compat success on 082b888013b4d47744289d4f5b7d29dd0854a766 ==> ab4b83b6af799821e78b0edc80456b6621ebd8bf

github-actions bot commented Dec 6, 2024

❌ Forge suite realistic_env_max_load failure on ab4b83b6af799821e78b0edc80456b6621ebd8bf

github-actions bot commented Dec 6, 2024

❌ Forge suite framework_upgrade failure on 082b888013b4d47744289d4f5b7d29dd0854a766 ==> ab4b83b6af799821e78b0edc80456b6621ebd8bf

grao1991 left a comment

Choose a reason for hiding this comment

github-actions bot commented Jan 21, 2025

trunk-io bot commented Dec 6, 2024 •

edited

Loading

✅ Forge suite `compat` success on `082b888013b4d47744289d4f5b7d29dd0854a766` ==> `ab4b83b6af799821e78b0edc80456b6621ebd8bf`

❌ Forge suite `realistic_env_max_load` failure on `ab4b83b6af799821e78b0edc80456b6621ebd8bf`

❌ Forge suite `framework_upgrade` failure on `082b888013b4d47744289d4f5b7d29dd0854a766` ==> `ab4b83b6af799821e78b0edc80456b6621ebd8bf`