Repository navigation
CI: run StressSpec with 7 nodes on the 2-vCPU hosted agents - #8553
Merged
Aaronontheweb merged 1 commit intoSep 9, 2026
Conversation
With the full 10-node nr-of-nodes-* defaults, 10 is also the smallest MNTR_STRESSSPEC_NODECOUNT that StressSpecConfig.Settings accepts: the joining phases alone need >= 7 nodes (3 seed + 4 singleton join phases), and the leaving/shutdown phases together remove 7 nodes' worth of nr-of-nodes-*, which requires totalNumberOfNodes - 3 >= 7, i.e. >= 10. So the env var alone can't shrink the run below 10. StressSpecConfig.BuildConfig now shrinks the phase counts automatically whenever the node count drops below 10: it drops the two "-large" one-by-one leave/shutdown phases (each mostly redundant with the "-small" one-by-one phase that already covers that code path, just at a different point in the cluster's lifecycle) and halves the simultaneous "shutdown" phase from 2 to 1, freeing exactly the 3 nodes a 7-node run needs. Set MNTR_STRESSSPEC_NODECOUNT=7 for the four hosted multi-node CI lanes (Windows/Linux x classic/Artery) in build-system/pr-validation.yaml; no change to azure-pipeline.mntr-template.yaml was needed since its existing per-job env parameter already carries arbitrary env vars. Versus Pekko's 10-node reference, the 7-node CI run no longer exercises leaving/shutting down one-by-one nodes from a still-large cluster (only from a small one), nor the full 2-node simultaneous shutdown -- it only overlaps a single node's shutdown with the rest of the suite. Added StressSpecConfigSpec.cs: plain (non multi-node) unit tests that construct StressSpecConfig.Settings at 10, 7, and 6 nodes and assert the phase-count arithmetic (including that 6 nodes still throws, since the joining phases alone need 7).
Aaronontheweb
merged commit Sep 9, 2026
6bad960
into
fix/stress-spec-leave-before-terminate
2 checks passed
Aaronontheweb
added a commit
that referenced
this pull request
Sep 12, 2026
…pec at 7 nodes Mirrors dev's #8553. build-system/azure-pipeline.mntr-template.yaml gained no way to pass extra environment variables into the test-execution step, so build-system/pr-validation.yaml's single MNTR lane (net_mntr_windows) could not set MNTR_STRESSSPEC_NODECOUNT even though StressSpec.cs already reads that variable. Added an `env` parameter (defaulting to `{}`, matching dev's template) to the MNTR template and wired it into the `${{ parameters.command }}` step, then set `MNTR_STRESSSPEC_NODECOUNT: "7"` on the v1.5 MNTR lane so StressSpec runs at 7 nodes instead of its 13-node default on the 2-vCPU hosted agents. 7 is validated as a safe floor by the preceding StressSpec commit's StressSpecConfigSpec tests (StressSpecConfig.BuildConfig shrinks every phase to fit at exactly 7 nodes; 6 still throws).
Aaronontheweb
added a commit
that referenced
this pull request
Sep 12, 2026
…pec at 7 nodes Mirrors dev's #8553. build-system/azure-pipeline.mntr-template.yaml gained no way to pass extra environment variables into the test-execution step, so build-system/pr-validation.yaml's single MNTR lane (net_mntr_windows) could not set MNTR_STRESSSPEC_NODECOUNT even though StressSpec.cs already reads that variable. Added an `env` parameter (defaulting to `{}`, matching dev's template) to the MNTR template and wired it into the `${{ parameters.command }}` step, then set `MNTR_STRESSSPEC_NODECOUNT: "7"` on the v1.5 MNTR lane so StressSpec runs at 7 nodes instead of its 13-node default on the 2-vCPU hosted agents. 7 is validated as a safe floor by the preceding StressSpec commit's StressSpecConfigSpec tests (StressSpecConfig.BuildConfig shrinks every phase to fit at exactly 7 nodes; 6 still throws).
Aaronontheweb
added a commit
that referenced
this pull request
Sep 12, 2026
…pec at 7 nodes Mirrors dev's #8553. build-system/azure-pipeline.mntr-template.yaml gained no way to pass extra environment variables into the test-execution step, so build-system/pr-validation.yaml's single MNTR lane (net_mntr_windows) could not set MNTR_STRESSSPEC_NODECOUNT even though StressSpec.cs already reads that variable. Added an `env` parameter (defaulting to `{}`, matching dev's template) to the MNTR template and wired it into the `${{ parameters.command }}` step, then set `MNTR_STRESSSPEC_NODECOUNT: "7"` on the v1.5 MNTR lane so StressSpec runs at 7 nodes instead of its 13-node default on the 2-vCPU hosted agents. 7 is validated as a safe floor by the preceding StressSpec commit's StressSpecConfigSpec tests (StressSpecConfig.BuildConfig shrinks every phase to fit at exactly 7 nodes; 6 still throws).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes
build-system/pr-validation.yamlsetMNTR_STRESSSPEC_NODECOUNTto 7. The spec already reads that variable; nothing in CI set it before, so every run used the 10-node default on 2-vCPU agents.StressSpecConfigbuilds its config through a newBuildConfig(totalNumberOfNodes)method. Below 10 nodes it scales three phase counts so the run stays valid: the one-by-one "large" leave and shutdown phases go from 1 to 0, and the simultaneous shutdown from 2 to 1. At 10 nodes the config is unchanged, byte for byte.StressSpecConfigSpecpins the arithmetic with plain unit tests: the 10-node defaults are unchanged, 7 nodes produce exactly the reduced counts and fit the budget, and 6 nodes still throw, which proves 7 is the true floor.No dispatcher or thread pool setting changes.
Why
The maintainer's direction: if the agent's size is the problem, scale the spec down, do not add threads. The spec was configurable by node count but 10 was already the minimum the default phase counts accept. The join side consumes 3 seeds plus four single-node phases, so any total needs at least 7. The leave and shutdown side consumes 7 against a budget of the total minus 3 reserved nodes, so it needs 10. Reducing the three phases above brings that side to 4, which fits at 7, and 7 is where the join side bottoms out. The agent traced the used-role count through every phase at 7 and it never goes negative.
What a 7-node run no longer exercises against Pekko's 10-node reference: leaving or shutting down a node one at a time from a still-large cluster, and shutting down two nodes at once. The 10-node run is one variable away on a bigger agent.
How it was checked
dotnet build src/core/Akka.Cluster.Tests.MultiNode -c Release -warnaserrorclean.StressSpecConfigSpec: 6 passed.StressSpecitself needs its node processes, so CI is the first run, and the four multi-node lanes on this PR run it at 7.