Adjust free disk space thresholds #4514
Conversation
There was a problem hiding this comment.
Pull Request Overview
This PR increases the default disk space thresholds for node health monitoring and shutdown. The changes raise the minimum required disk space from 0.5 GiB to 10 GiB (shutdown threshold) and the warning threshold from 1 GiB to 100 GiB.
Key changes:
- Increase shutdown threshold from 0.5 GiB to 10 GiB
- Increase warning threshold from 1 GiB to 100 GiB
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
StephenButtolph
left a comment
StephenButtolph
left a comment
There was a problem hiding this comment.
These are certainly better than the current values.
I do wonder if we should be even more aggressive (or make them a percentage of the volume size rather than hard numbers).
But I'd be happy to merge this at the least
| } | ||
|
|
||
| flags.SetDefaults(FlagsMap{ | ||
| config.SystemTrackerRequiredAvailableDiskSpaceKey: "1GB", |
There was a problem hiding this comment.
Looks like Github's runners don't have much disk space in them...
yacovm
force-pushed
the
changeLimits
branch
2 times, most recently
from
db6ca47 to
361eeba
Compare
|
@maru-ava do you have any idea why the bootstrap monitor task fails in CI? |
There was a problem hiding this comment.
The PR for which I enabled log collection reports the following FATAL (entry kind-control-plane/pods/bootstrap-test-e2e-5bs8z_avalanchego-node-cz6sv-0_31b6ce5d-f40a-4189-94d6-c0f7765a0300/avago/4.log in the artifact provided by the test run:
2025-11-14T18:36:31.211528364Z stdout F [11-14|18:36:31.211] FATAL node/node.go:1463 low on disk space. Shutting down... {"remainingDiskBytes": 6975516672}
This is consistent with my inline comments regarding the configuration changes you've proposed. In #4516 I also cleaned up configuration of the bootstrap monitor pods so that they start from the tmpnet configuration you've changed rather than being entirely distinct.
I recommend cherry-picking my commits from #4516 and reverting your change to tests/fixture/bootstrapmonitor/e2e/e2e_test.go
StephenButtolph
dismissed stale reviews from joshua-kim and themself
CI failing
Signed-off-by: Yacov Manevich <[email protected]>
…y override Signed-off-by: Yacov Manevich <[email protected]>
Signed-off-by: Yacov Manevich <[email protected]>
5 participants
Why this should be merged
The current threshold for free space are very low and as a result the warning may come too late (see #4517).
This commit changes them to something more reasonable.
How this works
Changes default thresholds
How this was tested
CI
Need to be documented in RELEASES.md?