Add shard heap usage to ClusterInfo#139557
Add shard heap usage to ClusterInfo#139557elasticsearchmachine merged 16 commits intoelastic:mainfrom
Conversation
elasticsearchmachine
added
serverless-linked
DiannaHohensee
force-pushed
the
2025/12/11/ES-12882-simulate-heap
branch
2 times, most recently
from
bc01ce5 to
1f71b0f
Compare
Relates ES-12882
DiannaHohensee
force-pushed
the
2025/12/11/ES-12882-simulate-heap
branch
from
1f71b0f to
9da594e
Compare
elasticsearchmachine
added
the
needs:triage
DiannaHohensee
added
:Distributed/Allocation
|
Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination) |
DiannaHohensee
removed
the
serverless-linked
| for (var entry : estimatedShardHeapUsages.entrySet()) { | ||
| assertThat(entry.getValue().shardHeapUsageBytes(), greaterThanOrEqualTo(0L)); | ||
| assertThat(entry.getValue().indexHeapUsageBytes(), greaterThanOrEqualTo(0L)); | ||
| } |
There was a problem hiding this comment.
We should only have to put a single document in each index and we'd get a non-zero value here wouldn't we? (because of the mapping size in bytes at least)
Then we could assert greaterThan(0L) which seems stronger?
There was a problem hiding this comment.
I'm not certain what you mean. The heap values are randomly generated for this test, anywhere from 0 to Long.MAX_VALUE. So even if some writes were done, that would not guarantee a non-zero value?
Stateful doesn't have real metrics for heap usage, that's only in serverless.
| int numEntries = randomIntBetween(0, 128); | ||
| Map<ShardId, ShardAndIndexHeapUsage> shardHeapUsageBuilder = new HashMap<>(numEntries); | ||
| for (int i = 0; i < numEntries; i++) { | ||
| shardHeapUsageBuilder.put(randomShardId(), new ShardAndIndexHeapUsage(randomLong(), randomLong())); |
There was a problem hiding this comment.
Should these be randomNonNegativeLong(), I think a negative value would be unexpected here right?
There was a problem hiding this comment.
Yes, thanks for catching this. Fixed 👍
…nternalClusterInfoServiceSchedulingTests to verify the new method is called
DiannaHohensee
left a comment
DiannaHohensee
left a comment
There was a problem hiding this comment.
Updated to non-negative longs, and added a few verifies that the new collection method is called by the cluster info service (lots of mocks). 1ff495e
| int numEntries = randomIntBetween(0, 128); | ||
| Map<ShardId, ShardAndIndexHeapUsage> shardHeapUsageBuilder = new HashMap<>(numEntries); | ||
| for (int i = 0; i < numEntries; i++) { | ||
| shardHeapUsageBuilder.put(randomShardId(), new ShardAndIndexHeapUsage(randomLong(), randomLong())); |
There was a problem hiding this comment.
Yes, thanks for catching this. Fixed 👍
| for (var entry : estimatedShardHeapUsages.entrySet()) { | ||
| assertThat(entry.getValue().shardHeapUsageBytes(), greaterThanOrEqualTo(0L)); | ||
| assertThat(entry.getValue().indexHeapUsageBytes(), greaterThanOrEqualTo(0L)); | ||
| } |
There was a problem hiding this comment.
I'm not certain what you mean. The heap values are randomly generated for this test, anywhere from 0 to Long.MAX_VALUE. So even if some writes were done, that would not guarantee a non-zero value?
Stateful doesn't have real metrics for heap usage, that's only in serverless.
DiannaHohensee
added
the
auto-merge-without-approval
DiannaHohensee
removed
the
auto-merge-without-approval
|
Oh, this needs to wait for the serverless changes because there's a change to an interface that serverless implements. Doh. |
|
Pinging @elastic/es-distributed (Team:Distributed) |
DiannaHohensee
added
serverless-linked
…on-sliced-reindex * upstream/main: Update docs for v9.3.1 release (elastic#142887) Update docs for v9.2.6 release (elastic#142888) Improves visibility of vector index options and inference configuration (elastic#141653) Disable CAE in microsoft-graph-authz plugin (elastic#142848) Small improvements to `GetSnapshotsIT#testAllFeatures` (elastic#142825) Fix IndexSettingsTests synthetic ID tests (elastic#142654) [Test] Unmute tests of SnapshotShutdownIT (elastic#142921) Fixing metrics_info.json kibana definition file name (elastic#142813) [Packaging] Disable glibc 2.43 malloc huge pages in Wolfi images (elastic#142894) Mute org.elasticsearch.xpack.searchablesnapshots.SearchableSnapshotsTSDBSyntheticIdIntegTests testSearchableSnapshot elastic#142918 Add shard heap usage to ClusterInfo (elastic#139557) ESQL: Load script fields row-by-row (elastic#142807) ESQL: Consolidate doc values memory tracking (elastic#142816) ES-14124 Create Index Count Limit User documentation Page (elastic#142570) Add a es819 codec test to verify tryRead returns null if may contain duplicates (elastic#142409) Support arithmetic operations for dense_vectors: scalar version (elastic#141060) [Transform] Allow project_routing (elastic#142421) Refactor query rewrite async actions for knn and sparse_vector queries (elastic#142889) Do not mark bulk indexing requests as retried after primary relocations (elastic#142157)
Adds a map of shard heap usages to the ClusterInfo, as well as adding an interface for collection thereof to the EstimatedHeapUsageCollector, which the InternalClusterInfoService uses to supply the new shard heap values. This is a step towards simulating heap usage changes when a shard moves during shard allocation computation. Relates ES-12882
3 participants
Adds a map of shard heap usages to the ClusterInfo, as well as
adding an interface for collection thereof to the
EstimatedHeapUsageCollector, which the
InternalClusterInfoService uses to supply the new shard heap values.
This is a step towards simulating heap usage changes when a shard
moves during shard allocation computation.
Relates ES-12882