Skip to content

Expose per node counts#93439

Merged
idegtiarenko merged 3 commits intoelastic:mainfrom
idegtiarenko:expose_per_node_counts
Feb 2, 2023
Merged

Expose per node counts#93439
idegtiarenko merged 3 commits intoelastic:mainfrom
idegtiarenko:expose_per_node_counts

Conversation

@idegtiarenko
Copy link
Contributor

This PR adds a per node shard count, forecasted ingest load and forecasted shard sizes to the internal stats endpoint.

@idegtiarenko idegtiarenko added >non-issue :Distributed/Allocation All issues relating to the decision making around placing a shard (both master logic & on the nodes) Team:Distributed Meta label for distributed team. v8.7.0 labels Feb 2, 2023
@github-actions
Copy link
Contributor

github-actions bot commented Feb 2, 2023

Documentation preview:

@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (Team:Distributed)

return builder.startObject()
.field("shard_count", shards)
.field("forecasted_write_load", forecastedWriteLoad)
.humanReadableField("forecasted_disk_usage_bytes", "forecasted_disk_usage", ByteSizeValue.ofBytes(forecastedShardSize))
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be beneficial to also add actual disk usage here.
I think there is a confusion between forecasted disk usage and actual disk usage, may be adding both here would help to highlight the difference

- is_true: 'cluster_balance_stats.nodes.test-cluster-0'
- gte: { 'cluster_balance_stats.nodes.test-cluster-0.shard_count' : 0 }
- gte: { 'cluster_balance_stats.nodes.test-cluster-0.forecasted_write_load': 0.0 }
- gte: { 'cluster_balance_stats.nodes.test-cluster-0.forecasted_disk_usage_bytes' : 0 }
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed the structure (tiers are now nested under tiers field and added a new nodes field in the cluster_balance_stats) as well as renamed (total_write_load -> forecasted_write_load and total_shard_size -> forecasted_disk_usage). I think this better represents the actual values.
The endpoint is internal and has not been released in any version yet, so I think it is okay to do so.

Copy link
Contributor

@DaveCTurner DaveCTurner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, one (global) naming nit is that I'd prefer forecast over forecasted throughout. See e.g. https://www.techtarget.com/whatis/feature/Forecast-or-forecasted.

Copy link
Contributor

@DaveCTurner DaveCTurner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@idegtiarenko idegtiarenko merged commit 513dc2f into elastic:main Feb 2, 2023
@idegtiarenko idegtiarenko deleted the expose_per_node_counts branch February 2, 2023 15:13
mark-vieira pushed a commit to mark-vieira/elasticsearch that referenced this pull request Feb 2, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

:Distributed/Allocation All issues relating to the decision making around placing a shard (both master logic & on the nodes) >non-issue Team:Distributed Meta label for distributed team. v8.7.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants