Skip to content

[MetricBeat][StackMonitoring] fetching index summary from node/_stats endpoint#45049

Merged
pickypg merged 19 commits intomainfrom
metricbeat-index-summary-using-nodestats
Jul 4, 2025
Merged

[MetricBeat][StackMonitoring] fetching index summary from node/_stats endpoint#45049
pickypg merged 19 commits intomainfrom
metricbeat-index-summary-using-nodestats

Conversation

@filippomachi
Copy link
Copy Markdown
Contributor

@filippomachi filippomachi commented Jun 26, 2025

Proposed commit message

Fetching index summary metricset from node/_stats endpoint

Please explain:

  • WHAT: changing the source of data for index summary metrics
  • WHY: the metrics obtained from the Elasticsearch _stats endpoint are based on individual indices, and may become inaccurate or unreliable if indices are deleted or closed.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

None

How to test this PR locally

integration tests have been improved
try metricbeat integration locally closing/deleting indexes

Related issues

@botelastic botelastic Bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jun 26, 2025
@github-actions

This comment was marked as off-topic.

@mergify

This comment was marked as resolved.

@botelastic botelastic Bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jun 26, 2025
@filippomachi filippomachi added the backport-active-all Automated backport with mergify to all the active branches label Jun 26, 2025
@filippomachi filippomachi marked this pull request as ready for review July 4, 2025 15:41
@filippomachi filippomachi requested a review from a team as a code owner July 4, 2025 15:41
@pickypg pickypg added bug and removed enhancement labels Jul 4, 2025
@pickypg pickypg enabled auto-merge (squash) July 4, 2025 16:35
Copy link
Copy Markdown
Member

@pickypg pickypg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. This should be a notable improvement over the old approach for users that do a lot of index deletion.

Comment thread metricbeat/module/elasticsearch/index_summary/index_summary.go
@pickypg pickypg disabled auto-merge July 4, 2025 19:20
@pickypg pickypg enabled auto-merge (squash) July 4, 2025 19:22
@pickypg pickypg merged commit 63a537a into main Jul 4, 2025
48 checks passed
@pickypg pickypg deleted the metricbeat-index-summary-using-nodestats branch July 4, 2025 19:22
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Jul 4, 2025

@Mergifyio backport 8.17 8.18 8.19 9.0 9.1

@mergify
Copy link
Copy Markdown
Contributor

mergify Bot commented Jul 4, 2025

mergify Bot pushed a commit that referenced this pull request Jul 4, 2025
… endpoint (#45049)

Fetching index summary metricset from `_nodes/stats` endpoint instead of `_stats` so that as indexes are deleted, the summary stats are not effectively lost during the next poll because nodes cache their data. Gaps can still appear in charts if nodes are restarted (as their cache is zeroed out).

(cherry picked from commit 63a537a)
mergify Bot pushed a commit that referenced this pull request Jul 4, 2025
… endpoint (#45049)

Fetching index summary metricset from `_nodes/stats` endpoint instead of `_stats` so that as indexes are deleted, the summary stats are not effectively lost during the next poll because nodes cache their data. Gaps can still appear in charts if nodes are restarted (as their cache is zeroed out).

(cherry picked from commit 63a537a)
mergify Bot pushed a commit that referenced this pull request Jul 4, 2025
… endpoint (#45049)

Fetching index summary metricset from `_nodes/stats` endpoint instead of `_stats` so that as indexes are deleted, the summary stats are not effectively lost during the next poll because nodes cache their data. Gaps can still appear in charts if nodes are restarted (as their cache is zeroed out).

(cherry picked from commit 63a537a)
mergify Bot pushed a commit that referenced this pull request Jul 4, 2025
… endpoint (#45049)

Fetching index summary metricset from `_nodes/stats` endpoint instead of `_stats` so that as indexes are deleted, the summary stats are not effectively lost during the next poll because nodes cache their data. Gaps can still appear in charts if nodes are restarted (as their cache is zeroed out).

(cherry picked from commit 63a537a)

# Conflicts:
#	metricbeat/module/elasticsearch/index_summary/index_summary.go
mergify Bot pushed a commit that referenced this pull request Jul 4, 2025
… endpoint (#45049)

Fetching index summary metricset from `_nodes/stats` endpoint instead of `_stats` so that as indexes are deleted, the summary stats are not effectively lost during the next poll because nodes cache their data. Gaps can still appear in charts if nodes are restarted (as their cache is zeroed out).

(cherry picked from commit 63a537a)
pickypg pushed a commit that referenced this pull request Jul 4, 2025
… endpoint (#45049) (#45204)

Fetching index summary metricset from `_nodes/stats` endpoint instead of `_stats` so that as indexes are deleted, the summary stats are not effectively lost during the next poll because nodes cache their data. Gaps can still appear in charts if nodes are restarted (as their cache is zeroed out).

(cherry picked from commit 63a537a)

Co-authored-by: filippomachi <filippo.machi@elastic.co>
pickypg pushed a commit that referenced this pull request Jul 4, 2025
… endpoint (#45049) (#45200)

Fetching index summary metricset from `_nodes/stats` endpoint instead of `_stats` so that as indexes are deleted, the summary stats are not effectively lost during the next poll because nodes cache their data. Gaps can still appear in charts if nodes are restarted (as their cache is zeroed out).

(cherry picked from commit 63a537a)

Co-authored-by: filippomachi <filippo.machi@elastic.co>
pickypg added a commit that referenced this pull request Jul 4, 2025
…ummary from node/_stats endpoint (#45203)

* [MetricBeat][StackMonitoring] fetching index summary from node/_stats endpoint (#45049)

Fetching index summary metricset from `_nodes/stats` endpoint instead of `_stats` so that as indexes are deleted, the summary stats are not effectively lost during the next poll because nodes cache their data. Gaps can still appear in charts if nodes are restarted (as their cache is zeroed out).

(cherry picked from commit 63a537a)

# Conflicts:
#	metricbeat/module/elasticsearch/index_summary/index_summary.go

* Fix merge conflict (accept new changes exclusively)

---------

Co-authored-by: filippomachi <filippo.machi@elastic.co>
Co-authored-by: Chris Earle <pickypg@gmail.com>
pickypg pushed a commit that referenced this pull request Jul 4, 2025
… endpoint (#45049) (#45201)

Fetching index summary metricset from `_nodes/stats` endpoint instead of `_stats` so that as indexes are deleted, the summary stats are not effectively lost during the next poll because nodes cache their data. Gaps can still appear in charts if nodes are restarted (as their cache is zeroed out).

(cherry picked from commit 63a537a)

Co-authored-by: filippomachi <filippo.machi@elastic.co>
pickypg pushed a commit that referenced this pull request Jul 4, 2025
… endpoint (#45049) (#45202)

Fetching index summary metricset from `_nodes/stats` endpoint instead of `_stats` so that as indexes are deleted, the summary stats are not effectively lost during the next poll because nodes cache their data. Gaps can still appear in charts if nodes are restarted (as their cache is zeroed out).

(cherry picked from commit 63a537a)

Co-authored-by: filippomachi <filippo.machi@elastic.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport-active-all Automated backport with mergify to all the active branches bug Feature:Stack Monitoring Team:Monitoring Stack Monitoring team Team:Opex

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Stack Monitoring] Deleted indices can cause gaps in Elasticsearch overview

3 participants