Skip to content

[MetricBeat][StackMonitoring] fetching index summary from node/_stats endpoint #45049

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pickypg merged 19 commits into from
Jul 4, 2025
Merged

[MetricBeat][StackMonitoring] fetching index summary from node/_stats endpoint #45049

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
461d769
Added tests on event payload
filippomachi Jun 26, 2025
a425f98
Added tests
filippomachi Jun 27, 2025
41b5cec
Refactored logic
filippomachi Jun 27, 2025
c646e1d
Removing old logic
filippomachi Jun 27, 2025
f56b372
Refactored code
filippomachi Jun 27, 2025
31640d7
Fixing imports
filippomachi Jun 27, 2025
21b5011
Added tests
filippomachi Jun 27, 2025
7a6ce61
Improved naming
filippomachi Jun 27, 2025
b89eed0
Removed log
filippomachi Jun 27, 2025
0391371
Simplified solution
filippomachi Jun 28, 2025
99dd0b1
Removed old code
filippomachi Jul 3, 2025
200f2cf
Using just new code
filippomachi Jul 3, 2025
7579f82
Fixing imports
filippomachi Jul 3, 2025
36fb857
Bulk section is optional
filippomachi Jul 3, 2025
3eedd61
Improved tests
filippomachi Jul 4, 2025
1c0a092
Removed bulk section
filippomachi Jul 4, 2025
00246e4
Merge branch 'main' into metricbeat-index-summary-using-nodestats
filippomachi Jul 4, 2025
19b293d
Adding missing header
filippomachi Jul 4, 2025
9b6280e
Update CHANGELOG.next
pickypg Jul 4, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.next.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,7 @@ otherwise no tag is added. {issue}42208[42208] {pull}42403[42403]

*Metricbeat*

- Change index summary metricset to use `_nodes/stats` API instead of `_stats` API to avoid data gaps. {pull}45049[45049]
- Add support for `_nodes/stats` URIs that work with legacy versions of Elasticsearch {pull}44307[44307]
- Setting period for counter cache for Prometheus remote_write at least to 60sec {pull}38553[38553]
- Remove fallback to the node limit for the `kubernetes.pod.cpu.usage.limit.pct` and `kubernetes.pod.memory.usage.limit.pct` metrics calculation
Expand Down
97 changes: 97 additions & 0 deletions metricbeat/module/elasticsearch/index_summary/_meta/test/expected_event_7.17.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
{
"RootFields": {
"service": {
"name": "elasticsearch"
}
},
"ModuleFields": {
"cluster": {
"id": "1234",
"name": "helloworld"
}
},
"MetricSetFields": {
"primaries": {
"docs": {
"count": 464,
"deleted": 252
},
"store": {
"size": {
"bytes": 85621934
},
"total_data_set_size": {
"bytes": 85621934
}
},
"indexing": {
"index": {
"count": 390,
"time": {
"ms": 1815
}
}
},
"search": {
"query": {
"count": 313,
"time": {
"ms": 806
}
}
},
"segments": {
"count": 80,
"memory": {
"bytes": 114024
}
}
},
"total": {
"docs": {
"count": 464,
"deleted": 252
},
"store": {
"size": {
"bytes": 85621934
},
"total_data_set_size": {
"bytes": 85621934
}
},
"indexing": {
"index": {
"count": 390,
"time": {
"ms": 1815
}
}
},
"search": {
"query": {
"count": 313,
"time": {
"ms": 806
}
}
},
"segments": {
"count": 80,
"memory": {
"bytes": 114024
}
}
}
},
"Index": ".monitoring-es-8-mb",
"ID": "",
"Namespace": "",
"Timestamp": "0001-01-01T00:00:00Z",
"Error": null,
"Host": "",
"Service": "",
"Took": 0,
"Period": 0,
"DisableTimeSeries": false
}
97 changes: 97 additions & 0 deletions metricbeat/module/elasticsearch/index_summary/_meta/test/expected_event_8.17.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
{
"RootFields": {
"service": {
"name": "elasticsearch"
}
},
"ModuleFields": {
"cluster": {
"id": "1234",
"name": "helloworld"
}
},
"MetricSetFields": {
"primaries": {
"docs": {
"count": 8030785507,
"deleted": 23527
},
"indexing": {
"index": {
"count": 56600767530,
"time": {
"ms": 5360597433
}
}
},
"search": {
"query": {
"count": 7898882196,
"time": {
"ms": 5848049597
}
}
},
"segments": {
"count": 11284,
"memory": {
"bytes": 0
}
},
"store": {
"size": {
"bytes": 1686972455265
},
"total_data_set_size": {
"bytes": 1686972455265
}
}
},
"total": {
"docs": {
"count": 8030785507,
"deleted": 23527
},
"indexing": {
"index": {
"count": 56600767530,
"time": {
"ms": 5360597433
}
}
},
"search": {
"query": {
"count": 7898882196,
"time": {
"ms": 5848049597
}
}
},
"segments": {
"count": 11284,
"memory": {
"bytes": 0
}
},
"store": {
"size": {
"bytes": 1686972455265
},
"total_data_set_size": {
"bytes": 1686972455265
}
}
}
},
"Index": ".monitoring-es-8-mb",
"ID": "",
"Namespace": "",
"Timestamp": "0001-01-01T00:00:00Z",
"Error": null,
"Host": "",
"Service": "",
"Took": 0,
"Period": 0,
"DisableTimeSeries": false
}
97 changes: 97 additions & 0 deletions metricbeat/module/elasticsearch/index_summary/_meta/test/expected_event_xpack_7.17.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
{
"RootFields": {
"service": {
"name": "elasticsearch"
}
},
"ModuleFields": {
"cluster": {
"id": "1234",
"name": "helloworld"
}
},
"MetricSetFields": {
"primaries": {
"docs": {
"count": 464,
"deleted": 252
},
"store": {
"size": {
"bytes": 85621934
},
"total_data_set_size": {
"bytes": 85621934
}
},
"indexing": {
"index": {
"count": 390,
"time": {
"ms": 1815
}
}
},
"search": {
"query": {
"count": 313,
"time": {
"ms": 806
}
}
},
"segments": {
"count": 80,
"memory": {
"bytes": 114024
}
}
},
"total": {
"docs": {
"count": 464,
"deleted": 252
},
"store": {
"size": {
"bytes": 85621934
},
"total_data_set_size": {
"bytes": 85621934
}
},
"indexing": {
"index": {
"count": 390,
"time": {
"ms": 1815
}
}
},
"search": {
"query": {
"count": 313,
"time": {
"ms": 806
}
}
},
"segments": {
"count": 80,
"memory": {
"bytes": 114024
}
}
}
},
"Index": "",
"ID": "",
"Namespace": "",
"Timestamp": "0001-01-01T00:00:00Z",
"Error": null,
"Host": "",
"Service": "",
"Took": 0,
"Period": 0,
"DisableTimeSeries": false
}
97 changes: 97 additions & 0 deletions metricbeat/module/elasticsearch/index_summary/_meta/test/expected_event_xpack_8.17.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
{
"RootFields": {
"service": {
"name": "elasticsearch"
}
},
"ModuleFields": {
"cluster": {
"id": "1234",
"name": "helloworld"
}
},
"MetricSetFields": {
"primaries": {
"docs": {
"count": 8030785507,
"deleted": 23527
},
"indexing": {
"index": {
"count": 56600767530,
"time": {
"ms": 5360597433
}
}
},
"search": {
"query": {
"count": 7898882196,
"time": {
"ms": 5848049597
}
}
},
"segments": {
"count": 11284,
"memory": {
"bytes": 0
}
},
"store": {
"size": {
"bytes": 1686972455265
},
"total_data_set_size": {
"bytes": 1686972455265
}
}
},
"total": {
"docs": {
"count": 8030785507,
"deleted": 23527
},
"indexing": {
"index": {
"count": 56600767530,
"time": {
"ms": 5360597433
}
}
},
"search": {
"query": {
"count": 7898882196,
"time": {
"ms": 5848049597
}
}
},
"segments": {
"count": 11284,
"memory": {
"bytes": 0
}
},
"store": {
"size": {
"bytes": 1686972455265
},
"total_data_set_size": {
"bytes": 1686972455265
}
}
}
},
"Index": "",
"ID": "",
"Namespace": "",
"Timestamp": "0001-01-01T00:00:00Z",
"Error": null,
"Host": "",
"Service": "",
"Took": 0,
"Period": 0,
"DisableTimeSeries": false
}
Loading