Collect bulk indexing stats for Elasticsearch metricsets by ycombinator · Pull Request #17992 · elastic/beats

ycombinator · 2020-04-25T18:41:03Z

What does this PR do?

Collects new bulk indexing metrics being reported by Elasticsearch in the index, index_summary, and node_stats metricsets for the Stack Monitoring code path (i.e. when xpack.enabled: true is set in these metricsets' configurations).

These metricsets correspond to type:index_stats, type:indices_stats, and type:node_stats documents, respectively, in .monitoring-es-* indices used by the Stack Monitoring UI.

Why is it important?

To restore parity between legacy internal collection and Metricbeat collection for Elasticsearch monitoring. The changes corresponding to this PR for legacy internal collection were made in elastic/elasticsearch#52208.

Checklist

My code follows the style guidelines of this project
~~I have commented my code, particularly in hard-to-understand areas~~
~~I have made corresponding changes to the documentation~~
~~I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works.
- In addition to the unit test added in this PR, parity tests already exist that are currently failing and should start passing once the changes in this PR are released.
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Run the latest 8.0.0-SNAPSHOT build of Elasticsearch so it exposes the new bulk indexing metrics being collected by the code in this PR.

docker pull docker.elastic.co/elasticsearch/elasticsearch:8.0.0-SNAPSHOT
docker run --name elasticsearch -p 9200:9200 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:8.0.0-SNAPSHOT

Build Metricbeat with the changes in this PR.

cd $GOPATH/src/github.com/elastic/beats/metricbeat
mage build

Enable the elasticsearch-xpack module.

metricbeat modules enable elasticsearch-xpack

Create a minimal Metricbeat configuration for testing.

cat <<EOF > ./metricbeat.test.yml
metricbeat.config.modules:
  path: \${path.config}/modules.d/*.yml
  reload.enabled: false

output.console.enabled: true

monitoring.elasticsearch:
  hosts: [ "localhost:9200" ]
EOF

Here, we only enable monitoring.elasticsearch to generate some bulk indexing requests to Elasticsearch so we can see non-zero bulk metrics.

Run Metricbeat with the minimal configuration and look for the new bulk indexing metrics' fields. You may need to run these for ~30 seconds for some bulk indexing to happen and corresponding metrics to be generated by Elasticsearch.

For the index metricset, i.e.type:index_stats documents:

./metricbeat -c metricbeat.test.yml | jq 'select(.type == "index_stats") | .index_stats.total.bulk'

./metricbeat -c metricbeat.test.yml | jq 'select(.type == "index_stats") | .index_stats.primaries.bulk'

For the index_summary metricset, i.e.type:indices_stats documents:

./metricbeat -c metricbeat.test.yml | jq 'select(.type == "indices_stats") | .indices_stats._all.total.bulk'

./metricbeat -c metricbeat.test.yml | jq 'select(.type == "indices_stats") | .indices_stats._all.primaries.bulk'

For the node_stats metricset, i.e.type:node_stats documents:

./metricbeat -c metricbeat.test.yml | jq 'select(.type == "node_stats") | .node_stats.indices.bulk'

Related issues

Resolves [Metricbeat][parity-tests] Missing some newly added ES fields #17977

elasticmachine · 2020-04-25T18:44:02Z

Pinging @elastic/stack-monitoring (Stack monitoring)

elasticmachine · 2020-04-25T18:44:03Z

Pinging @elastic/integrations-services (Team:Services)

ycombinator · 2020-04-25T19:22:21Z

@igoristic I've requested your review on this PR as a functional reviewer. Please see the "How to test this PR locally" section in the PR description for instructions on how to functionally test this PR.

elasticmachine · 2020-04-30T11:50:14Z

💚 Build Succeeded

Expand to view the summary

Build stats

Build Cause: [Pull request Collect bulk indexing stats for Elasticsearch metricsets #17992 updated]
Start Time: 2020-05-01T02:05:06.360+0000
Duration: 82 min 29 sec (4948853)
Commit: a87c509

Test stats 🧪

Test	Results
Failed	0
Passed	3008
Skipped	744
Total	3752

jsoriano · 2020-04-30T11:57:25Z

Failure in statsd tests should be fixed by #18130, the other failures may be related.

ycombinator · 2020-04-30T19:20:36Z

Looking at the CI errors from the elasticsearch module system tests, they are legitimate and need to be addressed in this PR.

This PR adds bulk to the list of metrics requested from the Elasticsearch Stats API. This metric is only available starting Elasticsearch 7.8.0. However, our tests use Elasticsearch 7.5.2. Which is why the tests are failing.

To address this correctly, we need to add bulk to the list of metrics requested from the Elasticsearch Stats API if the Elasticsearch version is >= 7.8.0. This will be similar to the work done recently in the logstash module in #17497.

Accordingly, I'm moving this PR back into draft state. Once I've implemented the necessary changes, I'll put it back into review.

igoristic · 2020-04-30T22:09:11Z

Looking at the CI errors from the elasticsearch module system tests, they are legitimate and need to be addressed in this PR ...

Yeah I was wondering if I should wait for the CI to turn green before I can test it

ycombinator · 2020-04-30T22:10:13Z

Yeah I was wondering if I should wait for the CI to turn green before I can test it

That would be ideal but there might be unrelated test failures. I'll ping you here where it's ready to test.

ycombinator · 2020-05-01T03:11:00Z

@igoristic CI is looking good now. There are failures but they are unrelated to this PR. Please go ahead and test it. Thanks!

igoristic

Tested all case scenarios mentioned in step 5, and those fields are no longer null (like they are in master):

Thank you for fixing this @ycombinator 🙇

…8229) * Collecting new index_stats bulk metrics * Collecting new indices_stats bulk metrics * Collecting new node_stats bulk metrics * Adding CHANGELOG entry * Request bulk stats * Request bulk metrics only if supported * Fixing code and tests * Fixing code so only service URI path is replaced * Updating unit tests

chrisronline · 2020-05-11T15:12:33Z

metricbeat/module/elasticsearch/index/data_xpack.go

+	TotalOperations   int `json:"total_operations"`
+	TotalTimeInMillis int `json:"total_time_in_millis"`
+	TotalSizeInBytes  int `json:"total_size_in_bytes"`
+	AvgTimeInMillis   int `json:"throttle_time_in_millis"`


@ycombinator Is this a typo? Should it be json:"avg_time_in_millis"?

Yes, it should 😞. Good eye, nice catch! I assume parity tests failed? I'll put up a PR to fix it. Thanks!

PR is up: #18420

ycombinator added Feature:Stack Monitoring Team:Services (Deprecated) Label for the former Integrations-Services team labels Apr 25, 2020

ycombinator added Metricbeat Metricbeat module needs_backport PR is waiting to be backported to other branches. v7.8.0 v8.0.0 [zube]: In Review labels Apr 25, 2020

ycombinator requested a review from igoristic April 25, 2020 19:19

ycombinator requested a review from a team April 30, 2020 09:56

jsoriano approved these changes Apr 30, 2020

View reviewed changes

ycombinator added 5 commits April 30, 2020 10:47

Collecting new index_stats bulk metrics

c0856e2

Collecting new indices_stats bulk metrics

d99715c

Collecting new node_stats bulk metrics

b44b6b7

Adding CHANGELOG entry

0f57b9b

Request bulk stats

65be0c9

ycombinator marked this pull request as draft April 30, 2020 19:20

ycombinator added in progress Pull request is currently in progress. and removed [zube]: In Review labels Apr 30, 2020

Request bulk metrics only if supported

3d9ccad

ycombinator marked this pull request as ready for review April 30, 2020 21:33

Fixing code and tests

2736063

ycombinator added 2 commits April 30, 2020 18:31

Fixing code so only service URI path is replaced

4e88e8c

Updating unit tests

a87c509

igoristic approved these changes May 5, 2020

View reviewed changes

ycombinator merged commit c87c8e7 into elastic:master May 5, 2020

ycombinator deleted the mb-es-xp-fix-parity branch May 5, 2020 08:52

ycombinator mentioned this pull request May 5, 2020

[7.x] Collect bulk indexing stats for Elasticsearch metricsets (#17992) #18229

Merged

andresrc added the [zube]: Done label May 5, 2020

chrisronline reviewed May 11, 2020

View reviewed changes

ycombinator mentioned this pull request May 11, 2020

Fixing typo in serialized JSON property name #18420

Merged

andresrc removed the [zube]: Done label May 12, 2020

Conversation

ycombinator commented Apr 25, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Why is it important?

Checklist

How to test this PR locally

Related issues

Uh oh!

elasticmachine commented Apr 25, 2020

Uh oh!

elasticmachine commented Apr 25, 2020

Uh oh!

ycombinator commented Apr 25, 2020

Uh oh!

elasticmachine commented Apr 30, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

Uh oh!

jsoriano commented Apr 30, 2020

Uh oh!

ycombinator commented Apr 30, 2020

Uh oh!

igoristic commented Apr 30, 2020

Uh oh!

ycombinator commented Apr 30, 2020

Uh oh!

ycombinator commented May 1, 2020

Uh oh!

igoristic left a comment

Choose a reason for hiding this comment

Uh oh!

chrisronline May 11, 2020

Choose a reason for hiding this comment

Uh oh!

ycombinator May 11, 2020

Choose a reason for hiding this comment

Uh oh!

ycombinator May 11, 2020

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

ycombinator commented Apr 25, 2020 •

edited

Loading

elasticmachine commented Apr 30, 2020 •

edited

Loading