Add composite-engine stats and parquet ingest rejection/pool metrics - #22134
Conversation
PR Reviewer Guide 🔍(Review updated until commit 4c52a4c)Here are some key observations to aid the review process:
|
PR Code Suggestions ✨Latest suggestions up to 4c52a4c Explore these optional code suggestions:
Previous suggestionsSuggestions up to commit 4c52a4c
Suggestions up to commit 8c07c42
Suggestions up to commit e456288
Suggestions up to commit 456a043
Suggestions up to commit 2966865
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #22134 +/- ##
============================================
- Coverage 73.43% 73.31% -0.12%
+ Complexity 75926 75827 -99
============================================
Files 6070 6070
Lines 344613 344613
Branches 49579 49579
============================================
- Hits 253057 252653 -404
- Misses 71400 71807 +407
+ Partials 20156 20153 -3 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
2966865 to
456a043
Compare
|
Persistent review updated to latest commit 456a043 |
|
❌ Gradle check result for 456a043: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
c3ae008 to
e456288
Compare
|
Persistent review updated to latest commit e456288 |
|
❌ Gradle check result for e456288: null Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
e456288 to
8c07c42
Compare
|
Persistent review updated to latest commit 8c07c42 |
Signed-off-by: Kamal Nayan <askkamal@amazon.com>
8c07c42 to
4c52a4c
Compare
|
Persistent review updated to latest commit 4c52a4c |
|
❌ Gradle check result for 4c52a4c: null Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
|
Persistent review updated to latest commit 4c52a4c |
…pensearch-project#22134) Signed-off-by: Kamal Nayan <askkamal@amazon.com> Co-authored-by: Kamal Nayan <askkamal@amazon.com>
…pensearch-project#22134) Signed-off-by: Kamal Nayan <askkamal@amazon.com> Co-authored-by: Kamal Nayan <askkamal@amazon.com>
Description
1. Composite-engine stats provider (new)
Exposes per-format stats for the composite engine via
GET /_plugins/composite/{index}/_statsandGET /_plugins/composite/_nodes/_stats.refreshrefresh_total,refresh_time_millisrefresh(merge-on-refresh breakdown)refresh_merge_total,refresh_merge_time_millismergemerge_total,merge_time_millis,merge_failureswritewrite_primary_failures,write_secondary_failuresmappingmapping_update_executed_total(counts only actually-applied dynamic mapping updates)2. Parquet
native_write_rejections+ bounded write poolnative_write.native_write_rejections, incremented when theparquet_native_writethread pool rejects a background write.3. Parquet node-level ingest pool stats
Adds a
native_ingest_poolblock to parquet per-node stats reporting the liveparquet_native_writepool:threads,queue_depth,active,rejected,largest_queue,completed— giving visibility into ingest backpressure before rejections occur.Notes
Timing is recorded via the shared
StatsRecorderutility. There are no changes to the indexing/refresh/merge/flush execution flow beyond the intentional pool bound — all instrumentation is side-effect-free counter emission.Testing
precommitclean on both pluginsCompositeShardStatsTests+ParquetIngestPoolStatsTests)CompositeStatsEndpointIT)Related Issues
Resolves #[Issue number to be closed when this PR is merged]
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.