feat(wlm): Add additional search settings and override_request_values to WLM groups - #21523
Conversation
PR Reviewer Guide 🔍(Review updated until commit 6a38df0)Here are some key observations to aid the review process:
|
PR Code Suggestions ✨Latest suggestions up to 6a38df0 Explore these optional code suggestions:
Previous suggestionsSuggestions up to commit 5bdf642
Suggestions up to commit b01f234
Suggestions up to commit 368789a
Suggestions up to commit 8e64571
Suggestions up to commit 42afadf
|
|
❌ Gradle check result for 9a8c5ae: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
|
Persistent review updated to latest commit 42afadf |
|
❌ Gradle check result for 42afadf: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
|
Persistent review updated to latest commit 8e64571 |
|
❌ Gradle check result for 8e64571: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
|
Persistent review updated to latest commit 368789a |
|
❌ Gradle check result for 368789a: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
cwperks
left a comment
There was a problem hiding this comment.
Thank you @dzane17. Overall the change looks good to me, but is it possible to add any integ tests to demonstrate that the overrides are taking when a request is explicitly given values but maps to a workload group that specifies to override?
idk we can can do anything contrived and cancel a request after a few seconds even if the timeout passed to a request is 1m?
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #21523 +/- ##
============================================
- Coverage 73.48% 73.43% -0.05%
+ Complexity 74736 74701 -35
============================================
Files 5983 5983
Lines 339062 339120 +58
Branches 48882 48895 +13
============================================
- Hits 249162 249042 -120
- Misses 70064 70289 +225
+ Partials 19836 19789 -47 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
|
@cwperks A true end-to-end functional test is tricky. The integ test Override behavior is already thoroughly covered in unit tests |
|
Persistent review updated to latest commit b01f234 |
|
❌ Gradle check result for b01f234: null Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
|
❌ Gradle check result for b01f234: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
… to workload groups Signed-off-by: David Zane <davizane@amazon.com>
Signed-off-by: David Zane <davizane@amazon.com>
|
Persistent review updated to latest commit 5bdf642 |
|
Persistent review updated to latest commit 6a38df0 |
Description
Adds additional WLM search settings and introduces
override_request_valuesto control how workload group settings interact with request-level parameters.New search settings
Workload groups can now configure the following settings in addition to the existing
search.default_search_timeout:search.cancel_after_time_interval— time after which a search request is cancelledsearch.max_concurrent_shard_requests— max concurrent shard requests per node (>= 1)search.batched_reduce_size— number of shard results to reduce at once (>= 2)override_request_values— boolean (defaultfalse) controlling whether WLM settings override explicitly set request-level valuesWhy
override_request_valuesis neededOpenSearch convention is that nothing should override request-level parameters — they represent the caller's explicit intent. However, WLM groups are configured by system administrators to protect the cluster. There should be a mechanism for WLM group settings to take precedence over a potentially malicious or uninformed individual who sets aggressive request parameters (e.g., unbounded timeouts or excessive concurrency).
At the same time, WLM group settings should not immediately override request values by default because that can cause unpredictable response behavior which breaks existing clients. Therefore
override_request_valuesdefaults tofalse— admins must explicitly opt in to the override behavior after understanding the impact on their workloads.Settings update semantics
Settings use merge semantics on update:
"settings": {}or"settings": null— clears all search settings and setsoverride_request_valuesto false"settings": {"key": "value"}— merges with existing (adds/updates the key)"settings": {"key": null}— removes that specific key from existing settingsThe
override_request_valuesfield is always present in the GET response (defaults to"false"when not explicitly set).Related Issues
Part of #20555
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.