Use binary serde for resource usage headers - #21230
Conversation
PR Reviewer Guide 🔍(Review updated until commit 2e2537a)Here are some key observations to aid the review process:
|
PR Code Suggestions ✨Latest suggestions up to 2e2537a Explore these optional code suggestions:
Previous suggestionsSuggestions up to commit f416371
Suggestions up to commit 337f418
Suggestions up to commit 1ed369e
Suggestions up to commit b629cff
Suggestions up to commit 73ab078
|
|
❌ Gradle check result for 73ab078: TIMEOUT Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
73ab078 to
b629cff
Compare
PR Code Analyzer ❗AI-powered 'Code-Diff-Analyzer' found issues on commit 7d57080.
The table above displays the top 10 most important findings. Pull Requests Author(s): Please update your Pull Request according to the report above. Repository Maintainer(s): You can Thanks. |
|
Persistent review updated to latest commit b629cff |
|
❌ Gradle check result for b629cff: TIMEOUT Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
b629cff to
7d57080
Compare
ansjcy
left a comment
There was a problem hiding this comment.
LGTM, the numbers are pretty impressive. 2 caveats:
- we need to accept the resource usgae data loss during rolling upgrade
- Please add change log entry to reflect this.
|
Persistent review updated to latest commit 15a1691 |
jainankitk
left a comment
There was a problem hiding this comment.
@kaushalmahi12 - What do you think about this change? Mostly looks good to me.
|
❌ Gradle check result for 15a1691: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
@dzane17 I'm of the opinion that we should have a flag just incase something breaks and allow a rollback to the old format of the usage headers. |
15a1691 to
1ed369e
Compare
|
Persistent review updated to latest commit 1ed369e |
|
Persistent review updated to latest commit 337f418 |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #21230 +/- ##
============================================
+ Coverage 73.43% 73.53% +0.09%
- Complexity 75103 75144 +41
============================================
Files 6016 6016
Lines 341072 341105 +33
Branches 49091 49095 +4
============================================
+ Hits 250469 250834 +365
+ Misses 70682 70281 -401
- Partials 19921 19990 +69 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
|
Thanks, @dzane17 @kaushalmahi12 could you take a look at the PR? |
Signed-off-by: David Zane <davizane@amazon.com>
Signed-off-by: David Zane <davizane@amazon.com>
Signed-off-by: David Zane <davizane@amazon.com>
Signed-off-by: David Zane <davizane@amazon.com>
337f418 to
f416371
Compare
|
Persistent review updated to latest commit f416371 |
|
❌ Gradle check result for f416371: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
|
❌ Gradle check result for f416371: TIMEOUT Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
Signed-off-by: David Zane <davizane@amazon.com>
|
Persistent review updated to latest commit 2e2537a |
Description
Replaces JSON serialization/deserialization of
TaskResourceInfoin theTASK_RESOURCE_USAGEresponse header with Base64-encoded binary using the existingWriteableinterface. This reduces CPU overhead on the search path where the resource usage header is parsed for every shard.Problem
The
TASK_RESOURCE_USAGEheader carries per-shard resource usage data (CPU, memory) from data nodes to the coordinator. The current implementation serializes viatoXContent(JSON) and deserializes viaXContentParser(Jackson) on every shard response. Since a single search request can hit thousands of shards, this serde cost is multiplied accordingly. Profiling has shown ~7% CPU overhead from this serde.Solution
Serialize
TaskResourceInfousingwriteTo/readFromStream(binary), Base64-encode the result into a string. Base64 encoding is necessary becauseThreadContextresponse headers only support string values. Even ifThreadContextsupported rawbyte[]headers, benchmarks show the additional speedup is marginal (see data below), making the Base64 approach a good tradeoff that avoids modifying the publicThreadContextclass.Backwards compatibility
This change is fully BWC across a rolling upgrade.
V_3_7_0or later (resolved from the parent task ID via cluster state), otherwise serialize using the original JSON format.Rolling upgrade verification: Verified end-to-end with a 3-node cluster running released
3.6.1, upgrading one node at a time to this PR (3.7.0). Search workload driven against all three nodes at every phase: 852 searches total, 0 failures, no hangs, no stuck tasks, no parse errors on any node.Benchmark Results (JMH, per 1000 shards)
The end-to-end round-trip (serialize on data node + deserialize on coordinator) is 5.9x faster with Binary+Base64 compared to JSON. Raw binary column is included to show the theoretical outcome if
ThreadContextsupportedbyte[]headers natively.Each test result ran:
One iteration means JMH ran the target operation continuously for 10 seconds, then divided the total time by the number of completed operations to get a time per operation. Each operation performs 1000 TaskResourceInfo serde cycles — one per shard — to simulate a large search request hitting 1000 shards. The final values in the table are the average ns/op across all 3 measurement iterations.
Header Size Reduction
Based on a typical
TaskResourceInfo:{ "action": "indices:data/read/search[phase/query]", "taskId": 1234567, "parentTaskId": 1234566, "nodeId": "U0rMsZg9RGOxdnVfMtMNVA", "taskResourceUsage": { "cpuTimeInNanos": 15000000, "memoryInBytes": 2048000 } }Binary+Base64 reduces bytes sent over the wire by ~31% compared to JSON.
Related Issues
Resolves #17407
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.