Expose shard changes action as a rest api#118608
Expose shard changes action as a rest api#118608salvatore-campagna merged 15 commits intoelastic:mainfrom
Conversation
elasticsearchmachine
added
the
needs:triage
| // forget follower API | ||
| new RestForgetFollowerAction() | ||
| ); | ||
| return Stream.concat( |
There was a problem hiding this comment.
Arrays.asList returns a fixed-size list. As a result, adding later the additional handler based on the value of Build.current().isSnapshot() might fail with an UnsupportedOperationException.
There was a problem hiding this comment.
Maybe just create an array list with an if statement for the shard changes action? Looks simpler to me than all this stream stuff.
salvatore-campagna
added
>non-issue
:StorageEngine/Logs
|
Pinging @elastic/es-storage-engine (Team:StorageEngine) |
| builder.field("number_of_operations", response.getOperations().length); | ||
| builder.field("operations"); | ||
| builder.startArray(); | ||
| for (final Translog.Operation operation : response.getOperations()) { |
There was a problem hiding this comment.
Maybe we can avoid this...as serializing each operation for large requests might be way too much?
There was a problem hiding this comment.
I see that you already left out the actual content of the write operations, which is good for the use case this rest api is going to be used. Let's for now keep the response as is here. If this turns out the be problematic we can always drop the operations array from the response.
| builder.field("number_of_operations", response.getOperations().length); | ||
| builder.field("operations"); | ||
| builder.startArray(); | ||
| for (final Translog.Operation operation : response.getOperations()) { |
There was a problem hiding this comment.
I see that you already left out the actual content of the write operations, which is good for the use case this rest api is going to be used. Let's for now keep the response as is here. If this turns out the be problematic we can always drop the operations array from the response.
| // forget follower API | ||
| new RestForgetFollowerAction() | ||
| ); | ||
| return Stream.concat( |
There was a problem hiding this comment.
Maybe just create an array list with an if statement for the shard changes action? Looks simpler to me than all this stream stuff.
| import static org.elasticsearch.rest.RestRequest.Method.GET; | ||
|
|
||
| /** | ||
| * A REST handler that retrieves shard changes in a specific index whose name is provided as a parameter. |
There was a problem hiding this comment.
Maybe at the top of the java docs, mention that this rest action is only loaded in snapshot builds and its use is for benchmarking purposes only?
There was a problem hiding this comment.
I added a note below in the Javadoc.
| assertOK(client().performRequest(bulkRequest(indexName, randomIntBetween(10, 20)))); | ||
|
|
||
| final Request shardChangesRequest = new Request("GET", shardChangesEndpoint(indexName)); | ||
| assertOK(client().performRequest(shardChangesRequest)); |
…#722) Benchmarking the new API introduced in PR elastic/elasticsearch#118608 to evaluate synthetic source recovery latency. The benchmark will target indices or data streams in the `elastic/logs`, `elastic/security`, and `tsdb` tracks. Key notes: - A retention lease must be added before indexing to prevent missing `seq_no` values during shard recovery. - The API in PR elastic/elasticsearch#118937 supports data streams and index aliases, allowing flexibility in targeting indices, especially for data stream use cases.
3 participants
This PR adds a new REST API
/{index}/ccr/shard_changesto retrieve shard-level changes, including translog operations, mapping versions, and sequence numbers. It is required for a new set of Rally benchmarks, which will be used to evaluate the impact of synthetic source on recovery time. The API accepts parameters likefrom_seq_no,max_batch_size,poll_timeout, andmax_operations_countand is exposed only in snapshot builds.