Determine remote cluster version by joshua-adams-1 · Pull Request #142494 · elastic/elasticsearch

joshua-adams-1 · 2026-02-13T16:41:51Z

Extends the reindexing flow to determine the remote cluster version (if applicable), before slicing.

Relates: https://github.com/elastic/elasticsearch-team/issues/2088

joshua-adams-1 · 2026-02-13T16:53:36Z

This is step 4.1 of https://github.com/elastic/elasticsearch-team/issues/2088.

The current reindexing flow is:

Initialise and execute the reindexing task
Slice the request (if applicable). For each slice, send a reindexing REST request and go back to step 1.
Now we're working with one slice only, we open a scroll for that slice, and iterate through each batch of 1000 items, reindexing them into the destination index. If we're reindexing from remote, then for each slice we determine the remote cluster version, and modify our scroll requests to comply with bwc. As a note, if there are N slices, we're doing N calls to the remote cluster to get the same information (which will be refactored later).
We close each scroll
Once all slices are finished, we finish the task

To support PIT we need to:

Open a PIT
Search using the PIT
Close the PIT.

Since one PIT is shared between all slices, the opening of the PIT must be done before we slice (ie a step 1.5 in the above flow). Before we open a PIT we need to decide whether we should. To decide, we need to check whether we're reindexing from a remote cluster. If we are, the remote version must be high enough to support PIT, or else we'll default to scroll. This change adds the logic to determine the remote cluster version. It does not USE the remote version.

A couple of things to note:

This logic already exists, but I am moving it higher by extracting it into a utils class.
Since I am hiding my new code behind a feature flag, there is no additional overhead to remote reindexing requests in the interim, which would be calculating the remote version twice.
Once we use PIT, it is redundant to then redetermine the remote cluster version on every slice, so I shall be removing that code completely. However, to do this, the remote version must be passed from leader to worker task, and I didn't want to obfuscate these changes.

Next Steps:

Now we have the remote version, we can use it inside BulkByScrollParallelizationHelper.executeSlicedAction to open the PIT before slicing
We can extend LeaderBulkByScrollTaskState to store the remote cluster version, and then remove the redundant check from inside RemoteScrollableHitSource

joshua-adams-1 · 2026-02-13T16:59:58Z

modules/reindex/src/main/java/org/elasticsearch/reindex/TransportReindexAction.java

+     * while it is under development.
+     */
+    // TODO - DELETE. Only needed for local development
+    static boolean REINDEX_PIT_SEARCH_ENABLED = new FeatureFlag("reindex_pit_search_enabled").isEnabled();


Added this as a placeholder until #142035 is merged

joshua-adams-1 · 2026-02-13T17:01:13Z

modules/reindex/src/test/java/org/elasticsearch/reindex/remote/RemoteReindexingUtilsTests.java

+     * Verifies that lookupRemoteVersion correctly parses historical and
+     * forward-compatible main action responses.
+     */
+    public void testLookupRemoteVersion() throws Exception {


This, testLookupRemoteVersionFailsWithoutContentType and testWrapExceptionToPreserveStatus were modified from RemoteScrollableHitSourceTests. The rest I added to plug testing gaps

PeteGillinElastic

Thanks, this looks like a good start. I have one comment about a potential change of approach. It may not work, but it might be cleaner if it does. (I have a few tiny nits as well, sorry...)

I haven't looked at the tests, I figured it would be worth converging on the overall approach first.

PeteGillinElastic · 2026-02-16T18:24:09Z

modules/reindex/src/main/java/org/elasticsearch/reindex/remote/RemoteReindexingUtils.java

+public class RemoteReindexingUtils {
+
+    public static void lookupRemoteVersion(RejectAwareActionListener<Version> listener, ThreadPool threadPool, RestClient client) {
+        execute(new Request("GET", ""), MAIN_ACTION_PARSER, listener, threadPool, client);


Micro-nit: I would consider "/" marginally more readable than "". Obviously they're equivalent, but (a) I think that / is the canonical form which the empty path would be normalized into, and (b) I think it's more obvious to the reader what the parameter is if we use "/". (I realize you're just moving this code, but no harm in improving it!)

PeteGillinElastic · 2026-02-16T18:25:38Z

modules/reindex/src/main/java/org/elasticsearch/reindex/remote/RemoteReindexingUtils.java

+        RestClient client
+    ) {
+        // Preserve the thread context so headers survive after the call
+        java.util.function.Supplier<ThreadContext.StoredContext> contextSupplier = threadPool.getThreadContext().newRestorableContext(true);


Nit: We can import j.u.f.Supplier here. (We couldn't in the place you're moving it from because that's importing a different Supplier.)

PeteGillinElastic · 2026-02-16T18:54:18Z

modules/reindex/src/main/java/org/elasticsearch/reindex/Reindexer.java

    private final ScriptService scriptService;
    private final ReindexSslConfig reindexSslConfig;
    private final ReindexMetrics reindexMetrics;
+    Version remoteVersion;


I would prefer not to add this mutable state to Reindexer, if we can avoid it. It makes it a bit harder to reason about, because whenever we're looking at any code we'll have to think carefully to figure out whether this thing will have been initialized yet or not.

Can we instead make lookupRemoteVersion take an ActionListener<Version> instead of ActionListener<Void>, and plumb the thing version through as an additional method parameter to initTask and then BulkByScrollParallelizationHelper.initTaskState, and then through to the LeaderBulkByScrollTaskState or WorkerBulkByScrollTaskState?

I haven't tried that, so I could be wrong, but it feels like it should be possible.

If you end up having to have this mutable state, please can we make it private, and add a getter if needed, so that at least it can only be set from within this class.

Actually, come to think of it: Is there a reason why you trigger the lookupRemoteVersion call in the transport action, as a wrapper around initTask, rather than triggering it inside the implementation of initTask? Again, I haven't tried it — but, if you can do the latter, you're reducing the amount of the internals of Reindexer that you're exposing to the transport action, right?

…hua-adams-1/elasticsearch into reindexing-plumb-pittable-hit-source

joshua-adams-1 · 2026-02-24T11:59:56Z

...index/src/main/java/org/elasticsearch/reindex/remote/RemoteScrollablePaginatedHitSource.java

    protected void doStart(RejectAwareActionListener<Response> searchListener) {
-        lookupRemoteVersion(RejectAwareActionListener.withResponseHandler(searchListener, version -> {
-            remoteVersion = version;
+        if (remoteVersion != null) {


When the feature flag is turned on, the remote version will be non null, and we'll save the cost of the second lookupRemoteVersion call

joshua-adams-1 · 2026-02-24T12:06:23Z

...ndex/src/main/java/org/elasticsearch/reindex/BulkByPaginatedSearchParallelizationHelper.java

            client,
-            listener.delegateFailure((l, v) -> executeSlicedAction(task, request, action, l, client, node, workerAction))
+            listener.delegateFailure(
+                (l, v) -> executeSlicedAction(task, request, action, l, client, node, null, version -> workerAction.run())


This is called exclusively by u-b-q and d-b-q (which has no concept of remote versions since they're executed on the local node). Therefore, setting the remote version parameter to null has no effect. Subsequent changes to introduce PIT will be behind the REINDEX_PIT_SEARCH_ENABLED feature flag anyways, but in the future, once that flag is lifted, this will fail the 'use pit' check, and default to scroll

joshua-adams-1 · 2026-02-24T12:06:48Z

modules/reindex/src/main/java/org/elasticsearch/reindex/Reindexer.java

+                reindexSslConfig,
+                request,
+                wrapWithMetrics(listener, reindexMetrics, startTime, request.getRemoteInfo() != null),
+                remoteVersion


This is unchanged, except I pass the remoteVersion downstream

joshua-adams-1 · 2026-02-24T12:07:57Z

modules/reindex/src/main/java/org/elasticsearch/reindex/Reindexer.java

                sslConfig
            );
            this.destinationIndexIdMapper = destinationIndexMode(state).idFieldMapperWithoutFieldData();
+            this.remoteVersion = remoteVersion;


I store it as a state variable so it can be referenced by buildScrollableResultSource below.

joshua-adams-1 · 2026-02-24T12:09:07Z

modules/reindex/src/main/java/org/elasticsearch/reindex/Reindexer.java

                    remoteInfo,
-                    searchRequest
+                    searchRequest,
+                    remoteVersion


In the case where we've enabled the PIT feature flag, but we're trying to reindex from remote from a remote node with a version < PIT was introduced, then we need to fallback to using scroll. However, since we've already done the remote version lookup, we may as well pass this value in.

If this is a scrollable workflow, then remoteVersion is null.

joshua-adams-1 · 2026-02-24T12:11:43Z

.../src/test/java/org/elasticsearch/reindex/remote/RemoteScrollablePaginatedHitSourceTests.java

+     * Creates a RemoteScrollablePaginatedHitSource with a pre-resolved initial remote version so that doStart skips the version lookup.
+     * The mock client serves only the given response paths (one request = one path when using initial version).
+     */
+    private RemoteScrollablePaginatedHitSource sourceWithInitialRemoteVersion(Version initialRemoteVersion, String... paths)


This is only called by testDoStartSkipsVersionLookupWhenInitialRemoteVersionSet

version

…hua-adams-1/elasticsearch into reindexing-plumb-pittable-hit-source

elasticsearchmachine · 2026-02-24T12:18:48Z

Pinging @elastic/es-distributed (Team:Distributed)

samxbr · 2026-02-24T20:02:46Z

modules/reindex/src/main/java/org/elasticsearch/reindex/Reindexer.java

+        RemoteInfo remoteInfo = request.getRemoteInfo();
+        assert reindexSslConfig != null : "Reindex ssl config must be set";
+        RestClient restClient = buildRestClient(remoteInfo, reindexSslConfig, task.getId(), synchronizedList(new ArrayList<>()));
+        RejectAwareActionListener<Version> rejectAwareListener = new RejectAwareActionListener<>() {


Currently the remote version lookup has retries as part of starting the search. It'd be nice to add retries here too?

Good catch - will add now

modules/reindex/src/main/java/org/elasticsearch/reindex/Reindexer.java

modules/reindex/src/main/java/org/elasticsearch/reindex/remote/RemoteReindexingUtils.java

…hua-adams-1/elasticsearch into reindexing-plumb-pittable-hit-source

samxbr · 2026-02-25T19:11:06Z

modules/reindex/src/main/java/org/elasticsearch/reindex/remote/RemoteReindexingUtils.java

+        try {
+            client.performRequestAsync(request, new ResponseListener() {
+                @Override
+                public void onSuccess(org.elasticsearch.client.Response response) {


Fully qualified name seems unnecessary, just import the class? It's easier to read. Same for other places.

samxbr · 2026-02-25T19:19:54Z

modules/reindex/src/main/java/org/elasticsearch/reindex/Reindexer.java

+            exponentialBackoff(request.getRetryBackoffInitialTime(), request.getMaxRetries()),
+            threadPool,
+            restClient,
+            task.getWorkerState()::countSearchRetry,


I think task.getWorkerState() would throw if the task is not a worker. Leader task would also call lookupRemoteVersionAndExecute, right?

I guess it's fine as long as slicing is not enabled for remote source, but it does seem a bit trappy.

Actually I think we could just remove this runnable, countSearchRetry increments the searchRetries counter which is eventually reported in task response. It seems inaccurate to count the retries for fetching remote version towards search retries? Even though the old behaviour does that, it feels more like unintentional.

samxbr

LGTM, just have some non-blocking minor comments.

Extends the reindexing flow to determine the remote cluster version (if applicable), before slicing. Relates: elastic/elasticsearch-team#2088

…cations * upstream/main: (35 commits) Create ARM bulk sqrI8 implementation (elastic#142461) Rework get-snapshots predicates (elastic#143161) Refactor downsampling fetchers and producers (elastic#140357) ESQL: Unmute test and add extra logging to generative test validation (elastic#143168) Fix metadata fields being nullified/loaded by unmapped_fields setting (elastic#143155) Determine remote cluster version (elastic#142494) Populate failure message for aborted clones (elastic#143206) Allow kibana_system role to read and manage logs streams (elastic#143053) Mute org.elasticsearch.xpack.esql.CsvIT test {csv-spec:eval.DocsLength} elastic#143224 Mute org.elasticsearch.xpack.esql.CsvIT test {csv-spec:eval.DocsByteLength} elastic#143223 Mute org.elasticsearch.xpack.esql.CsvIT test {csv-spec:docs.DocsBitLength} elastic#143222 Fix FloatVectorScorerSupplier bulkScore bug (elastic#143211) ESQL: Add data node execution for external sources (elastic#143209) [ESQL] Cleanup commands docs (elastic#143058) [ML]Fix latest transforms disregarding updates when sort and sync fields are non-monotonic (elastic#142856) Mute org.elasticsearch.index.mapper.IpFieldMapperTests testSyntheticSourceInObject elastic#143212 Tests: Fix StoreDirectoryMetricsIT (elastic#143084) ESQL: Add distribution strategy for external sources (elastic#143194) CSV IT spec (elastic#142585) Fix VectorScorerOSQBenchmark.score to read corrections properly (elastic#143137) ...

Extends the reindexing flow to determine the remote cluster version (if applicable), before slicing. Relates: elastic/elasticsearch-team#2088

joshua-adams-1 added 5 commits February 11, 2026 16:05

Initial plumbing - need to address TODOs

28bc7f9

Creates Reindexer.lookupRemoteVersion

9427763

Refactor RemoteScrollableHitSource to use the utils class

3430f18

Clean up code

66f7deb

Clean up TODOs

a434f0d

joshua-adams-1 self-assigned this Feb 13, 2026

joshua-adams-1 added >non-issue :Distributed/Reindex Issues relating to reindex that are not caused by issues further down labels Feb 13, 2026

elasticsearchmachine added the v9.4.0 label Feb 13, 2026

[CI] Auto commit changes from spotless

7516540

joshua-adams-1 requested review from PeteGillinElastic, samxbr and szybia February 13, 2026 16:54

joshua-adams-1 commented Feb 13, 2026

View reviewed changes

PeteGillinElastic reviewed Feb 16, 2026

View reviewed changes

joshua-adams-1 and others added 10 commits February 23, 2026 14:54

Nits

1739e0f

Move remote lookup into execute

a805729

Add Tests

d5471f0

Merge branch 'main' into reindexing-plumb-pittable-hit-source

b0c22a9

Merge conflicts

6b86337

Merge branch 'reindexing-plumb-pittable-hit-source' of github.com:jos…

61b187a

…hua-adams-1/elasticsearch into reindexing-plumb-pittable-hit-source

[CI] Auto commit changes from spotless

13bdc0b

Clean up

1dd0afc

Merge branch 'reindexing-plumb-pittable-hit-source' of github.com:jos…

9cf2f11

…hua-adams-1/elasticsearch into reindexing-plumb-pittable-hit-source

[CI] Auto commit changes from spotless

460140c

joshua-adams-1 commented Feb 24, 2026

View reviewed changes

joshua-adams-1 added 3 commits February 24, 2026 12:15

Update RemoteScrollablePaginatedHitSourceTests to randomly set remote

5fd8ab0

version

Merge branch 'reindexing-plumb-pittable-hit-source' of github.com:jos…

298f6fb

…hua-adams-1/elasticsearch into reindexing-plumb-pittable-hit-source

Merge branch 'main' into reindexing-plumb-pittable-hit-source

6192133

joshua-adams-1 marked this pull request as ready for review February 24, 2026 12:18

elasticsearchmachine added the Team:Distributed Meta label for distributed team. label Feb 24, 2026

joshua-adams-1 and others added 2 commits February 24, 2026 12:27

Close REST client

ec2517b

[CI] Auto commit changes from spotless

3ba1670

joshua-adams-1 requested a review from PeteGillinElastic February 24, 2026 14:24

samxbr reviewed Feb 24, 2026

View reviewed changes

joshua-adams-1 added 5 commits February 25, 2026 11:22

Add retry logic to remote version lookup

363f2c2

Add retry logic unit tests

2a4e999

Merge branch 'reindexing-plumb-pittable-hit-source' of github.com:jos…

5899ba4

…hua-adams-1/elasticsearch into reindexing-plumb-pittable-hit-source

Merge branch 'main' into reindexing-plumb-pittable-hit-source

04fe410

Merge main

8552f60

samxbr reviewed Feb 25, 2026

View reviewed changes

samxbr approved these changes Feb 25, 2026

View reviewed changes

joshua-adams-1 added 4 commits February 27, 2026 10:04

Fix fully qualified imports

40ed11f

Remove retryCount from lookupRemoteVersionWithRetries

1700728

Add ticket to TODO comment

6f6e96a

Merge branch 'main' into reindexing-plumb-pittable-hit-source

5252944

joshua-adams-1 added the auto-merge-without-approval Automatically merge pull request when CI checks pass (NB doesn't wait for reviews!) label Feb 27, 2026

elasticsearchmachine merged commit 0da8e46 into elastic:main Feb 27, 2026
35 checks passed

joshua-adams-1 deleted the reindexing-plumb-pittable-hit-source branch February 27, 2026 11:35

prwhelan mentioned this pull request Feb 27, 2026

[Transform] Clean up internal tests #143246

Merged

This was referenced Feb 27, 2026

Open and Close PIT for Reindexing #143248

Merged

Reindexing Remote Tests #143461

Closed

Conversation

joshua-adams-1 commented Feb 13, 2026

Uh oh!

joshua-adams-1 commented Feb 13, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

PeteGillinElastic left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joshua-adams-1 Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elasticsearchmachine commented Feb 24, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

samxbr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

joshua-adams-1 Feb 24, 2026 •

edited

Loading