Fix FollowingEngine#lookupPrimaryTerm when sequence numbers are disabled#143935
Merged
tlrx merged 8 commits intoelastic:mainfrom Mar 12, 2026
Merged
Conversation
lookupPrimaryTerm is called by FollowingEngine when a duplicate operation is detected on the primary (i.e., an operation with a seq_no that was already processed). It retrieves the primary term of the existing operation on the follower primary shard so that TransportBulkShardOperationsAction can rewrite the duplicate with the correct term before replicating it to followers replicas, ensuring consistency between primary and replicas. But the method currently only fetch the _seq_no indexed with SeqNoIndexOptions.POINTS_AND_DOC_VALUES option, not DOC_VALUES_ONLY. This is now fixed. Also, if the _seq_no cannot be fetched because it has been merged away on the follower primary, the method now returns `OptionalLong.empty()` as it would for duplicate that are past the global checkpoint. Since _seq_no should be retained for operations below the global checkpoint this is not a production issue, rather a safety net to avoid throwing the IllegalStateException.
tlrx
added
>bug
:Distributed/CCR
Collaborator
|
Pinging @elastic/es-distributed (Team:Distributed) |
Collaborator
|
Hi @tlrx, I've created a changelog YAML for you. |
19 tasks
Contributor
did you meant "beyond" instead of below? Also, this manifests in tests only, right? |
Member
Author
Yes, beyond.
I think it can manifest in production too, throwing an illegal state exception on time-series/logsdb follower primary shard and therefore failing the engine. |
Contributor
But in that case we shouldn't be pruning sequence numbers beyond the global checkpoint? I think that I'm missing something here |
tlrx
added
the
auto-merge-without-approval
Member
Author
|
Thanks Francisco |
michalborek
pushed a commit
to michalborek/elasticsearch
that referenced
this pull request
Mar 23, 2026
…led (elastic#143935) lookupPrimaryTerm is called by FollowingEngine when a duplicate operation is detected on the primary (i.e., an operation with a seq_no that was already processed). It retrieves the primary term of the existing operation on the follower primary shard so that TransportBulkShardOperationsAction can rewrite the duplicate with the correct term before replicating it to followers replicas, ensuring consistency between primary and replicas. But the method currently only fetch the _seq_no indexed with SeqNoIndexOptions.POINTS_AND_DOC_VALUES option, not DOC_VALUES_ONLY. This is now fixed. Also, if the _seq_no cannot be fetched because it has been merged away on the follower primary, the method now returns OptionalLong.empty() as it would for duplicate that are past the global checkpoint. Since _seq_no should be retained for operations beyond the global checkpoint this is not a production issue, rather a safety net to avoid throwing the IllegalStateException. The only test we have is FollowEnginTests.testProcessOnPrimary but making it work for the DOC_VALUES_ONLY is already a pain, even more with DISABLE_SEQUENCE_NUMBERS.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
lookupPrimaryTerm is called by FollowingEngine when a duplicate operation is detected on the primary (i.e., an operation with a seq_no that was already processed). It retrieves the primary term of the existing operation on the follower primary shard so that TransportBulkShardOperationsAction can rewrite the duplicate with the correct term before replicating it to followers replicas, ensuring consistency between primary and replicas.
But the method currently only fetch the _seq_no indexed with SeqNoIndexOptions.POINTS_AND_DOC_VALUES option, not DOC_VALUES_ONLY.
This is now fixed.
Also, if the _seq_no cannot be fetched because it has been merged away on the follower primary, the method now returns
OptionalLong.empty()as it would for duplicate that are past the global checkpoint. Since _seq_no should be retained for operations beyond the global checkpoint this is not a production issue, rather a safety net to avoid throwing the IllegalStateException.The only test we have is FollowEnginTests.testProcessOnPrimary but making it work for the DOC_VALUES_ONLY is already a pain, even more with DISABLE_SEQUENCE_NUMBERS.