Skip to content

[DiskBBQ] Fix index sorting on flush#144938

Merged
iverase merged 9 commits intoelastic:mainfrom
iverase:dislbbq_flush_sort
Mar 26, 2026
Merged

[DiskBBQ] Fix index sorting on flush#144938
iverase merged 9 commits intoelastic:mainfrom
iverase:dislbbq_flush_sort

Conversation

@iverase
Copy link
Copy Markdown
Contributor

@iverase iverase commented Mar 25, 2026

We are currently not taking into account the provided Sorter.DocMap on flush which means we might be writing vectors with wrong docIds during flush. Those vectors get properly assigned in merge but flushed segments are wrong.

@elasticsearchmachine elasticsearchmachine added the Team:Search Relevance Meta label for the Search Relevance team in Elasticsearch label Mar 25, 2026
@elasticsearchmachine
Copy link
Copy Markdown
Collaborator

Pinging @elastic/es-search-relevance (Team:Search Relevance)

@iverase iverase added the >bug label Mar 25, 2026
Copy link
Copy Markdown
Contributor

@tteofili tteofili left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice spot, thx @iverase !

@elasticsearchmachine
Copy link
Copy Markdown
Collaborator

Hi @iverase, I've created a changelog YAML for you.

@iverase iverase added the auto-backport Automatically create backport pull requests when merged label Mar 25, 2026
Copy link
Copy Markdown
Contributor

@john-wagster john-wagster left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@iverase iverase merged commit 2d9ba47 into elastic:main Mar 26, 2026
36 checks passed
@iverase iverase deleted the dislbbq_flush_sort branch March 26, 2026 07:48
@elasticsearchmachine
Copy link
Copy Markdown
Collaborator

💔 Backport failed

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 144938

szybia added a commit to szybia/elasticsearch that referenced this pull request Mar 26, 2026
* upstream/main: (146 commits)
  Revert "[Native] Gradle-related tweaks to improve handling of the simdvec native library (elastic#144539)"
  Fix ArrayIndexOutOfBoundsException in fetch phase with partial results (elastic#144385)
  ESQL: Correctly manage NULL data type for SUM (elastic#144942)
  [ESQL] Fixes GroupedTopNBenchmark not executing (elastic#144944)
  Fix reader context leak when query response serialization fails (elastic#144708)
  Validate individual offset values in BULK_OFFSETS bounds checks (elastic#144643)
  Merge main21 source set into main in simdvec (elastic#144921)
  [TEST] Unmute TsidExtractingIdFieldMapperTests (elastic#144848)
  [Native] Gradle-related tweaks to improve handling of the simdvec native library (elastic#144539)
  Fix `ThreadedActionListenerTests#testRejectionHandling` (elastic#144795)
  Add new DLM Frozen Tier Transition execution plugin and service (elastic#144595)
  Prometheus: execute query_range via parsed EsqlStatement plan (elastic#144416)
  Investigate `testBulkIndexingRequestSplitting` failure (elastic#144766)
  Add test utility for wrapping directories in FilterDirectory layer (elastic#143563)
  Fix ES|QL decay tests with negative scale (elastic#144657)
  Fix circuit breaker leak in percolator query construction (elastic#144827)
  Use XPerFieldDocValuesFormat in AbstractTSDBSyntheticIdCodec (elastic#144744)
  [DOCS] Document how reindex work in CPS (elastic#144016)
  Fix Int4 vector library tests failing on Java 21 (elastic#144830)
  [DiskBBQ] Fix index sorting on flush (elastic#144938)
  ...
sidosera pushed a commit to sidosera/elasticsearch that referenced this pull request Mar 26, 2026
seanzatzdev pushed a commit to seanzatzdev/elasticsearch that referenced this pull request Mar 26, 2026
We are currently not taking into account the provided Sorter.DocMap on flush which means we might be writing 
vectors with wrong docIds during flush. Those vectors get properly assigned in merge but flushed segments are 
wrong.
seanzatzdev pushed a commit to seanzatzdev/elasticsearch that referenced this pull request Mar 26, 2026
seanzatzdev pushed a commit to seanzatzdev/elasticsearch that referenced this pull request Mar 27, 2026
We are currently not taking into account the provided Sorter.DocMap on flush which means we might be writing 
vectors with wrong docIds during flush. Those vectors get properly assigned in merge but flushed segments are 
wrong.
seanzatzdev pushed a commit to seanzatzdev/elasticsearch that referenced this pull request Mar 27, 2026
mamazzol pushed a commit to mamazzol/elasticsearch that referenced this pull request Mar 30, 2026
We are currently not taking into account the provided Sorter.DocMap on flush which means we might be writing 
vectors with wrong docIds during flush. Those vectors get properly assigned in merge but flushed segments are 
wrong.
mamazzol pushed a commit to mamazzol/elasticsearch that referenced this pull request Mar 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auto-backport Automatically create backport pull requests when merged backport pending >bug :Search Relevance/Vectors Vector search Team:Search Relevance Meta label for the Search Relevance team in Elasticsearch v9.2.8 v9.3.3 v9.4.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants