Skip to content

Remove writing redundant binary doc value copy of jVector based vectors. - #715

Merged
akash-shankaran merged 4 commits into
opensearch-project:mainfrom
akash-shankaran:skip-redundant-docvalues-jvector
Sep 6, 2026
Merged

akash-shankaran merged 4 commits into
opensearch-project:mainfrom
akash-shankaran:skip-redundant-docvalues-jvector

Conversation

@akash-shankaran

@akash-shankaran akash-shankaran commented Aug 29, 2026

Copy link
Copy Markdown
Collaborator

Description

Presently jvector writes 2 copies of vector files:

  1. Store the vectors inline (which end up in .data-jvector files)
  2. vectorField with DocValuesType.BINARY (which end up in .dvd files)

The .dvd copy is never read by jVector, as all its readers rely on being served from the inline vector graph.
This leads to higher storage and added overhead on index build, and merge times.

Given modifying doc values of existing indices could lead to issues, this change is safe for newer indices created in v3.9.0.0 onwards.

Advantages:

  1. This should result in a significant storage improvement, and bring the jvector-plugin on parity with other opensearch based engines.
  2. Side benefits: Reduce merge times, and index build times.

Experimental Data:
Measured on sift-100k (128d) PQ, 1 shard force-merged to 1 segment:
index size goes from 125.18 MB to 76.50 MB (-38.9%), with data-jvector file being byte-identical and no impact on recall@k and recall@1. The removed ~50MB bytes is one fp32 copy (100000 x 128 x 4).

Related Issues

Resolves #716

Check List

  • New functionality includes testing.
  • New functionality has been documented.
  • API changes companion pull request created.
  • Commits are signed per the DCO using --signoff.
  • Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Akash Shankaran <akash.shankaran1@gmail.com>
Signed-off-by: Akash Shankaran <akash.shankaran1@gmail.com>
Comment thread src/main/java/org/opensearch/knn/index/mapper/LuceneFieldMapper.java Outdated
Signed-off-by: Akash Shankaran <akash.shankaran1@gmail.com>
Comment thread src/main/java/org/opensearch/knn/index/mapper/LuceneFieldMapper.java Outdated
Signed-off-by: Akash Shankaran <akash.shankaran1@gmail.com>
@akash-shankaran
akash-shankaran merged commit 1faf485 into opensearch-project:main Sep 6, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEATURE] Fix redundant Vector data writing

2 participants