Add base64 encoded vector indexing support for knn_vector fields

[navneet1v]

Description

Vectors can now be indexed as base64-encoded strings in addition to JSON arrays. Float vectors use little-endian byte encoding (symmetric with the doc_values binary output format), while byte/binary vectors use raw byte encoding. This enables efficient bulk ingestion pipelines that avoid JSON array serialization overhead.

Some Benchmarking details added in the GH issue. #3322

Related Issues

Resolves #3322

Check List

• New functionality includes testing.
• New functionality has been documented.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

PR Reviewer Guide 🔍

(Review updated until commit 7ca2011)

Here are some key observations to aid the review process:

🧪 PR contains tests

🔒 No security concerns identified

✅ No TODO sections

🔀 No multiple PR themes

⚡ Recommended focus areas for review Possible Issue For float vectors, per-dimension processing and validation occur after decoding the entire base64 string. If perDimensionProcessor.process() or perDimensionValidator.validate() throw exceptions for any dimension, the entire operation fails after already allocating the full float array. This differs from the array parsing path where validation happens incrementally per element. For large vectors with invalid values late in the sequence, this wastes memory and processing time. ByteBuffer.wrap(decoded).order(ByteOrder.LITTLE_ENDIAN).asFloatBuffer().get(array); for (int idx = 0; idx < numFloats; idx++) { array[idx] = perDimensionProcessor.process(array[idx]); perDimensionValidator.validate(array[idx]); } return Optional.of(array);

[github-actions]

PR Code Suggestions ✨

Latest suggestions up to 7ca2011

Explore these optional code suggestions:

Category	Suggestion	Impact
Possible issue	Fix binary vector dimension validation The validateVectorDimension call for byte vectors uses decoded.length directly, but for binary vectors it should validate against decoded.length * 8 since binary dimension is measured in bits. This causes incorrect dimension validation for binary data types when using base64 encoding. src/main/java/org/opensearch/knn/index/mapper/KNNVectorFieldMapper.java [839-853] } else if (token == XContentParser.Token.VALUE_STRING) { final byte[] decoded; try { decoded = BASE64_DECODER.decode(context.parser().text()); } catch (IllegalArgumentException e) { throw new IllegalArgumentException( String.format(Locale.ROOT, "Invalid base64 encoding for vector field [%s]: %s", name(), e.getMessage()), e ); } - validateVectorDimension(dimension, decoded.length, dataType); + int actualDimension = (dataType == VectorDataType.BINARY) ? decoded.length * 8 : decoded.length; + validateVectorDimension(dimension, actualDimension, dataType); // Per-dimension validation is intentionally skipped: base64-decoded bytes are inherently // in [-128, 127] range, cannot be NaN/Inf, and have no decimal component — all checks // that validateByte() performs are satisfied by construction for raw byte values. return Optional.of(decoded); Suggestion importance[1-10]: 10 __ Why: This is a critical bug. The code validates decoded.length against dimension for both BYTE and BINARY types, but BINARY dimension is in bits while decoded.length is in bytes. This causes incorrect validation for binary vectors, allowing wrong dimensions to pass or rejecting correct ones.	High
General	Ensure atomic per-dimension processing The per-dimension processing and validation modifies the array in-place after decoding. If perDimensionProcessor.process() throws an exception partway through, the array will be left in a partially modified state, potentially causing inconsistent data. src/main/java/org/opensearch/knn/index/mapper/KNNVectorFieldMapper.java [910-918] int numFloats = decoded.length / Float.BYTES; validateVectorDimension(dimension, numFloats, vectorDataType); final float[] array = new float[numFloats]; ByteBuffer.wrap(decoded).order(ByteOrder.LITTLE_ENDIAN).asFloatBuffer().get(array); for (int idx = 0; idx < numFloats; idx++) { - array[idx] = perDimensionProcessor.process(array[idx]); - perDimensionValidator.validate(array[idx]); + float processed = perDimensionProcessor.process(array[idx]); + perDimensionValidator.validate(processed); + array[idx] = processed; } Suggestion importance[1-10]: 4 __ Why: While the suggestion improves code clarity by avoiding in-place modification before validation, the practical impact is minimal. If perDimensionProcessor.process() or perDimensionValidator.validate() throws an exception, the entire parsing operation fails and the partially modified array is discarded anyway.	Low

---

Previous suggestions

Suggestions up to commit eb230f5

Category	Suggestion	Impact
Possible issue	Apply per-dimension validation to byte vectors The byte vector path directly returns the decoded bytes without applying perDimensionProcessor or perDimensionValidator to each element, unlike the array parsing path. This inconsistency means base64-encoded byte vectors bypass validation and processing that array-encoded vectors receive, potentially allowing invalid values through. src/main/java/org/opensearch/knn/index/mapper/KNNVectorFieldMapper.java [839-850] } else if (token == XContentParser.Token.VALUE_STRING) { final byte[] decoded; try { decoded = BASE64_DECODER.decode(context.parser().text()); } catch (IllegalArgumentException e) { throw new IllegalArgumentException( String.format(Locale.ROOT, "Invalid base64 encoding for vector field [%s]: %s", name(), e.getMessage()), e ); } validateVectorDimension(dimension, decoded.length, dataType); + for (int i = 0; i < decoded.length; i++) { + float value = perDimensionProcessor.processByte(decoded[i]); + perDimensionValidator.validateByte(value); + decoded[i] = (byte) value; + } return Optional.of(decoded); } Suggestion importance[1-10]: 8 __ Why: The suggestion correctly identifies that base64-encoded byte vectors bypass perDimensionProcessor and perDimensionValidator that are applied in the array parsing path (lines 831-838). This inconsistency could allow invalid values through, making this a significant correctness issue.	Medium

Suggestions up to commit 314aae6

Category	Suggestion	Impact
General	Apply per-dimension processing to bytes The getBytesFromContext method returns raw decoded bytes without applying perDimensionProcessor or perDimensionValidator to each byte element. For consistency with the array parsing path (which processes each byte), consider validating and processing each decoded byte individually to ensure uniform behavior across input formats. src/main/java/org/opensearch/knn/index/mapper/KNNVectorFieldMapper.java [839-850] } else if (token == XContentParser.Token.VALUE_STRING) { final byte[] decoded; try { decoded = BASE64_DECODER.decode(context.parser().text()); } catch (IllegalArgumentException e) { throw new IllegalArgumentException( String.format(Locale.ROOT, "Invalid base64 encoding for vector field [%s]: %s", name(), e.getMessage()), e ); } validateVectorDimension(dimension, decoded.length, dataType); + for (int i = 0; i < decoded.length; i++) { + float value = perDimensionProcessor.processByte(decoded[i]); + perDimensionValidator.validateByte(value); + decoded[i] = (byte) value; + } return Optional.of(decoded); Suggestion importance[1-10]: 7 __ Why: The suggestion correctly identifies that the base64 path for bytes skips perDimensionProcessor and perDimensionValidator, while the array path applies them. This inconsistency could lead to different behavior between input formats. However, the impact depends on whether these processors/validators are actually configured for byte types in practice.	Medium

Suggestions up to commit bf699a5

Category	Suggestion	Impact
General	Add dimension context to validation errors If perDimensionValidator.validate() throws an exception for any element, the partially populated array is abandoned but the exception doesn't indicate which dimension failed. Consider wrapping validation exceptions with the dimension index to aid debugging of malformed base64 vectors. src/main/java/org/opensearch/knn/index/mapper/KNNVectorFieldMapper.java [907-914] int numFloats = decoded.length / Float.BYTES; validateVectorDimension(dimension, numFloats, vectorDataType); final ByteBuffer buffer = ByteBuffer.wrap(decoded).order(ByteOrder.LITTLE_ENDIAN); final float[] array = new float[numFloats]; for (int idx = 0; idx < numFloats; idx++) { array[idx] = perDimensionProcessor.process(buffer.getFloat()); - perDimensionValidator.validate(array[idx]); + try { + perDimensionValidator.validate(array[idx]); + } catch (IllegalArgumentException e) { + throw new IllegalArgumentException( + String.format(Locale.ROOT, "Invalid value at dimension %d in base64 vector for field [%s]: %s", idx, name(), e.getMessage()), + e + ); + } } Suggestion importance[1-10]: 5 __ Why: This suggestion improves error messages by adding the dimension index when perDimensionValidator.validate() fails, which would help users identify which specific value in the base64-encoded vector is invalid. However, the impact is moderate as it only enhances debugging information rather than fixing a functional issue.	Low
	Validate non-empty decoded byte array The base64 decoding for byte vectors doesn't validate that the decoded byte array is non-empty before returning. An empty base64 string would decode to an empty byte array, bypassing dimension validation if dimension is 0. Add an explicit check to ensure the decoded array has at least one element. src/main/java/org/opensearch/knn/index/mapper/KNNVectorFieldMapper.java [839-850] } else if (token == XContentParser.Token.VALUE_STRING) { final byte[] decoded; try { decoded = BASE64_DECODER.decode(context.parser().text()); } catch (IllegalArgumentException e) { throw new IllegalArgumentException( String.format(Locale.ROOT, "Invalid base64 encoding for vector field [%s]: %s", name(), e.getMessage()), e ); } + if (decoded.length == 0) { + throw new IllegalArgumentException( + String.format(Locale.ROOT, "Base64 encoded vector for field [%s] cannot be empty", name()) + ); + } validateVectorDimension(dimension, decoded.length, dataType); return Optional.of(decoded); } Suggestion importance[1-10]: 3 __ Why: While the suggestion correctly identifies that an empty base64 string could decode to an empty array, the validateVectorDimension method at line 849 should already handle this case by checking if decoded.length matches the expected dimension. The additional check is redundant unless dimension is 0, which would be a configuration issue rather than a runtime validation concern.	Low

Suggestions up to commit 0f2273e

Category	Suggestion	Impact
Possible issue	Advance parser before early return The getBytesFromContext method returns early when processing base64 strings without calling context.parser().nextToken(). This inconsistency with other branches may cause parser state issues. Ensure the parser advances to the next token before returning to maintain consistent parser state across all code paths. src/main/java/org/opensearch/knn/index/mapper/KNNVectorFieldMapper.java [839-850] } else if (token == XContentParser.Token.VALUE_STRING) { final byte[] decoded; try { decoded = BASE64_DECODER.decode(context.parser().text()); } catch (IllegalArgumentException e) { throw new IllegalArgumentException( String.format(Locale.ROOT, "Invalid base64 encoding for vector field [%s]: %s", name(), e.getMessage()), e ); } validateVectorDimension(dimension, decoded.length, dataType); + context.parser().nextToken(); return Optional.of(decoded); } Suggestion importance[1-10]: 9 __ Why: The suggestion correctly identifies a critical parser state inconsistency. The VALUE_STRING branch returns early without calling context.parser().nextToken(), while other branches (lines 832, 838) do advance the parser. This can cause parser state corruption and subsequent parsing errors.	High
General	Validate before storing in array Processing and validation occur after reading from the buffer, which means invalid values are stored in the array before validation. If validation fails, the array contains invalid data. Consider validating immediately after processing each float value to fail fast and avoid storing invalid data. src/main/java/org/opensearch/knn/index/mapper/KNNVectorFieldMapper.java [909-914] final ByteBuffer buffer = ByteBuffer.wrap(decoded).order(ByteOrder.LITTLE_ENDIAN); final float[] array = new float[numFloats]; for (int idx = 0; idx < numFloats; idx++) { - array[idx] = perDimensionProcessor.process(buffer.getFloat()); - perDimensionValidator.validate(array[idx]); + float value = perDimensionProcessor.process(buffer.getFloat()); + perDimensionValidator.validate(value); + array[idx] = value; } Suggestion importance[1-10]: 3 __ Why: While the suggestion is technically correct about the order of operations, the practical impact is minimal. The validation will still throw an exception before the array is returned, so no invalid data escapes the method. The improvement is primarily stylistic for fail-fast behavior.	Low

Suggestions up to commit c9ce3b7

Category	Suggestion	Impact
Possible issue	Handle invalid base64 decoding errors Add error handling for invalid base64 input to prevent IllegalArgumentException from propagating uncaught. Wrap the decode operation in a try-catch block and throw a more descriptive exception that includes the field name and parsing context. src/main/java/org/opensearch/knn/index/mapper/KNNVectorFieldMapper.java [839-843] } else if (token == XContentParser.Token.VALUE_STRING) { - final byte[] decoded = BASE64_DECODER.decode(context.parser().text()); + final byte[] decoded; + try { + decoded = BASE64_DECODER.decode(context.parser().text()); + } catch (IllegalArgumentException e) { + throw new IllegalArgumentException( + String.format(Locale.ROOT, "Invalid base64 encoding for field [%s]", name()), + e + ); + } validateVectorDimension(dimension, decoded.length, dataType); // Raw decoded bytes are inherently valid: in [-128, 127], no NaN/Inf/decimal possible return Optional.of(decoded); Suggestion importance[1-10]: 7 __ Why: The suggestion correctly identifies that BASE64_DECODER.decode() can throw IllegalArgumentException for invalid base64 input. Adding a try-catch block with a more descriptive error message that includes the field name improves error handling and debugging. However, this is a defensive improvement rather than fixing a critical bug, as the exception would still be caught and reported at a higher level.	Medium

[github-actions]

Persistent review updated to latest commit c9ce3b7

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.79%. Comparing base (9b030f4) to head (7ca2011).

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #3350      +/-   ##
============================================
- Coverage     83.87%   83.79%   -0.08%     
- Complexity     4351     4352       +1     
============================================
  Files           452      452              
  Lines         15715    15744      +29     
  Branches       2046     2050       +4     
============================================
+ Hits          13181    13193      +12     
- Misses         1745     1767      +22     
+ Partials        789      784       -5     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

Persistent review updated to latest commit 0f2273e

[github-actions]

Persistent review updated to latest commit bf699a5

[github-actions]

Persistent review updated to latest commit 314aae6

[github-actions]

Persistent review updated to latest commit eb230f5

[github-actions]

Persistent review updated to latest commit 7ca2011