Add support for binary and byte field support in doc_values

[navneet1v]

Description

Add support for binary and byte field support in doc_values

Related Issues

Resolves #3315

Check List

• New functionality includes testing.
• New functionality has been documented.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

PR Reviewer Guide 🔍

(Review updated until commit 556ef1a)

Here are some key observations to aid the review process:

🧪 PR contains tests

🔒 No security concerns identified

✅ No TODO sections

🔀 No multiple PR themes

⚡ Recommended focus areas for review Possible Issue The toIntArray method converts signed bytes to ints without preserving the unsigned interpretation. For example, byte value -128 becomes int -128, not 128. If the consumer expects unsigned byte values (0-255 range), this will produce incorrect results. This matters when the API contract or downstream code expects unsigned byte semantics for BYTE/BINARY vectors. private static int[] toIntArray(final byte[] bytes) { final int[] ints = new int[bytes.length]; for (int i = 0; i < bytes.length; i++) { ints[i] = bytes[i]; } return ints; }

[github-actions]

PR Code Suggestions ✨

Latest suggestions up to 556ef1a

Explore these optional code suggestions:

Category	Suggestion	Impact
Possible issue	Fix byte-to-int sign extension issue The conversion from byte[] to int[] doesn't handle byte sign extension correctly. When a byte is cast to int, Java sign-extends it (e.g., byte -1 becomes int -1), but the test expects unsigned interpretation (e.g., byte -86 should become int 170, not -86). Apply bitwise AND with 0xFF to convert bytes to unsigned integers. src/main/java/org/opensearch/knn/index/KNNVectorDVLeafFieldData.java [182-188] private static int[] toIntArray(final byte[] bytes) { final int[] ints = new int[bytes.length]; for (int i = 0; i < bytes.length; i++) { - ints[i] = bytes[i]; + ints[i] = bytes[i] & 0xFF; } return ints; } Suggestion importance[1-10]: 9 __ Why: This is a critical bug. Without the & 0xFF mask, negative bytes are sign-extended to negative integers (e.g., byte -86 becomes int -86 instead of 170). The test cases in the PR expect unsigned interpretation, as shown by binaryVector1 = { -86 } being compared against the result. This will cause incorrect serialization of byte/binary vectors in array format.	High

---

Previous suggestions

Suggestions up to commit 5302f07

Category	Suggestion	Impact
Possible issue	Convert bytes to unsigned integers The conversion from byte[] to int[] doesn't preserve unsigned byte values correctly. Java bytes are signed (-128 to 127), but when converting to int, negative bytes remain negative. For proper representation of byte values (0-255), use bytes[i] & 0xFF to convert to unsigned int. src/main/java/org/opensearch/knn/index/KNNVectorDVLeafFieldData.java [182-188] private static int[] toIntArray(final byte[] bytes) { final int[] ints = new int[bytes.length]; for (int i = 0; i < bytes.length; i++) { - ints[i] = bytes[i]; + ints[i] = bytes[i] & 0xFF; } return ints; } Suggestion importance[1-10]: 3 __ Why: The suggestion raises a valid point about signed byte conversion, but the impact depends on the intended use case. The PR's test cases show that signed byte values (e.g., -3, -128) are expected to remain negative in the int[] output, suggesting the current implementation may be intentional. Without clear requirements for unsigned conversion, this is a minor style/interpretation issue rather than a critical bug.	Low

Suggestions up to commit 5302f07

Category	Suggestion	Impact
Possible issue	Fix signed byte to int conversion The conversion from byte[] to int[] does not preserve unsigned byte values correctly. Java bytes are signed (-128 to 127), but when converting to int, negative bytes remain negative. For proper unsigned representation (0-255), use bytes[i] & 0xFF to mask the sign extension. src/main/java/org/opensearch/knn/index/KNNVectorDVLeafFieldData.java [182-188] private static int[] toIntArray(final byte[] bytes) { final int[] ints = new int[bytes.length]; for (int i = 0; i < bytes.length; i++) { - ints[i] = bytes[i]; + ints[i] = bytes[i] & 0xFF; } return ints; } Suggestion importance[1-10]: 9 __ Why: The suggestion correctly identifies a critical bug in the toIntArray method. Java bytes are signed (-128 to 127), and direct assignment to int preserves the sign, resulting in negative values. For proper unsigned representation (0-255), the code must use bytes[i] & 0xFF to mask sign extension. This is essential for correct serialization of BYTE/BINARY vectors as JSON numeric arrays.	High

[github-actions]

Persistent review updated to latest commit 5302f07

[github-actions]

Persistent review updated to latest commit 556ef1a

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.51%. Comparing base (11bd374) to head (556ef1a).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #3340      +/-   ##
============================================
- Coverage     83.58%   83.51%   -0.07%     
+ Complexity     4308     4303       -5     
============================================
  Files           451      451              
  Lines         15584    15588       +4     
  Branches       2018     2020       +2     
============================================
- Hits          13026    13019       -7     
- Misses         1770     1779       +9     
- Partials        788      790       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[shatejas]

Looks good overall