feat: add SsFormat encoding library for SpanFE bypass #4292
Conversation
product-auto-label
bot
added
size: l
Summary of ChangesHello @rahul2393, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request introduces core encoding utilities for Spanner's key management and routing. It provides a robust Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces the SsFormat encoding library and the TargetRange class, which are foundational for sortable string format (ssformat) encoding utilities used by Spanner for key ordering and routing. The SsFormat class provides various encoding methods for different data types, including integers, strings, doubles, timestamps, and UUIDs, supporting both increasing and decreasing order. The TargetRange class represents a key range and includes logic for merging ranges. Overall, it is a good starting point. However, I've identified a potential correctness issue in the byte sequence encoding and some areas for improved maintainability.
google-cloud-spanner/src/main/java/com/google/cloud/spanner/spi/v1/SsFormat.java
Show resolved
Hide resolved
google-cloud-spanner/src/main/java/com/google/cloud/spanner/spi/v1/SsFormat.java
Outdated
Show resolved
Hide resolved
google-cloud-spanner/src/main/java/com/google/cloud/spanner/spi/v1/TargetRange.java
Show resolved
Hide resolved
This commit adds the foundational SsFormat class that provides sortable string format (ssformat) encoding utilities. This encoding is used by Spanner for key ordering and routing. Key features: - Composite tag encoding for interleaved tables - Signed/unsigned integer encoding (increasing/decreasing) - String and bytes encoding with proper escaping - Double encoding with proper sign handling - Timestamp and UUID encoding - Null value markers with configurable ordering - TargetRange class for key range representation Includes unit tests for all encoding functions. This is part of the experimental location-aware routing for improved latency.
rahul2393
force-pushed
the
feat/ssformat-encoding
branch
from
77c4f60 to
a6b8fc2
Compare
rahul2393
force-pushed
the
feat/ssformat-encoding
branch
from
fcdf2cc to
5ad36f7
Compare
Added missing clirr exemptions for the protobuf library upgrade from GeneratedMessage to GeneratedMessageV3: - 5001: Removed superclass (GeneratedMessage/GeneratedMessage$Builder) - 7005: Parameter type changes (BuilderParent types) - 7006: Return type changes (internalGetFieldAccessorTable) - 7014: Method made final (getDescriptor) These exemptions are applied to all proto modules: - proto-google-cloud-spanner-admin-instance-v1 - proto-google-cloud-spanner-admin-database-v1 - proto-google-cloud-spanner-v1 - proto-google-cloud-spanner-executor-v1
product-auto-label
bot
added
size: xl
| decreasing | ||
| ? ASCENDING_ZERO_ESCAPE | ||
| : ASCENDING_ZERO_ESCAPE); // After inversion, 0xFF becomes 0x00. Escape for 0x00 |
There was a problem hiding this comment.
This seems weird. Regardless of the value of decreasing, the same byte is written. Can this ternary operation be removed? If not, can we have a test that would have caught this?
| out.write( | ||
| decreasing | ||
| ? ASCENDING_FF_ESCAPE | ||
| : ASCENDING_FF_ESCAPE); // After inversion, 0x00 becomes 0xFF. Escape for 0xFF |
There was a problem hiding this comment.
Same as above: Either remove the ternary operation, or add a test that fails with this implementation, and succeeds after fixing this.
| private SsFormat() {} | ||
|
|
||
| private static final int IS_KEY = 0x80; | ||
| private static final int TYPE_MASK = 0x7f; |
There was a problem hiding this comment.
This seems to be unused. Can it be removed?
| import java.io.ByteArrayOutputStream; | ||
| import java.nio.charset.StandardCharsets; | ||
|
|
||
| public final class SsFormat { |
There was a problem hiding this comment.
nit: mark as @InternalApi
| private static final int TYPE_UINT_1 = 0; | ||
| private static final int TYPE_UINT_9 = 8; | ||
| private static final int TYPE_NEG_INT_8 = 9; | ||
| private static final int TYPE_NEG_INT_1 = 16; | ||
| private static final int TYPE_POS_INT_1 = 17; | ||
| private static final int TYPE_POS_INT_8 = 24; | ||
| private static final int TYPE_STRING = 25; | ||
| private static final int TYPE_NULL_ORDERED_FIRST = 27; | ||
| private static final int TYPE_NULLABLE_NOT_NULL_NULL_ORDERED_FIRST = 28; | ||
| private static final int TYPE_DECREASING_UINT_9 = 32; | ||
| private static final int TYPE_DECREASING_UINT_1 = 40; | ||
| private static final int TYPE_DECREASING_NEG_INT_8 = 41; | ||
| private static final int TYPE_DECREASING_NEG_INT_1 = 48; | ||
| private static final int TYPE_DECREASING_POS_INT_1 = 49; | ||
| private static final int TYPE_DECREASING_POS_INT_8 = 56; | ||
| private static final int TYPE_DECREASING_STRING = 57; | ||
| private static final int TYPE_NULLABLE_NOT_NULL_NULL_ORDERED_LAST = 59; | ||
| private static final int TYPE_NULL_ORDERED_LAST = 60; | ||
| private static final int TYPE_NEG_DOUBLE_8 = 66; | ||
| private static final int TYPE_NEG_DOUBLE_1 = 73; | ||
| private static final int TYPE_POS_DOUBLE_1 = 74; | ||
| private static final int TYPE_POS_DOUBLE_8 = 81; | ||
| private static final int TYPE_DECREASING_NEG_DOUBLE_8 = 82; | ||
| private static final int TYPE_DECREASING_NEG_DOUBLE_1 = 89; | ||
| private static final int TYPE_DECREASING_POS_DOUBLE_1 = 90; | ||
| private static final int TYPE_DECREASING_POS_DOUBLE_8 = 97; |
There was a problem hiding this comment.
Multiple of these constants seem to be unused. Can we remove the ones that are not used?
| return ByteString.EMPTY; | ||
| } | ||
| byte[] bytes = key.toByteArray(); | ||
| if (bytes.length > 0) { |
There was a problem hiding this comment.
nit: this if statement seems redundant, given the check above if (key == null || key.isEmpty()).
If it is needed: Can we have a test that covers both branches of this if statement?
| private static final byte SEP = (byte) 0x78; // 'x' | ||
|
|
||
| // For AppendCompositeTag | ||
| private static final int K_OBJECT_EXISTENCE_TAG = 0x7e; |
There was a problem hiding this comment.
This constant appears only to be used in the check at the start of appendCompositeTag, but otherwise unused (including in tests). Can it be removed? If not, can we add tests that use it?
| appendByteSequence(out, value, false); | ||
| } | ||
|
|
||
| public static void appendBytesDecreasing(ByteArrayOutputStream out, byte[] value) { |
There was a problem hiding this comment.
This appears unused (including by tests). Can we remove it, or add tests for it?
| byte[] buf = new byte[12]; | ||
| // Big-endian encoding | ||
| buf[0] = (byte) (hi >> 56); | ||
| buf[1] = (byte) (hi >> 48); | ||
| buf[2] = (byte) (hi >> 40); | ||
| buf[3] = (byte) (hi >> 32); | ||
| buf[4] = (byte) (hi >> 24); | ||
| buf[5] = (byte) (hi >> 16); | ||
| buf[6] = (byte) (hi >> 8); | ||
| buf[7] = (byte) hi; | ||
| buf[8] = (byte) (lo >> 24); | ||
| buf[9] = (byte) (lo >> 16); | ||
| buf[10] = (byte) (lo >> 8); | ||
| buf[11] = (byte) lo; | ||
| return buf; |
There was a problem hiding this comment.
You can simplify this to:
byte[] buf = new byte[12];
ByteBuffer.wrap(buf).order(ByteOrder.BIG_ENDIAN).putLong(hi).putInt(nanos);
return buf;
| byte[] buf = new byte[16]; | ||
| // Big-endian encoding | ||
| buf[0] = (byte) (high >> 56); | ||
| buf[1] = (byte) (high >> 48); | ||
| buf[2] = (byte) (high >> 40); | ||
| buf[3] = (byte) (high >> 32); | ||
| buf[4] = (byte) (high >> 24); | ||
| buf[5] = (byte) (high >> 16); | ||
| buf[6] = (byte) (high >> 8); | ||
| buf[7] = (byte) high; | ||
| buf[8] = (byte) (low >> 56); | ||
| buf[9] = (byte) (low >> 48); | ||
| buf[10] = (byte) (low >> 40); | ||
| buf[11] = (byte) (low >> 32); | ||
| buf[12] = (byte) (low >> 24); | ||
| buf[13] = (byte) (low >> 16); | ||
| buf[14] = (byte) (low >> 8); | ||
| buf[15] = (byte) low; | ||
| return buf; |
There was a problem hiding this comment.
Same as above: you can just use ByteBuffer for this
3 participants
This commit adds the foundational SsFormat class that provides sortable string format (ssformat) encoding utilities. This encoding is used by Spanner for key ordering and routing.
Key features:
This is part of the experimental location-aware routing for improved latency.