[compat] [server] [client] [test] Global RT DIV: Chunking Support #1385

KaiSernLim · 2024-12-11T10:06:43Z

Summary

Continuation from #1257.

This PR mainly focuses on adding chunking support for DIV messages when they are produced to Kafka topics, as the size of the DIV message can surpass the ~1MB Kafka message limit. The existing chunking mechanism is reused, including the CHUNK and CHUNKED_VALUE_MANIFEST values in the message's schemaId:

Every DIV message has GLOBAL_RT_DIV for the header byte in its KafkaKey. The corresponding KafkaMessageEnvelope has a Put payload utilizing the putValue field containing the GlobalRtDiv data, and which has the following schemaId:

If the DIV message is non-chunked, the KME schemaId is set to the current protocol version of GLOBAL_RT_DIV.
If the DIV message is chunk message, the KME schemaId is set to CHUNK.
If the DIV message is a chunk manifest message, the KME schemaId is set to CHUNKED_VALUE_MANIFEST. The schemaId of the ChunkedValueManifest will be the current protocol version of GLOBAL_RT_DIV.

ChunkAssembler is adapted, on the receiver side, to buffer, assemble, and deserialize DIV messages (chunked/non-chunked).

Changes

Added a new MessageType called GlobalRtDiv, which reuses the Put message type format and objects. When the Venice server encounters a message with KafkaKey containing the GlobalRtDiv header byte, it will know to process this message differently from a regular Put.
- The only indication of this new GlobalRtDiv message type is the header byte in KafkaKey. Otherwise, it's identical to a regular Put.
- KafkaMessageEnvelope.avsc will not be updated to avoid the unnecessary risk of incompatible avro formats when upgrading the cluster.
- The risk in not creating a dedicated GlobalRtDiv message type in KME is that the GlobalRtDiv objects will be processed as user records and stored in the storage engine, which seems to be much less scary than a cluster upgrade issue.
Changed bufferAndAssembleRecord() in ChunkAssembler to use a deserialization Function instead of a RecordDeserializer. This is because RecordDeserializer is on a separate inheritance path to the InternalAvroSpecificSerializer.
GlobalRtDiv messages should not be processed if they originate from remote VT and RT, because those are invalid scenarios. These two conditions are checked.

Minor Changes

Renamed resetUpstreamOffsetMap() to mergeUpstreamOffsets() in OffsetRecord.
Updated toString() in KafkaKey, which incorrectly assumed all messages would be ControlMessage, Put, or Delete. This misses Update messages and the new GlobalRtDiv message that is being added.
Added various helper methods (buildPutPayload() and buildManifestPayload()) in VeniceWriter for creating the Put payloads and when chunking is involved.

Testing

Unit Tests
1. testGlobalRtDivChunking() in VeniceWriterUnitTest
2. testShouldProcessRecordForGlobalRtDivMessage() in StoreIngestionTaskTest
3. testProcessGlobalRtDivMessage() in StoreIngestionTaskTest
Integration Test
1. testChunkedDiv() in TestGlobalRtDiv

Does this PR introduce any user-facing changes?

No. You can skip the rest of this section.

internal/venice-common/src/main/java/com/linkedin/venice/writer/VeniceWriter.java

internal/venice-common/src/test/java/com/linkedin/venice/writer/VeniceWriterUnitTest.java

sixpluszero · 2024-12-16T19:03:21Z

...ient/src/main/java/com/linkedin/davinci/kafka/consumer/LeaderFollowerStoreIngestionTask.java

@@ -1479,7 +1482,7 @@ protected void updateOffsetMetadataInOffsetRecord(PartitionConsumptionState part
        upstreamTopic = versionTopic;
      }
      if (upstreamTopic.isRealTime()) {
-        offsetRecord.resetUpstreamOffsetMap(partitionConsumptionState.getLatestProcessedUpstreamRTOffsetMap());
+        offsetRecord.mergeUpstreamOffsets(partitionConsumptionState.getLatestProcessedUpstreamRTOffsetMap());


Why do you do this change? Personally I think this makes less explicit about the offset map we are tracking

Because I feel that reset() is not the right word for it. It's not clearing the state or replacing the map entirely. It's updating the existing map with the values of the new map and retaining the values missing from the new map.

I can rename it to updateUpstreamOffsets().

sixpluszero · 2024-12-16T19:17:39Z

internal/venice-common/src/main/java/com/linkedin/venice/kafka/protocol/enums/MessageType.java

@@ -21,7 +21,8 @@
 */
 public enum MessageType implements VeniceEnumValue {
  PUT(0, Constants.PUT_KEY_HEADER_BYTE), DELETE(1, Constants.PUT_KEY_HEADER_BYTE),
-  CONTROL_MESSAGE(2, Constants.CONTROL_MESSAGE_KEY_HEADER_BYTE), UPDATE(3, Constants.UPDATE_KEY_HEADER_BYTE);
+  CONTROL_MESSAGE(2, Constants.CONTROL_MESSAGE_KEY_HEADER_BYTE), UPDATE(3, Constants.UPDATE_KEY_HEADER_BYTE),
+  GLOBAL_RT_DIV(4, Constants.GLOBAL_RT_DIV_KEY_HEADER_BYTE);


I wonder if it is ok to not add it into KME. I am thinking for example, Da Vinci and CDC, if the code is not upgraded, how can it be handling the global rt div? Or should we completely mute them from processing RT div message?
Will this controlled by store config or just server config?

In my opinion, Davinci and CDC don't need to process the RT div messages. Only the next leader has to read and process it, right. They can simply ignore them.

It's controlled by server config.

I mean, with this code change, we will eventually enable the feature to produce RT DIV snapshot to VT topic. This will be processed by follower, which also applicable to DVC. I know DVC won't be leader, but will the existing code handle unknown new msg type gracefully?

You're right. The reader of the msg type (to be able to ignore it) has to be deployed before enabling this feature. i don't know if there is other way to walk around it, feel free to suggest alternatives.

I didn't look very deep, but I think that the existing code will throw an exception when encountering an unknown message type.

Handling an unknown message type will be implemented in a future PR. Since we control the feature flag, we can choose not to enable the feature flag until all server instances support the new message type so there won't be any messages with this message type floating around.

We need to deploy the changes to all VT consumers before enabling this feature, which includes:

Server.

DaVinci.

ETL.

CDC.

And I think it is better to treat the new KME type as a Control Message, and I would like to learn why if we can't do that.

gaojieliu

I would like to call out another item we didn't discuss in the review meeting:

Cleanup the Global RT DIV messages from VT.

This new type of message can be large and with chunking support, we are leaking the data chunks as chunk id is unique.
I was thinking some strategy to let follower to send out a Kafka delete or Kafka message with empty value for the previous key after consuming a new Global RT DIV message, and if we don't do the cleanup, the size of Kafka topic might grow a lot depending on the sending frequency.
Maybe we don't need to implement such cleanup in the MVP, but I think eventually, we need some way to clean them up from the version topics.

...ient/src/main/java/com/linkedin/davinci/kafka/consumer/LeaderFollowerStoreIngestionTask.java

...ts/da-vinci-client/src/main/java/com/linkedin/davinci/kafka/consumer/StoreIngestionTask.java

...inci-client/src/main/java/com/linkedin/davinci/kafka/consumer/StoreIngestionTaskFactory.java

gaojieliu · 2024-12-17T19:11:30Z

internal/venice-common/src/main/java/com/linkedin/venice/kafka/protocol/enums/MessageType.java

@@ -21,7 +21,8 @@
 */
 public enum MessageType implements VeniceEnumValue {
  PUT(0, Constants.PUT_KEY_HEADER_BYTE), DELETE(1, Constants.PUT_KEY_HEADER_BYTE),
-  CONTROL_MESSAGE(2, Constants.CONTROL_MESSAGE_KEY_HEADER_BYTE), UPDATE(3, Constants.UPDATE_KEY_HEADER_BYTE);
+  CONTROL_MESSAGE(2, Constants.CONTROL_MESSAGE_KEY_HEADER_BYTE), UPDATE(3, Constants.UPDATE_KEY_HEADER_BYTE),
+  GLOBAL_RT_DIV(4, Constants.GLOBAL_RT_DIV_KEY_HEADER_BYTE);


We need to deploy the changes to all VT consumers before enabling this feature, which includes:

Server.

DaVinci.

ETL.

CDC.

gaojieliu · 2024-12-17T19:13:06Z

internal/venice-common/src/main/java/com/linkedin/venice/kafka/protocol/enums/MessageType.java

@@ -21,7 +21,8 @@
 */
 public enum MessageType implements VeniceEnumValue {
  PUT(0, Constants.PUT_KEY_HEADER_BYTE), DELETE(1, Constants.PUT_KEY_HEADER_BYTE),
-  CONTROL_MESSAGE(2, Constants.CONTROL_MESSAGE_KEY_HEADER_BYTE), UPDATE(3, Constants.UPDATE_KEY_HEADER_BYTE);
+  CONTROL_MESSAGE(2, Constants.CONTROL_MESSAGE_KEY_HEADER_BYTE), UPDATE(3, Constants.UPDATE_KEY_HEADER_BYTE),
+  GLOBAL_RT_DIV(4, Constants.GLOBAL_RT_DIV_KEY_HEADER_BYTE);


And I think it is better to treat the new KME type as a Control Message, and I would like to learn why if we can't do that.

gaojieliu · 2024-12-17T19:30:09Z

internal/venice-common/src/main/resources/avro/GlobalRtDivState/v1/GlobalRtDivState.avsc

+        "type": "map",
+        "doc": "A map that maps producer GUID -> producer state for realtime data.",
+        "values": {
+          "name": "ProducerPartitionState",


Is it possible to reference the existing definition in PartitionState.avsc?

@sushantmane
I recalled you did some experiment regarding referring existing schema definition, and how does that go?

At compile time, everything worked fine, but at runtime, I ran into schema not found exceptions. I didn’t look into it further and chose to stick with the existing pattern.

* Renamed `resetUpstreamOffsetMap()` to `mergeUpstreamOffsets()`. 😮‍💨 * Minor refactor around `validateMessage()` 🦯🦯 * Copied `KafkaMessageEnvelope.avsc` for a new protocol version. 🥣 * Created `GlobalRtDiv` new `MessageType` that is based on the `Put` messages, and renamed `GlobalDivState` avro object to `GlobalRtDivState`. 🌯 * Unified the code paths of `bufferAndAssembleRecord()` with deserialization . 🥓 * Revised `GlobalRtDiv` chunking support in `VeniceWriter` and adjacent. 🍭 * Reusing `Put` instead of creating new message type `GlobalRtDiv`. 🫐 1. Fixed `shouldProcessRecord()` condition in LFSIT. 🍊 2. Split `ChunkAssembler` for RT DIV into its own object. 🍐 3. `GlobalRtDiv` serializer is per-message to be safe, because it doesn't seem to be thread-safe. 🍋‍🟩 4. Fixed spotbugs. 🌶️ 5. Fixed `divChunkAssembler` for the SIT unit test. 🫨 * The `MessageType` for the KME needs to be `PUT`. Only the `KafkaKey` will have `GLOBAL_RT_DIV` as the `MessageType`. 🏯 * Minor cleanup in the `VeniceWriterUnitTest` and `Put` rather than `Object` in the `sendMessageFunction`. 🪀

…ity.

KaiSernLim requested review from sixpluszero and lluwm December 11, 2024 10:06

KaiSernLim self-assigned this Dec 11, 2024

KaiSernLim mentioned this pull request Dec 11, 2024

[compat][server][client][test] Global RT DIV improvement (part 2): Chunking support for DIV message #1257

Closed

2 tasks

KaiSernLim force-pushed the global-rt-div-chunking branch from 7ee8cd9 to d3ae029 Compare December 12, 2024 19:36

lluwm reviewed Dec 13, 2024

View reviewed changes

internal/venice-common/src/main/java/com/linkedin/venice/writer/VeniceWriter.java Outdated Show resolved Hide resolved

lluwm reviewed Dec 13, 2024

View reviewed changes

internal/venice-common/src/test/java/com/linkedin/venice/writer/VeniceWriterUnitTest.java Outdated Show resolved Hide resolved

sixpluszero reviewed Dec 16, 2024

View reviewed changes

gaojieliu reviewed Dec 17, 2024

View reviewed changes

KaiSernLim force-pushed the global-rt-div-chunking branch from b27282d to ad77fc1 Compare February 7, 2025 01:49

KaiSernLim added 6 commits February 10, 2025 16:56

Part 2 Chunking Support WIP from Lei Lu. 🛟

b6760ec

Rename mergeUpstreamOffsets() to updateUpstreamOffsets() for clar…

c599634

…ity.

Removed the ChunkAssembler for DIV.

d58c272

Reverting ChunkAssembler files post-review.

ed2d3c2

Fixed some tests.

1a22b2f

KaiSernLim force-pushed the global-rt-div-chunking branch from ad77fc1 to 1a22b2f Compare February 11, 2025 00:57

KaiSernLim mentioned this pull request Feb 11, 2025

[compat] Global RT DIV: Schema Changes #1523

Open

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[compat] [server] [client] [test] Global RT DIV: Chunking Support #1385

[compat] [server] [client] [test] Global RT DIV: Chunking Support #1385

KaiSernLim commented Dec 11, 2024 •

edited

Loading

sixpluszero Dec 16, 2024 •

edited

Loading

KaiSernLim Dec 16, 2024

sixpluszero Dec 16, 2024

lluwm Dec 16, 2024 •

edited

Loading

KaiSernLim Dec 16, 2024

sixpluszero Dec 16, 2024

lluwm Dec 16, 2024

KaiSernLim Dec 16, 2024 •

edited

Loading

gaojieliu Dec 17, 2024

gaojieliu Dec 17, 2024

gaojieliu left a comment

gaojieliu Dec 17, 2024

gaojieliu Dec 17, 2024

gaojieliu Dec 17, 2024

gaojieliu Feb 6, 2025

sushantmane Feb 6, 2025

[compat] [server] [client] [test] Global RT DIV: Chunking Support #1385

Are you sure you want to change the base?

[compat] [server] [client] [test] Global RT DIV: Chunking Support #1385

Conversation

KaiSernLim commented Dec 11, 2024 • edited Loading

Summary

Changes

Minor Changes

Testing

Does this PR introduce any user-facing changes?

sixpluszero Dec 16, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lluwm Dec 16, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

KaiSernLim Dec 16, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gaojieliu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

KaiSernLim commented Dec 11, 2024 •

edited

Loading

sixpluszero Dec 16, 2024 •

edited

Loading

lluwm Dec 16, 2024 •

edited

Loading

KaiSernLim Dec 16, 2024 •

edited

Loading