KAFKA-19570: Implement offline migration for streams groups#20288
Merged
Conversation
Offline migration essentially preserves offsets and nothing else. So effectively write tombstones for classic group type when a streams heartbeat is sent to with the group ID of an empty classic group, and write tombstones for the streams group type when a classic consumer attempts to join with a group ID of an empty streams group.
Contributor
There was a problem hiding this comment.
Pull Request Overview
This PR implements offline migration for Kafka Streams groups, allowing seamless conversion between classic consumer groups and streams groups when the target group is empty. The migration preserves offsets while writing tombstones for the previous group type.
Key changes:
- Added migration logic for empty classic groups to streams groups when a streams heartbeat is received
- Added migration logic for empty streams groups to classic groups when a classic consumer attempts to join
- Enhanced group creation methods to handle cross-group-type migrations
Reviewed Changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| GroupMetadataManager.java | Implements core migration logic with helper methods and updates group creation/join flows |
| GroupMetadataManagerTest.java | Adds comprehensive test coverage for both migration scenarios and edge cases |
| * The group will be materialized during the replay. | ||
| * | ||
| * If there is an empty classic consumer group of the same name, it will be deleted and a new streams | ||
| * group. |
There was a problem hiding this comment.
The comment is incomplete - it ends abruptly with 'a new streams' without completing the sentence. Should be 'a new streams group will be created.'
Suggested change
| * group. | |
| * group will be created. |
Comment on lines
+836
to
+837
| * If there is an empty classic consumer group of the same name, it will be deleted and a new streams | ||
| * group. |
Contributor
There was a problem hiding this comment.
Copilot's first comment looks correct, we're missing a word here.
Suggested change
| * If there is an empty classic consumer group of the same name, it will be deleted and a new streams | |
| * group. | |
| * If there is an empty classic consumer group of the same name, it will be deleted and a new streams | |
| * group wil be created. |
dongnuo123
reviewed
Aug 3, 2025
| return classicGroupJoinToConsumerGroup((ConsumerGroup) group, context, request, responseFuture); | ||
| } else if (group.type() == CONSUMER || group.type() == CLASSIC) { | ||
| } else if (group.type() == CONSUMER || group.type() == CLASSIC || group.type() == STREAMS && group.isEmpty()) { | ||
| return classicGroupJoinToClassicGroup(context, request, responseFuture); |
Contributor
There was a problem hiding this comment.
Should we add a comment to state the groups accepted by classicGroupJoinToClassicGroup here?
- existing classic group
- existing empty consumer group (will be converted to classic)
- existing empty streams group (will be converted to classic)
Member
Author
|
@squah-confluent @dongnuo123 Thanks for the comments. I addressed the them and merged trunk |
squah-confluent
approved these changes
Aug 25, 2025
bbejeck
approved these changes
Aug 25, 2025
eduwercamacaro
pushed a commit
to littlehorse-enterprises/kafka
that referenced
this pull request
Nov 12, 2025
…0288) Offline migration essentially preserves offsets and nothing else. So effectively write tombstones for classic group type when a streams heartbeat is sent to with the group ID of an empty classic group, and write tombstones for the streams group type when a classic consumer attempts to join with a group ID of an empty streams group. Reviewers: Bill Bejeck <bbejeck@apache.org>, Sean Quah <squah@confluent.io>, Dongnuo Lyu <dlyu@confluent.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Offline migration essentially preserves offsets and nothing else. So
effectively write tombstones for classic group type when a streams
heartbeat is sent to with the group ID of an empty classic group, and
write tombstones for the streams group type when a classic consumer
attempts to join with a group ID of an empty streams group.
Reviewers: Bill Bejeck bbejeck@apache.org, Sean Quah
squah@confluent.io, Dongnuo Lyu dlyu@confluent.io