Skip to content

KAFKA-15378: fix streams upgrade system test #14539

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mjsax merged 4 commits into from
Oct 20, 2023
+13 −9
Merged

KAFKA-15378: fix streams upgrade system test #14539

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
95f8526
KAFKA-15378: fix streams upgrade system test
mjsax Oct 7, 2023
8d5ef8d
disable state updater thread
mjsax Oct 13, 2023
677773f
re-enable state-updater after rebasing
mjsax Oct 18, 2023
3d9ea7d
Fix broken test: cf https://github.com/apache/kafka/commit/fcac880fd5…
mjsax Oct 18, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion tests/kafkatest/tests/streams/streams_broker_down_resilience_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,7 @@ def test_streams_runs_with_broker_down_initially(self, metadata_quorum):
processor_3 = StreamsBrokerDownResilienceService(self.test_context, self.kafka, configs)
processor_3.start()

broker_unavailable_message = "Broker may not be available"
broker_unavailable_message = "Node may not be available"

@mjsax mjsax Oct 18, 2023

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Log message was changed via fcac880


# verify streams instances unable to connect to broker, kept trying
self.wait_for_verification(processor, broker_unavailable_message, processor.LOG_FILE, 10)
Expand Down
14 changes: 8 additions & 6 deletions tests/kafkatest/tests/streams/streams_upgrade_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,11 +40,13 @@
metadata_1_versions = [str(LATEST_0_10_0)]
metadata_2_versions = [str(LATEST_0_10_1), str(LATEST_0_10_2), str(LATEST_0_11_0), str(LATEST_1_0), str(LATEST_1_1),
str(LATEST_2_4), str(LATEST_2_5), str(LATEST_2_6), str(LATEST_2_7), str(LATEST_2_8),
str(LATEST_3_0)]
# upgrading from version (2.4...3.0) is broken and only fixed later in 3.1
# we cannot test two bounce rolling upgrade because we know it's broken
# instead we add version 2.4...3.0 to the `metadata_2_versions` upgrade list
fk_join_versions = [str(LATEST_3_1), str(LATEST_3_2), str(LATEST_3_3)]
str(LATEST_3_0), str(LATEST_3_1), str(LATEST_3_2), str(LATEST_3_3)]
# upgrading from version (2.4...3.3) is broken and only fixed later in 3.3.3 (unreleased) and 3.4.0
# -> https://issues.apache.org/jira/browse/KAFKA-14646
# thus, we cannot test two bounce rolling upgrade because we know it's broken
# instead we add version 2.4...3.3 to the `metadata_2_versions` upgrade list
#fk_join_versions = [str(LATEST_3_4)]

@mjsax mjsax Oct 18, 2023

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mimaison You will need to uncomment this, and also add 3.5 release to this list in your PR, and reenable the corresponding @matix annotation, too.

@mimaison mimaison Oct 18, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Noted, I'll do that once this is merged. Thanks



"""
After each release one should first check that the released version has been uploaded to
Expand Down Expand Up @@ -202,7 +204,7 @@ def test_upgrade_downgrade_brokers(self, from_version, to_version):
@cluster(num_nodes=6)
@matrix(from_version=metadata_1_versions, to_version=[str(DEV_VERSION)])
@matrix(from_version=metadata_2_versions, to_version=[str(DEV_VERSION)])
@matrix(from_version=fk_join_versions, to_version=[str(DEV_VERSION)])
#@matrix(from_version=fk_join_versions, to_version=[str(DEV_VERSION)])
def test_rolling_upgrade_with_2_bounces(self, from_version, to_version):
"""
This test verifies that the cluster successfully upgrades despite changes in the metadata and FK
Expand Down
6 changes: 4 additions & 2 deletions tests/kafkatest/version.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,8 +107,10 @@ def supports_topic_ids_when_using_zk(self):
return self >= V_2_8_0

def supports_fk_joins(self):
# while we support FK joins since 2.4, rolling upgrade is broken in older versions and only fixed in 3.1
return hasattr(self, "version") and self >= V_3_1_2
# while we support FK joins since 2.4, rolling upgrade is broken in older versions
# it's only fixed in 3.3.3 (unreleased) and 3.4.0
# -> https://issues.apache.org/jira/browse/KAFKA-14646
return hasattr(self, "version") and self >= V_3_4_0

def get_version(node=None):
"""Return the version attached to the given node.
Expand Down