Skip to content

MINOR:Start processor inside verify message #6029

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
guozhangwang merged 2 commits into from
Dec 14, 2018
Merged

MINOR:Start processor inside verify message #6029

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d8971d8
MINOR:Timing issue, streams may start and rebalance before log watch …
bbejeck Dec 11, 2018
17834bd
MINOR: checkstyle fix
bbejeck Dec 13, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions streams/src/test/java/org/apache/kafka/streams/tests/StreamsNamedRepartitionTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,7 @@ public static void main(final String[] args) throws Exception {
final StreamsBuilder builder = new StreamsBuilder();

final KStream<String, String> sourceStream = builder.stream(inputTopic, Consumed.with(Serdes.String(), Serdes.String()));
sourceStream.peek((k, v) -> System.out.println(String.format("input data key=%s, value=%s", k, v)));

final KStream<String, String> mappedStream = sourceStream.selectKey((k, v) -> keyFunction.apply(v));

Expand Down
8 changes: 4 additions & 4 deletions tests/kafkatest/tests/streams/streams_named_repartition_topic_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ def __init__(self, test_context):
}

self.zookeeper = ZookeeperService(self.test_context, num_nodes=1)
self.kafka = KafkaService(self.test_context, num_nodes=1,
self.kafka = KafkaService(self.test_context, num_nodes=3,

@bbejeck bbejeck Dec 13, 2018

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed this value to be consistent with other system tests

zk=self.zookeeper, topics=self.topics)

self.producer = VerifiableProducer(self.test_context,
Expand All @@ -66,7 +66,6 @@ def test_upgrade_topology_with_named_repartition_topic(self):
for processor in processors:
processor.CLEAN_NODE_ENABLED = False
self.set_topics(processor)
processor.start()

@bbejeck bbejeck Dec 13, 2018

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the fix removed start here and below

self.verify_running(processor, 'REBALANCING -> RUNNING')

self.verify_processing(processors)
Expand All @@ -76,7 +75,6 @@ def test_upgrade_topology_with_named_repartition_topic(self):
self.verify_stopped(processor)
# will tell app to add operations before repartition topic
processor.ADD_ADDITIONAL_OPS = 'true'
processor.start()
self.verify_running(processor, 'UPDATED Topology')

self.verify_processing(processors)
Expand All @@ -89,7 +87,9 @@ def test_upgrade_topology_with_named_repartition_topic(self):

@staticmethod
def verify_running(processor, message):
with processor.node.account.monitor_log(processor.STDOUT_FILE) as monitor:
node = processor.node
with node.account.monitor_log(processor.STDOUT_FILE) as monitor:
processor.start()

@bbejeck bbejeck Dec 13, 2018

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the other part of the fix moved processor start here

@vvcephei vvcephei Dec 13, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the significance of after opening the monitor? Does it block on the log file somehow?

@bbejeck bbejeck Dec 13, 2018

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This closes a very small timing window, I've made similar changes in other system tests before

@mjsax mjsax Dec 14, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vvcephei If get don't get the monitor first, we might miss some output lines from the file in wait_until below and thus time out even if the correct output was written.

@vvcephei vvcephei Dec 14, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

monitor.wait_until(message,
timeout_sec=60,
err_msg="Never saw '%s' message " % message + str(processor.node.account))
Expand Down