Skip to content

MINOR: clarify node grouping of input topics using pattern subscription #7793

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
vvcephei merged 4 commits into from
Dec 6, 2019
Merged

MINOR: clarify node grouping of input topics using pattern subscription #7793

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
02618bb
doc
ableegoldman Dec 6, 2019
0eb7946
javadocs
ableegoldman Dec 6, 2019
8c7fb3e
alternative phrasing
ableegoldman Dec 6, 2019
a9a11b2
rephrase javadocs
ableegoldman Dec 6, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/streams/developer-guide/dsl-api.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -261,7 +261,7 @@ <h4><a id="streams_concepts_globalktable" href="#streams_concepts_globalktable">
<p>You <strong>must specify SerDes explicitly</strong> if the key or value types of the records in the Kafka input
topics do not match the configured default SerDes. For information about configuring default SerDes, available
SerDes, and implementing your own custom SerDes see <a class="reference internal" href="datatypes.html#streams-developer-guide-serdes"><span class="std std-ref">Data Types and Serialization</span></a>.</p>
<p class="last">Several variants of <code class="docutils literal"><span class="pre">stream</span></code> exist, for example to specify a regex pattern for input topics to read from).</p>
<p class="last">Several variants of <code class="docutils literal"><span class="pre">stream</span></code> exist. For example, you can specify a regex pattern for input topics to read from (note that a single nodegroup will be created for all matching topics, and therefore the number of tasks will scale with the maximum partition count rather than the total number of partitions).</p>

@vvcephei vvcephei Dec 6, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is nodegroup already a defined term by this point? Looking at this and the other changes, this might be kind of a confusing usage of the word "node". Maybe just make up something more intuitive, like "input topic group"?

@ableegoldman ableegoldman Dec 6, 2019

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, that's fair. I didn't know what that was when I first saw it in the code. What do you think about using "subtopology" here?

Alternatively, how about: note that all matching topics will be part of the same input topic group, and the work will not be parallelized for different topics if subscribed to in this way

@vvcephei vvcephei Dec 6, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think your "alternative" is the best phrasing yet.

@ableegoldman ableegoldman Dec 6, 2019

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Rephrased the javadocs as well -- thanks!

</td>
</tr>
<tr class="row-odd"><td><p class="first"><strong>Table</strong></p>
Expand Down
8 changes: 6 additions & 2 deletions streams/src/main/java/org/apache/kafka/streams/StreamsBuilder.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -148,7 +148,9 @@ public synchronized <K, V> KStream<K, V> stream(final Collection<String> topics,
* deserializers as specified in the {@link StreamsConfig config} are used.
* <p>
* If multiple topics are matched by the specified pattern, the created {@link KStream} will read data from all of
* them and there is no ordering guarantee between records from different topics.
* them and there is no ordering guarantee between records from different topics. These topics will all be grouped
* into a single node group, and as such the number of tasks generated will scale with the maximum partition count
* across the matching topics, and not with the total number of partitions summed over them.
* <p>
* Note that the specified input topics must be partitioned by key.
* If this is not the case it is the user's responsibility to repartition the data before any key based operation
Expand All @@ -167,7 +169,9 @@ public synchronized <K, V> KStream<K, V> stream(final Pattern topicPattern) {
* are defined by the options in {@link Consumed} are used.
* <p>
* If multiple topics are matched by the specified pattern, the created {@link KStream} will read data from all of
* them and there is no ordering guarantee between records from different topics.
* them and there is no ordering guarantee between records from different topics. These topics will all be grouped
* into a single node group, and as such the number of tasks generated will scale with the maximum partition count
* across the matching topics, and not with the total number of partitions summed over them.
* <p>
* Note that the specified input topics must be partitioned by key.
* If this is not the case it is the user's responsibility to repartition the data before any key based operation
Expand Down