Skip to content

KAFKA-4923: Add Exaclty-Once Semantics to Streams (testing)#2974

Closed
mjsax wants to merge 1 commit into
apache:trunkfrom
mjsax:kafka-4923-add-eos-to-streams-add-broker-check-and-system-test
Closed

KAFKA-4923: Add Exaclty-Once Semantics to Streams (testing)#2974
mjsax wants to merge 1 commit into
apache:trunkfrom
mjsax:kafka-4923-add-eos-to-streams-add-broker-check-and-system-test

Conversation

@mjsax

@mjsax mjsax commented May 4, 2017

Copy link
Copy Markdown
Member
  • add broker compatibility system tests

 - add broker compatibility system tests
@mjsax

mjsax commented May 4, 2017

Copy link
Copy Markdown
Member Author

Call for review @enothereska @dguy

@mjsax

mjsax commented May 4, 2017

Copy link
Copy Markdown
Member Author

@asfbot

asfbot commented May 4, 2017

Copy link
Copy Markdown

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/3460/
Test FAILed (JDK 8 and Scala 2.12).

@asfbot

asfbot commented May 4, 2017

Copy link
Copy Markdown

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.10/3463/
Test PASSed (JDK 7 and Scala 2.10).

@asfbot

asfbot commented May 4, 2017

Copy link
Copy Markdown

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/3469/
Test PASSed (JDK 8 and Scala 2.11).

Comment thread checkstyle/checkstyle.xml
<module name="BooleanExpressionComplexity">
<!-- default is 3 -->
<property name="max" value="4"/>
<property name="max" value="5"/>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you just add a suppression for the file that needs to have 5 booleans? There is little point in having rules if people are just going to keep on changing the values

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is also little point in adding an exception for each single file that break the rule IMHO. I just think that 4 (and even 5) is a rather small number anyway. (But this goes back to you other comment -- if we only check a single API key, we could keep 4 here.)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmm. well the point is to not break the rule! If you really feel you need to break the rule then add a suppression

}

private boolean brokerSupportsTransactions(final ApiVersionsResponse apiVersionsResponse) {
return apiVersionsResponse.apiVersion(ApiKeys.INIT_PRODUCER_ID.id) != null

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we actually need to check for all of these ApiKeys? i.e., we know that they are all going in as part of the same release, can we not just check for at least that version instead?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't think there is a strict need -- I thought it might be "cleaner" to check all. But maybe that overkill. Curious what others think about this.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess it doesn't harm.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree that it is safer to check all: though it may unlike to happen but we may remove some of the APIs moving forward.

@mjsax

mjsax commented May 4, 2017

Copy link
Copy Markdown
Member Author

@enothereska

Copy link
Copy Markdown
Contributor

What kind of testing is this PR supposed to cover (so I know if it's complete or not)?

@mjsax

mjsax commented May 4, 2017

Copy link
Copy Markdown
Member Author

It adds:

  • 0.11 Streams fails if EOS enabled and brokers are pre-0.11
  • 0.11 Streams work for pre-0.11 brokers if EOS is disabled

The whole system test should cover: https://github.com/apache/kafka/pull/2974/files#diff-7924cd3d80f77980ae421699e631e2ccR29

@dguy dguy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comments were only minor. LGTM

@mjsax

mjsax commented May 19, 2017

Copy link
Copy Markdown
Member Author

@guozhangwang Call for review and merging.

@asfbot

asfbot commented May 19, 2017

Copy link
Copy Markdown

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/4216/
Test PASSed (JDK 7 and Scala 2.11).

}

private boolean brokerSupportsTransactions(final ApiVersionsResponse apiVersionsResponse) {
return apiVersionsResponse.apiVersion(ApiKeys.INIT_PRODUCER_ID.id) != null

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree that it is safer to check all: though it may unlike to happen but we may remove some of the APIs moving forward.

asfgit pushed a commit that referenced this pull request May 22, 2017
…treams

 - add broker compatibility system tests

Author: Matthias J. Sax <matthias@confluent.io>

Reviewers: Damian Guy, Eno Thereska, Guozhang Wang

Closes #2974 from mjsax/kafka-4923-add-eos-to-streams-add-broker-check-and-system-test

(cherry picked from commit 495836a)
Signed-off-by: Guozhang Wang <wangguoz@gmail.com>
@guozhangwang

Copy link
Copy Markdown
Contributor

Merged to trunk and 0.11.0.

@asfgit asfgit closed this in 495836a May 22, 2017
@mjsax
mjsax deleted the kafka-4923-add-eos-to-streams-add-broker-check-and-system-test branch May 22, 2017 16:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants