Skip to content

Sync from apache/kafka (8 August 2019) and upgrade avro to 1.9.0#217

Merged
Ismael Juma (ijuma) merged 7 commits into
masterfrom
sync-upstream-aug-8
Aug 9, 2019
Merged

Sync from apache/kafka (8 August 2019) and upgrade avro to 1.9.0#217
Ismael Juma (ijuma) merged 7 commits into
masterfrom
sync-upstream-aug-8

Conversation

@ijuma

@ijuma Ismael Juma (ijuma) commented Aug 8, 2019 •
edited
Loading

Copy link
Copy Markdown
Member

The avro upgrade was needed to fix the following error during
':support-metrics-client:generateAvro':

Caused by: org.apache.velocity.exception.MethodInvocationException: Variable $velocityCount has not been set at ...

Conflicts:

  • gradle.properties -> trivial fix, scalaVersion is next to version, which
    is different in ccs kafka
  • gradle/dependencies.gradle -> removed unused httpclient and httpmime in
    the versions array since they were unused and were the reason for the
    divergence with upstream.

Committer Checklist (excluded from commit message)

  • Verify design and implementation
  • Verify test coverage and CI build status
  • Verify documentation (including upgrade notes)

Bruno Cadonna (cadonna) and others added 6 commits August 6, 2019 08:51
@cadonna @guozhangwang
Minor: Refactor methods to add metrics to sensor in `StreamsMetricsIm…
7663a6c 
…pl` (apache#7161)

Renames method names in StreamsMetricsImpl to make them consistent.

Reviewers: A. Sophie Blee-Goldman <sophie@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
@mjarvie @mjsax
KAFKA-8736: Streams performance improvement, use isEmpty() rather tha…
7ebcd50 
…n size() == 0 (apache#7164)

Reviewers: A. Sophie Blee-Goldman <sophie@confluent.io>, Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <guozhang@confluent.io>
@cmccabe @gwenshap
MINOR: some small style fixes to RoundRobinPartitioner
e4e2dd3 
Author: Colin P. Mccabe <cmccabe@confluent.io>

Reviewers: Gwen Shapira

Closes apache#7058 from cmccabe/rr-style-fixes
@ijuma
MINOR: Upgrade jackson-databind to 2.9.9.3 (apache#7125)
97b731b 
2.9.9.1 and 2.9.9.2 include security fixes while 2.9.9.3 fixes a regression
introduced in 2.9.9.2.

Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>
@mimaison @omkreddy
KAFKA-8599: Use automatic RPC generation in ExpireDelegationToken
926fb35 
Author: Mickael Maison <mickael.maison@gmail.com>

Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>, Viktor Somogyi <viktorsomogyi@gmail.com>

Closes apache#7098 from mimaison/KAFKA-8599
@ijuma
MINOR: Update dependencies for Kafka 2.4 (apache#7126)
66d81a0 
Scala 2.12.9 brings another 5% ~ 10% improvement in compiler performance,
improved compatibility with JDK 11/12/13, and experimental infrastructure for
build pipelining.

zstd update includes performance improvements, among which the
primary improvement is that decompression is ~7% faster.

Level | v1.4.0 | v1.4.1 | Delta
-- | -- | -- | --
1 | 1390 MB/s | 1453 MB/s | +4.5%
3 | 1208 MB/s | 1301 MB/s | +7.6%
5 | 1129 MB/s | 1233 MB/s | +9.2%
7 | 1224 MB/s | 1347 MB/s | +10.0%
16 | 1278 MB/s | 1430 MB/s | +11.8%

Jetty 9.4.19 includes a number of bug fixes:
https://github.com/eclipse/jetty.project/releases/tag/jetty-9.4.19.v20190610

Mockito 3.0.0 switched the Java requirement from 7 to 8.

Several updates to owaspDepCheckPlugin (4.0.2 -> 5.2.1).

The rest are patch updates.

Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>
@ijuma Ismael Juma (ijuma) changed the title Sync from apache/kafka - master (8 August 2019) Sync from apache/kafka (8 August 2019) Aug 8, 2019
stanislavkozlovski
Stanislav Kozlovski (stanislavkozlovski) reviewed Aug 8, 2019
View reviewed changes
Comment thread gradle/dependencies.gradle Outdated

@stanislavkozlovski Stanislav Kozlovski (stanislavkozlovski) Aug 8, 2019 •
edited
Loading

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

httpclient is present in AK/trunk - https://github.com/apache/kafka/blob/66d81a0e50680916f541703049dd4e911542b5ed/gradle/dependencies.gradle#L72

Best remove it from there if really is unused?

@ijuma Ismael Juma (ijuma) Aug 8, 2019

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's used there, but not here. I'll reduce the divergence further.

stanislavkozlovski
Stanislav Kozlovski (stanislavkozlovski) reviewed Aug 8, 2019
View reviewed changes
Comment thread gradle/dependencies.gradle Outdated

@stanislavkozlovski Stanislav Kozlovski (stanislavkozlovski) Aug 8, 2019

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This httpcomponents is unused and not in AK.

@ijuma Ismael Juma (ijuma) Aug 8, 2019

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's used here, but not in AK. I'll reduce the divergence.

@ijuma
Ismael Juma (ijuma) force-pushed the sync-upstream-aug-8 branch 3 times, most recently from 4760719 to 89677a2 Compare August 8, 2019 14:56
@ijuma Ismael Juma (ijuma) changed the title Sync from apache/kafka (8 August 2019) Sync from apache/kafka (8 August 2019) and upgrade avro to 1.9.0 Aug 8, 2019
@ijuma
Sync from apache/kafka (8 August 2019) and upgrade avro to 1.9.0
430f014 
The avro upgrade was needed to fix the following error during
':support-metrics-client:generateAvro':

Caused by: org.apache.velocity.exception.MethodInvocationException: Variable $velocityCount has not been set at /org/apache/avro/compiler/specific/templates/java/classic/record.vm[line 82, column 150]

Conflicts:
 * gradle.properties -> trivial fix, `scalaVersion` is next to `version`, which
 is different in ccs kafka
 * gradle/dependencies.gradle -> reduce divergence with apache kafka by using
 `httpclient` instead of `httpcomponents` from the `versions` array. Remove
 unused `httpmime` and `httpcomponents` from said array.

* apache-github/trunk:
  MINOR: Update dependencies for Kafka 2.4 (apache#7126)
  KAFKA-8599: Use automatic RPC generation in ExpireDelegationToken
  MINOR: Upgrade jackson-databind to 2.9.9.3 (apache#7125)
  MINOR: some small style fixes to RoundRobinPartitioner
  KAFKA-8736: Streams performance improvement, use isEmpty() rather than size() == 0 (apache#7164)
  Minor: Refactor methods to add metrics to sensor in `StreamsMetricsImpl` (apache#7161)
@ijuma

Ismael Juma (ijuma) commented Aug 8, 2019

Copy link
Copy Markdown
Member Author

Flaky tests that failed:

kafka.admin.ReassignPartitionsClusterTest.testHwAfterPartitionReassignment
kafka.api.AdminClientIntegrationTest.testIncrementalAlterConfigsForLog4jLogLevels
kafka.api.AdminClientIntegrationTest.testIncrementalAlterConfigsForLog4jLogLevelsCanResetLoggerToCurrentRoot
kafka.api.PlaintextConsumerTest.testFetchInvalidOffset
kafka.api.SaslGssapiSslEndToEndAuthorizationTest.testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl
kafka.api.SaslPlainSslEndToEndAuthorizationTest.testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl

Restarted the build.

@ijuma

Ismael Juma (ijuma) commented Aug 9, 2019

Copy link
Copy Markdown
Member Author

More flaky failures (subset of the previous ones):

kafka.api.AdminClientIntegrationTest.testIncrementalAlterConfigsForLog4jLogLevels
kafka.api.AdminClientIntegrationTest.testIncrementalAlterConfigsForLog4jLogLevelsCanResetLoggerToCurrentRoot
kafka.api.DelegationTokenEndToEndAuthorizationTest.testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl

Stanislav Kozlovski (@stanislavkozlovski) has a PR for the log4j log levels issue. There are JIRAs in Apache Kafka for the other flaky tests.

@ijuma

Ismael Juma (ijuma) commented Aug 9, 2019

Copy link
Copy Markdown
Member Author

Merging to master.

@ijuma
Ismael Juma (ijuma) merged commit 430f014 into master Aug 9, 2019
@ijuma
Ismael Juma (ijuma) deleted the sync-upstream-aug-8 branch January 25, 2020 16:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer
@stanislavkozlovski Stanislav Kozlovski (stanislavkozlovski) stanislavkozlovski left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@ijuma @stanislavkozlovski @cadonna @mjarvie @cmccabe @mimaison