CONFLUENT: Sync from apache/kafka/trunk (17 March 2021)#536
Merged
Conversation
Reviewers: A. Sophie Blee-Goldman <sophie@confluent.io>, Chia-Ping Tsai <chia7712@gmail.com>
Here is the fix. The reason of [CVE-2020-27223](https://nvd.nist.gov/vuln/detail/CVE-2020-27223) was DOS vulnerability for Quoted Quality CSV headers and [patched in 9.4.37.v20210219](GHSA-m394-8rww-3jr7). This PR updates Jetty dependency into the following version, 9.4.38.v20210224. Author: Lee Dongjin <dongjin@apache.org> Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com> Closes apache#10245 from dongjinleekr/feature/KAFKA-12400
…-405 for tiered storage feature in Kafka. (apache#10173) KIP-405 introduces tiered storage feature in Kafka. With this feature, Kafka cluster is configured with two tiers of storage - local and remote. The local tier is the same as the current Kafka that uses the local disks on the Kafka brokers to store the log segments. The new remote tier uses systems, such as HDFS or S3 or other cloud storages to store the completed log segments. Consumers fetch the records stored in remote storage through the brokers with the existing protocol. We introduced a few SPIs for plugging in log/index store and remote log metadata store. This involves two parts 1. Storing the actual data in remote storage like HDFS, S3, or other cloud storages. 2. Storing the metadata about where the remote segments are stored. The default implementation uses an internal Kafka topic. You can read KIP for more details at https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage Reviewers: Jun Rao <junrao@gmail.com>
Reviewers: Mickael Maison <mickael.maison@gmail.com>
Only validation and unit test stages are enabled Co-authored-by: Peng.Lei <73098678+xiao-penglei@users.noreply.github.com>
…rray into a string (apache#9950) Cast SMT transformation for bytes -> string. Without this fix, the conversion becomes ByteBuffer.toString(), which always gives this useless result: "java.nio.HeapByteBuffer[pos=0 lim=4 cap=4]" With this change, the byte array is converted into a base64 string of the byte buffer content. Reviewers: Mickael Maison <mickael.maison@gmail.com>, Randall Hauch <rhauch@gmail.com>, Konstantine Karantasis <k.karantasis@gmail.com>
…becomeLeaderOrFollower` (apache#10234) This patch refactors the code, which constructs the `LeaderAndIsrResponse` in `ReplicaManager#becomeLeaderOrFollower`, to improve the readability and to remove unnecessary operations. Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
apache#9758) 1. rename INVALID_HIGHWATERMARK to INVALID_HIGH_WATERMARK 2. replace FetchResponse.AbortedTransaction by FetchResponseData.AbortedTransaction 3. remove redundant constructors from FetchResponse.PartitionData 4. rename recordSet to records 5. add helpers "recordsOrFail" and "recordsSize" to FetchResponse to process record casting Reviewers: Ismael Juma <ismael@juma.me.uk>
Reviewers: Luke Chen <showuon@gmail.com>, Dong Lin <lindong28@gmail.com>
KAFKA-12393: Document multi-tenancy considerations Addressed review feedback by @dajac and @rajinisivaram Ported from apache/kafka-site#334 Reviewers: Bill Bejeck <bbejeck@apache.org>
…10078) Reviewers: David Jacot <djacot@confluent.io>
…non api dependencies (apache#10203) Gradle 7.0 is required for Java 16 compatibility and it removes a number of deprecated APIs. Fix most issues preventing the upgrade to Gradle 7.0. The remaining ones are more complicated and should be handled in a separate PR. Details of the changes: * Release tarball no longer includes includes test, sources, javadoc and test sources jars (these are still published to the Maven Central repository). * Replace `compile` with `api` or `implementation` - note that `implementation` dependencies appear with `runtime` scope in the pom file so this is a (positive) change in behavior * Add missing dependencies that were uncovered by the usage of `implementation` * Replace `testCompile` with `testImplementation` * Replace `runtime` with `runtimeOnly` and `testRuntime` with `testRuntimeOnly` * Replace `configurations.runtime` with `configurations.runtimeClasspath` * Replace `configurations.testRuntime` with `configurations.testRuntimeClasspath` (except for the usage in the `streams` project as that causes a cyclic dependency error) * Use `java-library` plugin instead of `java` * Use `maven-publish` plugin instead of deprecated `maven` plugin - this changes the commands used to publish and to install locally, but task aliases for `install` and `uploadArchives` were added for backwards compatibility * Removed `-x signArchives` line from the readme since it was wrong (it was a no-op before and it fails now, however) * Replaces `artifacts` block with an approach that works with the `maven-publish` plugin * Don't publish `jmh-benchmark` module - the shadow jar is pretty large and not particularly useful (before this PR, we would publish the non shadow jars) * Replace `version` with `archiveVersion`, `baseName` with `archiveBaseName` and `classifier` with `archiveClassifier` * Update Gradle and plugins to the latest stable version (7.0 is not stable yet) * Use `plugin` DSL to configure plugins * Updated notable changes for 3.0 Reviewers: Chia-Ping Tsai <chia7712@gmail.com>, Randall Hauch <rhauch@gmail.com>
This patch enables delete topic support for the new KIP-500 controller. Also fixes the following: - Fix a bug where feature level records were not correctly replayed. - Fix a bug in TimelineHashMap#remove where the wrong type was being returned. Reviewers: Jason Gustafson <jason@confluent.io>, Justine Olshan <jolshan@confluent.io>, Ron Dagostino <rdagostino@confluent.io>, Chia-Ping Tsai <chia7712@gmail.com>, Jun Rao <junrao@gmail.com> Co-authored-by: Jason Gustafson <jason@confluent.io>
Topic deletions should be atomic. This fixes a build error caused by merging of both apache#10253 and apache#10184 at about the same time. Reviewers: David Arthur <mumrah@gmail.com>
…apache#10260) One gauge is missing in ReplicaManager#removeMetrics Reviewers: Chia-Ping Tsai <chia7712@gmail.com>, Anna Sophie Blee-Goldman < ableegoldman@apache.org>
…0256) This patch ensures that the constant max batch size defined in `KafkaRaftClient` is propagated to the constructed log configuration in `KafkaMetadataLog`. We also ensure that the fetch max size is set consistently with appropriate testing. Reviewers: Chia-Ping Tsai <chia7712@gmail.com>, David Arthur <mumrah@gmail.com>
Need to add missing docs for the record-e2e-latency metrics, and new TRACE recording level Reviewers: Walker Carlson <wcarlson@confluent.io>
Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>
added committed offset, high watermark and idle duration to the taskMetadata. Reviewers: Boyang Chen <boyang@confluent.io>
…pache#10240) Removed broker number checks for invalid replication factor when doing the forwarding, in order to reduce false alarms for clients. Reviewers: Jason Gustafson <jason@confluent.io>
…#10274) Reviewers: David Arthur <mumrah@gmail.com>
…ed (apache#10252) This patch implements additional handling logic for `RemoveTopic` records: - Update `MetadataPartitions` to ensure addition of deleted partitions to `localRemoved` set - Ensure topic configs are removed from `ConfigRepository` - Propagate deleted partitions to `GroupCoordinator` so that corresponding offset commits can be removed This patch also changes the controller topic id generation logic to use `Uuid.randomUuid` rather than `Random`. Reviewers: Ismael Juma <ismael@juma.me.uk>, Chia-Ping Tsai <chia7712@gmail.com>
…ing (apache#10279) The KIP-500 server requires users to run `kafka-storage.sh` to format log directories before the server will start. If the directory is not formatted, the error message complains about a missing `meta.properties` file. It is useful for the message to refer users to `kafka-storage.sh` directly since formatting is a new requirement. This patch also reduces the log level of a very spammy log message in `BrokerLifecycleManager`. Reviewers: Colin P. McCabe <cmccabe@apache.org>
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
…e#10273) Reviewers: Jason Gustafson <jason@confluent.io>, Chia-Ping Tsai <chia7712@gmail.com>
- Use consistent options for `javadoc` and `aggregatedJavadoc` - `aggregatedJavadoc` depends on `compileJava` - `connect-api` inherits `options.links` - `streams` and `streams-test-utils` javadoc exclusions should be more specific to avoid unexpected behavior in `aggregatedJavadoc` when the javadoc for multiple modules is generated together Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
…tegration tests (apache#10118) Author: Luke Chen <showuon@gmail.com> Reviewers: Chris Egerton <chrise@confluent.io>, Randall Hauch <rhauch@gmail.com>
Reviewers: Colin P. McCabe <cmccabe@apache.org>, David Arthur <mumrah@gmail.com>
…pache#10249) The KIP-500 early access release will not support creating a partition with a manual partition assignment that includes a broker that is not currently online. This patch disables system tests for Raft-based metadata quorums where the test depends on this functionality to pass. Reviewers: Colin P. McCabe <cmccabe@apache.org>
Reviewers: Mickael Maison <mickael.maison@gmail.com>, Viktor Somogyi-Vass <viktorsomogyi@gmail.com>
…pache#10268) Reviewers: Mickael Maison <mickael.maison@gmail.com>
Reviewers: Rohan Desai <rohan@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
…he build (apache#10307) This is useful when debugging build issues. I also removed two printlns that are now redundant, so this makes the build more informative and less noisy at the same time. Example output: > Starting build with version 3.0.0-SNAPSHOT using Gradle 6.8.3, Java 15 and Scala 2.13.5 Reviewers: A. Sophie Blee-Goldman <sophie@confluent.io>
…e#10310) Initially we want to be strict about the loss of committed data for the `@metadata` topic. This patch ensures that truncation below the high watermark is not allowed. Note that `MockLog` already had the logic to do so, so the patch adds a similar check to `KafkaMetadataLog`. Reviewers: David Jacot <djacot@confluent.io>, Boyang Chen <boyang@confluent.io>
…G_SHUTDOWN (apache#10311) Always invoke TaskManager#handleRevocation when the thread is in PENDING_SHUTDOWN Reviewers: Walker Carlson <wcarlson@confluent.io>
…icRemoved (apache#10303) Reviewers: David Jacot <djacot@confluent.io>, Chia-Ping Tsai <chia7712@gmail.com>
This is required to run a release as it is checked in release.py Reviewers: Konstantine Karantasis <k.karantasis@gmail.com>
…n manager (apache#10135) share forwarding channel and auto topic creation channel. Reviewers: dengziming <dengziming1993@gmail.com>, Jason Gustafson <jason@confluent.io>
…pache#10232) 1. Create a reason string to be used for INFO log entry whenever we request re-join or reset generation state. 2. Some minor cleanups. Reviewers: A. Sophie Blee-Goldman <sophie@confluent.io>
An accidental change of logging level for streams from apache#9579, correcting it. Reviewers: Bill Bejeck <bbejeck@gmail.com>
Timeout is a duration not a point in time. Reviewers: Bruno Cadonna <cadonna@confluent.io>, Anna Sophie Blee-Goldman <ableegoldman@apache.org>
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
…pache#10267) Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>
…en FetchResponse is full (apache#10318) The incremental FetchSessionCache sessions deprioritizes partitions where a response is returned. This may happen if log metadata such as log start offset, hwm, etc is returned, or if data for that partition is returned. When a fetch response fills to maxBytes, data may not be returned for partitions even if the fetch offset is lower than the fetch upper bound. However, the fetch response will still contain updates to metadata such as hwm if that metadata has changed. This can lead to degenerate behavior where a partition's hwm or log start offset is updated resulting in the next fetch being unnecessarily skipped for that partition. At first this appeared to be worse, as hwm updates occur frequently, but starvation should result in hwm movement becoming blocked, allowing a fetch to go through and then becoming unstuck. However, it'll still require one more fetch request than necessary to do so. Consumers may be affected more than replica fetchers, however they often remove partitions with fetched data from the next fetch request and this may be helping prevent starvation. Reviewers: Lucas Bradstreet <lucas@confluent.io>, Chia-Ping Tsai <chia7712@gmail.com>, Rajini Sivaram <rajinisivaram@googlemail.com>
This was useful while debugging a JDK 16 test failure, I noticed these were missing. Reviewers: David Jacot <david.jacot@gmail.com>
…ure with Raft (apache#10322) This test was failing when used with a Raft-based metadata quorum but succeeding with a ZooKeeper-based quorum. This patch increases the consumers' session timeouts to 30 seconds, which fixes the Raft case and also eliminates flakiness that has historically existed in the Zookeeper case. This patch also fixes a minor logging bug in RaftReplicaManager.endMetadataChangeDeferral() that was discovered during the debugging of this issue, and it adds an extra logging statement in RaftReplicaManager.handleMetadataRecords() when a single metadata batch is applied to mirror the same logging statement that occurs when deferred metadata changes are applied. In the Raft system test case the consumer was sometimes receiving a METADATA response with just 1 alive broker, and then when that broker rolled the consumer wouldn't know about any alive nodes. It would have to wait until the broker returned before it could reconnect, and by that time the group coordinator on the second broker would have timed-out the client and initiated a group rebalance. The test explicitly checks that no rebalances occur, so the test would fail. It turns out that the reason why the ZooKeeper configuration wasn't seeing rebalances was just plain luck. The brokers' metadata caches in the ZooKeeper configuration show 1 alive broker even more frequently than the Raft configuration does. If we tweak the metadata.max.age.ms value on the consumers we can easily get the ZooKeeper test to fail, and in fact this system test has historically been flaky for the ZooKeeper configuration. We can get the test to pass by setting session.timeout.ms=30000 (which is longer than the roll time of any broker), or we can increase the broker count so that the client never sees a METADATA response with just a single alive broker and therefore never loses contact with the cluster for an extended period of time. We have plenty of system tests with 3+ brokers, so we choose to keep this test with 2 brokers and increase the session timeout. Reviewers: Ismael Juma <ismael@juma.me.uk>
Reviewers: David Jacot <david.jacot@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>
1. Remove org.apache.log4j from allowed import list of shell, trogdor subpackage; they uses slf4j, not log4. 2. Remove org.slf4j from allowed import list of clients, server subpackage: org.slf4j is allowed globally. 3. Remove org.apache.log4j from streams subpackage's allowed import list Reviewers: David Jacot <david.jacot@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>
Conflicts: * Jenkinsfile: `install` -> `publishToMavenLocal`, drop ARM build and other changes that don't make sense for Confluent's version of `Jenkinsfile`. * build.gradle: keep Confluent changes for automatic skipping signing for specific version patterns (upstream only does it if the version ends with `SNAPSHOT`). Commits: * apache-github/trunk: (59 commits) MINOR: Remove redundant allows in import-control.xml (apache#10339) MINOR: remove some specifying types in tool command (apache#10329) KAFKA-12455: Fix OffsetValidationTest.test_broker_rolling_bounce failure with Raft (apache#10322) MINOR: Add toString to various Kafka Metrics classes (apache#10330) KAFKA-12330; FetchSessionCache may cause starvation for partitions when FetchResponse is full (apache#10318) KAFKA-12427: Don't update connection idle time for muted connections (apache#10267) MINOR; Various code cleanups (apache#10319) HOTFIX: timeout issue in removeStreamThread() (apache#10321) revert stream logging level back to ERROR (apache#10320) KAFKA-12352: Make sure all rejoin group and reset state has a reason (apache#10232) KAFKA-10348: Share client channel between forwarding and auto creation manager (apache#10135) MINOR: Update year in NOTICE (apache#10308) KAFKA-12398: Fix flaky test `ConsumerBounceTest.testClose` (apache#10243) MINOR: Remove redundant inheritance from FilteringJmxReporter #onMetricRemoved (apache#10303) KAFKA-12462: proceed with task revocation in case of thread in PENDING_SHUTDOWN (apache#10311) KAFKA-12460; Do not allow raft truncation below high watermark (apache#10310) MINOR: Log project, gradle, java and scala versions at the start of the build (apache#10307) KAFKA-10357: Add missing repartition topic validation (apache#10305) MINOR: Improve error message in MirrorConnectorsIntegrationBaseTest (apache#10268) MINOR: Add missing unit tests for Mirror Connect (apache#10192) ...
Ismael Juma (ijuma)
force-pushed
the
sync-ak-trunk-march-17
branch
from
March 17, 2021 13:31
00797c5 to
bea34ef
Compare
| doLast { | ||
| logger.info('Git hook installed successfully.') | ||
| } | ||
| } |
Member
Author
There was a problem hiding this comment.
Reformatted these two methods to follow the convention.
|
|
||
| skipSigning = project.hasProperty('skipSigning') && skipSigning.toBoolean() | ||
| shouldSign = !skipSigning && !version.endsWith("SNAPSHOT") && !version.contains("-alpha") && !version.matches("[0-9]*\\.[0-9]*\\.[0-9]*-[0-9]*-(ccs|ce)") && project.gradle.startParameter.taskNames.any { it.contains("upload") } | ||
| shouldSign = !skipSigning && !version.endsWith("SNAPSHOT") && !version.contains("-alpha") && !version.matches("[0-9]*\\.[0-9]*\\.[0-9]*-[0-9]*-(ccs|ce)") |
Member
Author
There was a problem hiding this comment.
Apart from the reformat above, this is the only other conflict. Kept the local version pattern matching.
Rohit Shekhar (rohitshekhar29)
approved these changes
Mar 17, 2021
Member
Author
|
Flaky unrelated:
|
Closed
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Conflicts:
install->publishToMavenLocal, drop ARM build andother changes that don't make sense for Confluent's version of
Jenkinsfile.for specific version patterns (upstream only does it if the version ends
with
SNAPSHOT).Commits:
MINOR: Remove redundant allows in import-control.xml (MINOR: Remove redundant allows in import-control.xml apache/kafka#10339)
MINOR: remove some specifying types in tool command (MINOR: remove some specifying types in tool command apache/kafka#10329)
KAFKA-12455: Fix OffsetValidationTest.test_broker_rolling_bounce
failure with Raft (KAFKA-12455: OffsetValidationTest.test_broker_rolling_bounce fail: Raft apache/kafka#10322)
MINOR: Add toString to various Kafka Metrics classes (MINOR: Add toString to various Kafka Metrics classes apache/kafka#10330)
KAFKA-12330; FetchSessionCache may cause starvation for partitions
when FetchResponse is full (KAFKA-12330; FetchSessionCache may cause starvation for partitions when FetchResponse is full apache/kafka#10318)
KAFKA-12427: Don't update connection idle time for muted connections
(KAFKA-12427: Don't update connection idle time for muted connections apache/kafka#10267)
MINOR; Various code cleanups (MINOR; Various code cleanups apache/kafka#10319)
HOTFIX: timeout issue in removeStreamThread() (HOTFIX: timeout issue in Remove thread apache/kafka#10321)
revert stream logging level back to ERROR (MINOR: revert stream logging level back to ERROR apache/kafka#10320)
KAFKA-12352: Make sure all rejoin group and reset state has a reason
(KAFKA-12352: Make sure all rejoin group and reset state has a reason apache/kafka#10232)
KAFKA-10348: Share client channel between forwarding and auto creation
manager (KAFKA-10348: Share client channel between forwarding and auto creation manager apache/kafka#10135)
MINOR: Update year in NOTICE (MINOR: Update year in NOTICE apache/kafka#10308)
KAFKA-12398: Fix flaky test
ConsumerBounceTest.testClose(KAFKA-12398: Fix flaky testConsumerBounceTest.testCloseapache/kafka#10243)MINOR: Remove redundant inheritance from FilteringJmxReporter
KAFKA-12462: proceed with task revocation in case of thread in
PENDING_SHUTDOWN (KAFKA-12462: proceed with task revocation in case of thread in PENDING_SHUTDOWN apache/kafka#10311)
KAFKA-12460; Do not allow raft truncation below high watermark
(KAFKA-12460; Do not allow raft truncation below high watermark apache/kafka#10310)
MINOR: Log project, gradle, java and scala versions at the start of
the build (MINOR: Log project, gradle, java and scala versions at the start of the build apache/kafka#10307)
KAFKA-10357: Add missing repartition topic validation (KAFKA-10357: Add missing repartition topic validation apache/kafka#10305)
MINOR: Improve error message in MirrorConnectorsIntegrationBaseTest
(MINOR: output meaningful error message apache/kafka#10268)
MINOR: Add missing unit tests for Mirror Connect (MINOR: Add missing unit tests for Mirror Connect apache/kafka#10192)
...
Committer Checklist (excluded from commit message)