Merge master 3/10/2021#531
Closed
Walker Carlson (wcarlson5) wants to merge 31 commits into
Closed
Conversation
Reviewers: A. Sophie Blee-Goldman <sophie@confluent.io>, Chia-Ping Tsai <chia7712@gmail.com>
Here is the fix. The reason of [CVE-2020-27223](https://nvd.nist.gov/vuln/detail/CVE-2020-27223) was DOS vulnerability for Quoted Quality CSV headers and [patched in 9.4.37.v20210219](GHSA-m394-8rww-3jr7). This PR updates Jetty dependency into the following version, 9.4.38.v20210224. Author: Lee Dongjin <dongjin@apache.org> Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com> Closes apache#10245 from dongjinleekr/feature/KAFKA-12400
…-405 for tiered storage feature in Kafka. (apache#10173) KIP-405 introduces tiered storage feature in Kafka. With this feature, Kafka cluster is configured with two tiers of storage - local and remote. The local tier is the same as the current Kafka that uses the local disks on the Kafka brokers to store the log segments. The new remote tier uses systems, such as HDFS or S3 or other cloud storages to store the completed log segments. Consumers fetch the records stored in remote storage through the brokers with the existing protocol. We introduced a few SPIs for plugging in log/index store and remote log metadata store. This involves two parts 1. Storing the actual data in remote storage like HDFS, S3, or other cloud storages. 2. Storing the metadata about where the remote segments are stored. The default implementation uses an internal Kafka topic. You can read KIP for more details at https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage Reviewers: Jun Rao <junrao@gmail.com>
Reviewers: Mickael Maison <mickael.maison@gmail.com>
Only validation and unit test stages are enabled Co-authored-by: Peng.Lei <73098678+xiao-penglei@users.noreply.github.com>
…rray into a string (apache#9950) Cast SMT transformation for bytes -> string. Without this fix, the conversion becomes ByteBuffer.toString(), which always gives this useless result: "java.nio.HeapByteBuffer[pos=0 lim=4 cap=4]" With this change, the byte array is converted into a base64 string of the byte buffer content. Reviewers: Mickael Maison <mickael.maison@gmail.com>, Randall Hauch <rhauch@gmail.com>, Konstantine Karantasis <k.karantasis@gmail.com>
…becomeLeaderOrFollower` (apache#10234) This patch refactors the code, which constructs the `LeaderAndIsrResponse` in `ReplicaManager#becomeLeaderOrFollower`, to improve the readability and to remove unnecessary operations. Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
apache#9758) 1. rename INVALID_HIGHWATERMARK to INVALID_HIGH_WATERMARK 2. replace FetchResponse.AbortedTransaction by FetchResponseData.AbortedTransaction 3. remove redundant constructors from FetchResponse.PartitionData 4. rename recordSet to records 5. add helpers "recordsOrFail" and "recordsSize" to FetchResponse to process record casting Reviewers: Ismael Juma <ismael@juma.me.uk>
Reviewers: Luke Chen <showuon@gmail.com>, Dong Lin <lindong28@gmail.com>
KAFKA-12393: Document multi-tenancy considerations Addressed review feedback by @dajac and @rajinisivaram Ported from apache/kafka-site#334 Reviewers: Bill Bejeck <bbejeck@apache.org>
…10078) Reviewers: David Jacot <djacot@confluent.io>
…non api dependencies (apache#10203) Gradle 7.0 is required for Java 16 compatibility and it removes a number of deprecated APIs. Fix most issues preventing the upgrade to Gradle 7.0. The remaining ones are more complicated and should be handled in a separate PR. Details of the changes: * Release tarball no longer includes includes test, sources, javadoc and test sources jars (these are still published to the Maven Central repository). * Replace `compile` with `api` or `implementation` - note that `implementation` dependencies appear with `runtime` scope in the pom file so this is a (positive) change in behavior * Add missing dependencies that were uncovered by the usage of `implementation` * Replace `testCompile` with `testImplementation` * Replace `runtime` with `runtimeOnly` and `testRuntime` with `testRuntimeOnly` * Replace `configurations.runtime` with `configurations.runtimeClasspath` * Replace `configurations.testRuntime` with `configurations.testRuntimeClasspath` (except for the usage in the `streams` project as that causes a cyclic dependency error) * Use `java-library` plugin instead of `java` * Use `maven-publish` plugin instead of deprecated `maven` plugin - this changes the commands used to publish and to install locally, but task aliases for `install` and `uploadArchives` were added for backwards compatibility * Removed `-x signArchives` line from the readme since it was wrong (it was a no-op before and it fails now, however) * Replaces `artifacts` block with an approach that works with the `maven-publish` plugin * Don't publish `jmh-benchmark` module - the shadow jar is pretty large and not particularly useful (before this PR, we would publish the non shadow jars) * Replace `version` with `archiveVersion`, `baseName` with `archiveBaseName` and `classifier` with `archiveClassifier` * Update Gradle and plugins to the latest stable version (7.0 is not stable yet) * Use `plugin` DSL to configure plugins * Updated notable changes for 3.0 Reviewers: Chia-Ping Tsai <chia7712@gmail.com>, Randall Hauch <rhauch@gmail.com>
This patch enables delete topic support for the new KIP-500 controller. Also fixes the following: - Fix a bug where feature level records were not correctly replayed. - Fix a bug in TimelineHashMap#remove where the wrong type was being returned. Reviewers: Jason Gustafson <jason@confluent.io>, Justine Olshan <jolshan@confluent.io>, Ron Dagostino <rdagostino@confluent.io>, Chia-Ping Tsai <chia7712@gmail.com>, Jun Rao <junrao@gmail.com> Co-authored-by: Jason Gustafson <jason@confluent.io>
Topic deletions should be atomic. This fixes a build error caused by merging of both apache#10253 and apache#10184 at about the same time. Reviewers: David Arthur <mumrah@gmail.com>
…apache#10260) One gauge is missing in ReplicaManager#removeMetrics Reviewers: Chia-Ping Tsai <chia7712@gmail.com>, Anna Sophie Blee-Goldman < ableegoldman@apache.org>
…0256) This patch ensures that the constant max batch size defined in `KafkaRaftClient` is propagated to the constructed log configuration in `KafkaMetadataLog`. We also ensure that the fetch max size is set consistently with appropriate testing. Reviewers: Chia-Ping Tsai <chia7712@gmail.com>, David Arthur <mumrah@gmail.com>
Need to add missing docs for the record-e2e-latency metrics, and new TRACE recording level Reviewers: Walker Carlson <wcarlson@confluent.io>
Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>
added committed offset, high watermark and idle duration to the taskMetadata. Reviewers: Boyang Chen <boyang@confluent.io>
…pache#10240) Removed broker number checks for invalid replication factor when doing the forwarding, in order to reduce false alarms for clients. Reviewers: Jason Gustafson <jason@confluent.io>
…#10274) Reviewers: David Arthur <mumrah@gmail.com>
…ed (apache#10252) This patch implements additional handling logic for `RemoveTopic` records: - Update `MetadataPartitions` to ensure addition of deleted partitions to `localRemoved` set - Ensure topic configs are removed from `ConfigRepository` - Propagate deleted partitions to `GroupCoordinator` so that corresponding offset commits can be removed This patch also changes the controller topic id generation logic to use `Uuid.randomUuid` rather than `Random`. Reviewers: Ismael Juma <ismael@juma.me.uk>, Chia-Ping Tsai <chia7712@gmail.com>
…ing (apache#10279) The KIP-500 server requires users to run `kafka-storage.sh` to format log directories before the server will start. If the directory is not formatted, the error message complains about a missing `meta.properties` file. It is useful for the message to refer users to `kafka-storage.sh` directly since formatting is a new requirement. This patch also reduces the log level of a very spammy log message in `BrokerLifecycleManager`. Reviewers: Colin P. McCabe <cmccabe@apache.org>
Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
…e#10273) Reviewers: Jason Gustafson <jason@confluent.io>, Chia-Ping Tsai <chia7712@gmail.com>
- Use consistent options for `javadoc` and `aggregatedJavadoc` - `aggregatedJavadoc` depends on `compileJava` - `connect-api` inherits `options.links` - `streams` and `streams-test-utils` javadoc exclusions should be more specific to avoid unexpected behavior in `aggregatedJavadoc` when the javadoc for multiple modules is generated together Reviewers: Chia-Ping Tsai <chia7712@gmail.com>
…tegration tests (apache#10118) Author: Luke Chen <showuon@gmail.com> Reviewers: Chris Egerton <chrise@confluent.io>, Randall Hauch <rhauch@gmail.com>
Reviewers: Colin P. McCabe <cmccabe@apache.org>, David Arthur <mumrah@gmail.com>
…pache#10249) The KIP-500 early access release will not support creating a partition with a manual partition assignment that includes a broker that is not currently online. This patch disables system tests for Raft-based metadata quorums where the test depends on this functionality to pass. Reviewers: Colin P. McCabe <cmccabe@apache.org>
Member
|
I only saw this now, #536 includes the changes here and a few more commits. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
18 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
More detailed description of your change,
if necessary. The PR title and PR message become
the squashed commit message, so use a separate
comment to ping reviewers.
Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.
Committer Checklist (excluded from commit message)