synck wit origin#1
Merged
Merged
Conversation
This PR improves the JavaDoc of the transformer interface. Reviewers: Guozhang Wang <guozhang@confluent.io>, John Roesler <john@confluent.io>
Part 1 of the suppression API. * add the DSL suppress method and config objects * add the processor, but only in "identity" mode (i.e., it will forward only if the suppression spec says to forward immediately) * add tests Reviewers: Matthias J. Sax <matthias@confluent.io>, Bill Bejeck <bill@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
…o documentation Author: Manikumar Reddy <manikumar.reddy@gmail.com> Reviewers: Dong Lin <lindong28@gmail.com> Closes #5563 from omkreddy/KAFKA-5066-KAFKA-METRICS-CONFIG
This contains important fixes: https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.9.7 Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>
…t commit value (#5690) This fixes a regression caused by KAFKA-4682 (KIP-211) which caused offset commit failures after upgrading from an older version which used the v1 inter-broker format.
This is Part 2 of suppression. Part 1 was #5567 In an effort to control the scope of the review, this PR is just the tests for buffered suppression. Reviewers: Bill Bejeck <bill@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Reviewers: Mickael Maison <mickael.maison@gmail.com>, Rajini Sivaram <rajinisivaram@googlemail.com>
This commit implements the changes described in KIP-320 for the persistence of leader epoch information in the offset commit protocol. Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>
… altering configs (#5651) Values for `message.format.version` and `log.message.format.version` should be verified before topic creation or config change. Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
…ut (#5712) Reviewers: Jun Rao <junrao@gmail.com>. Ismael Juma <ismael@juma.me.uk>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
…l them on close (#5713) Lazy allocation helps when there are a large number of connections that have been accepted, but where no data has been received from the clients. Each buffer is often around 16k (max TLS record size). Nulling the buffers should not make a difference in the current implementation since we release the reference to the channel and transport layer after we close them, but it's a good practice to release medium/large buffers after `close` is called. Reviewers: Jun Rao <junrao@gmail.com>, Ismael Juma <ismael@juma.me.uk>
*More detailed description of your change, if necessary. The PR title and PR message become the squashed commit message, so use a separate comment to ping reviewers.* *Summary of testing strategy (including rationale) for the feature or bug fix. Unit and/or integration tests are expected for any behaviour change and system tests should be considered for larger changes.* Author: Michał Borowiecki <mbor81@gmail.com> Reviewers: Arjun Satish <arjun@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io> Closes #5700 from mihbor/KAFKA-7434
Highlights: * 10% compilation speed improvement in Scala 2.12.7: https://www.scala-lang.org/news/2.12.7 * 10% better decompression speed in lz4 1.8.2 (lz4-java 1.5.0 includes lz4 1.8.3): https://github.com/lz4/lz4-java/blob/master/CHANGES.md#150 https://github.com/lz4/lz4/releases Author: Ismael Juma <ismael@juma.me.uk> Reviewers: Dong Lin <lindong28@gmail.com> Closes #5715 from ijuma/scala-2.12.7-and-other-updates
Reviewers: Matthias J. Sax <matthias@confluent.io>
…5710) The Suppression buffer stores the full record context, not just the key and value, so its changelog/restore loop will also need to preserve this information. This change is a precondition to that, creating an option to register a state restore callback to receive the full consumer record. Reviewers: Bill Bejeck <bill@confluent.io>, Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
Reviewers: Bill Bejeck <bill@confluent.io>, Matthias J. Sax <matthias@confluent.io>
…al (#5717) As part of KAFKA-6514, the `apiVersion` tag was added to the `RequestsPerSec` metric. A thread unsafe `HashMap` was used in the implementation even though it can be accessed by multiple threads. Fix it by replacing it with the thread-safe `Pool`. Reviewers: Ismael Juma <ismael@juma.me.uk>
Switches to normal year format instead of week date years and day of month instead of day of year. This is directly from #4820, but separated into a different JIRA/PR to keep the fixes independent. Original authorship should be maintained in the commit. Author: Amit Sela <amitsela33@gmail.com> Reviewers: Ewen Cheslack-Postava <ewen@confluent.io> Closes #5718 from ewencp/fix-header-converter-date-format
Allow to cast LogicalType to string by calling the serialized (Java) object's toString(). Added tests for `BigDecimal` and `Date` as whole record and as fields. Author: Amit Sela <amitsela33@gmail.com> Reviewers: Randall Hauch <rhauch@gmail.com>, Robert Yokota <rayokota@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io> Closes #4820 from amitsela/cast-transform-bytes
Reviewers: John Roesler <john@confluent.io>, Bill Bejeck <bill@confluent.io>, Matthias J. Sax <matthias@confluent.io>
Reviewer: John Roesler <john@confluent.io>, Guozhang Wang <guozhang@confluent.io>, Matthias J. Sax <matthias@confluent.io>
Reviewer: Bill Bejeck <bill@confluent.io>, Guozhang Wang <guozhang@confluent.io>, Matthias J. Sax <matthias@confluent.io>
Reviewers: John Roesler <john@confluent.io>, Matthias J Sax <matthias@confluent.io>
…() and onJoinPrepare() `ConsumerCoordinator.onJoinPrepare()` currently makes multiple copies of the set of assigned partitions. We can let `subscriptions.assignedPartitions()` return a view of the underlying partition set, copy it only once and re-use the copied value. Author: radai-rosenblatt <radai.rosenblatt@gmail.com> Reviewers: Ismael Juma <ismael@juma.me.uk>, Dong Lin <lindong28@gmail.com> Closes #5124 from radai-rosenblatt/copy-all-the-things
In some tests, the check monitoring the JMX tool log output doesn’t quite wait long enough before failing. Increasing the timeout from 10 to 20 seconds.
This is Part 4 of suppression (durability) Part 1 was #5567 (the API) Part 2 was #5687 (the tests) Part 3 was #5693 (in-memory buffering) Implement a changelog for the suppression buffer so that the buffer state may be recovered on restart or recovery. As of this PR, suppression is suitable for general usage. Reviewers: Bill Bejeck <bill@confluent.io>, Guozhang Wang <guozhang@confluent.io>, Matthias J. Sax <matthias@confluent.io>
…shListener (#6017) This is a follow-up PR from the previous PR #5779, where KTabeSource always get old values from the store even if sendOldValues. It gets me to make a pass over all the KTable/KStreamXXX processor to push the sendOldValues at the callers in order to avoid unnecessary store reads. More details: ForwardingCacheFlushListener and TupleForwarder both need sendOldValues as parameters. a. For ForwardingCacheFlushListener it is not needed at all, since its callers XXXCachedStore already use the sendOldValues values passed from TupleForwarder to avoid getting old values from underlying stores. b. For TupleForwarder, it actually only need to pass the boolean flag to the cached store; and then it does not need to keep it as its own variable since the cached store already respects the boolean to pass null or the actual value.. The only other minor bug I found from the pass in on KTableJoinMerge, where we always pass old values and ignores sendOldValues. Reviewers: Matthias J. Sax <mjsax@apache.org>
…error to client (#5925) Older versions of the Produce API should return an error if zstd is used. This validation existed, but it was done during request parsing, which means that instead of returning an error code, the broker disconnected. This patch fixes the issue by moving the validation outside of the parsing logic. It also fixes several other record validations which had the same problem. Reviewers: Jason Gustafson <jason@confluent.io>
…ayed (#5962) When a consumer first joins a group, it doesn't have an assigned memberId. If the rebalance is delayed for some reason, the client may disconnect after a request timeout and retry. Since the client had not received its memberId, then we do not have a way to detect the retry and expire the previously generated member id. This can lead to unbounded growth in the size of the group until the rebalance has completed. This patch fixes the problem by proactively completing all JoinGroup requests for new members after a timeout of 5 minutes. If the client is still around, we expect it to retry. Reviewers: Stanislav Kozlovski <stanislav_kozlovski@outlook.com>, Boyang Chen <bchen11@outlook.com>, Guozhang Wang <wangguoz@gmail.com>
…hangelog topic when start offset of local checkpoint is smaller than that of changelog topic (#5946) Reviewer: Matthias J. Sax <matthias@confluent.io>, John Roesler <john@confluent.io>
…6016) Reviewer: Matthias J. Sax <matthias@confluent.io>, John Roesler <john@confluent.io>
Reviewers: Guozhang Wang <guozhang@confluent.io>, Nikolay Izhikov <nIzhikov@gmail.com>, Ismael Juma <ismael@confluent.io>, Bill Bejeck <bill@confluent.io>
Reviewers: Jason Gustafson <jason@confluent.io>
The example in the producer's javadoc contained an inconsistent value for `delivery.timeout.ms`. This patch removes the inconsistent config and several unnecessary overrides in order to simplify the example. Reviewers: huxi <huxi_2b@hotmail.com>, Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
…#6023) Ensure that channel and selection keys are removed from `Selector` collections before propagating connect exceptions. They are currently cleared on the next `poll()`, but we can't ensure that callers (NetworkClient for example) wont try to connect again before the next `poll` and hence we should clear the collections before re-throwing exceptions from `connect()`. Reviewers: Jason Gustafson <jason@confluent.io>
Document the new metrics added in #5795 Reviewers: Bill Bejeck <bill@confluent.io>, Guozhang Wang <wangguoz@gmail.com>
…ircumstances, potential DOS (#5929) Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <guozhang@confluent.io>
Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <guozhang@confluent.io>, Bill Bejeck <bill@confluent.io>
This PR fixes a flaky system test. I ran six runs of branch builder, and each run was parameterized to repeat the test 25 times for a total of 150 runs. All test runs passed. https://jenkins.confluent.io/job/system-test-kafka-branch-builder/2122/ https://jenkins.confluent.io/job/system-test-kafka-branch-builder/2123/ https://jenkins.confluent.io/job/system-test-kafka-branch-builder/2124/ https://jenkins.confluent.io/job/system-test-kafka-branch-builder/2128/ https://jenkins.confluent.io/job/system-test-kafka-branch-builder/2129/ https://jenkins.confluent.io/job/system-test-kafka-branch-builder/2130/ Reviewers: Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <guozhang@confluent.io>, John Roesler <vvcephei@users.noreply.github.com>
#6021) Updating the documentation for table operation because I believe it is incorrect. In PR #5163 the table operation stopped disabling the changelog topic by default and instead moved that optimization to a configuration that is not enabled by default. This PR updates the documentation to reflect the change in behavior and point to the new configuration for optimization. Reviewers: Bill Bejeck <bbejeck@gmail.com>, Guozhang Wang <wangguoz@gmail.com>
Reviewers: Jason Gustafson <jason@confluent.io>
After a recent leader election, the leaders high-water mark might lag behind the offset at the beginning of the new epoch (as well as the previous leader's HW). This can lead to offsets going backwards from a client perspective, which is confusing and leads to strange behavior in some clients. This change causes Partition#fetchOffsetForTimestamp to throw an exception to indicate the offsets are not yet available from the leader. For new clients, a new OFFSET_NOT_AVAILABLE error is added. For existing clients, a LEADER_NOT_AVAILABLE is thrown. This is an implementation of [KIP-207](https://cwiki.apache.org/confluence/display/KAFKA/KIP-207%3A+Offsets+returned+by+ListOffsetsResponse+should+be+monotonically+increasing+even+during+a+partition+leader+change). Reviewers: Colin Patrick McCabe <colin@cmccabe.xyz>, Dhruvil Shah <dhruvil@confluent.io>, Jason Gustafson <jason@confluent.io>
Reviewers: Bill Bejeck <bill@confluent.io>, Guozhang Wang <guozhang@confluent.io>
This patch adds additional information in the log message after a fetch failure to make debugging easier. Reviewers: David Arthur <mumrah@gmail.com>
This PR addresses a few issues with this system test flakiness. This PR is a cherry-picked duplicate of #6041 but for the trunk branch, hence I won't repeat the inline comments here. 1. Need to grab the monitor before a given operation to observe logs for signal 2. Relied too much on a timely rebalance and only sent a handful of messages. I've updated the test and ran it here https://jenkins.confluent.io/job/system-test-kafka-branch-builder/2143/ parameterized for 15 repeats all passed. Reviewers: Guozhang Wang <wangguoz@gmail.com>
…DelegationTokenCache Author: Satish Duggana <satishd@apache.org> Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com> Closes #6037 from satishd/KAFKA-7742
…n timeout) API Author: Matthias Wessendorf <mwessend@redhat.com> Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com> Closes #6052 from matzew/use_new_poll_api
This patch disables support for WADL output in the Connect REST API since it was never intended to be exposed. Reviewers: Randall Hauch <rhauch@gmail.com>, Jason Gustafson <jason@confluent.io>
Switch to lambda when ever possible instead of old anonymous way in tools module Author: Srinivas Reddy <srinivas96alluri@gmail.com> Author: Srinivas Reddy <mrsrinivas@users.noreply.github.com> Reviewers: Ryanne Dolan <ryannedolan@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Manikumar Reddy <manikumar.reddy@gmail.com> Closes #6013 from mrsrinivas/tools-switch-to-java8
Change-Id: Ia44c6c659418bbed5367645b814725365daba820
…xception **User Interface Improvement :** If topic doesn't exist then Kafka describe command should throw topic doesn't exist exception, like alter and delete commands Author: Manohar Vanam <manohar.crazy09@gmail.com> Reviewers: Vahid Hashemian <vahid.hashemian@gmail.com>, Jason Gustafson <jason@confluent.io>, Manikumar Reddy <manikumar.reddy@gmail.com> Closes #5211 from ManoharVanam/KAFKA-7054
Previous PR #6043 reduced throughput for VerifiableProducer in base class, but the streams_standby_replica_test needs higher throughput for consumer to complete verification in 60 seconds Update system test and kicked off branch builder with 25 repeats https://jenkins.confluent.io/job/system-test-kafka-branch-builder/2201/ Reviewers: Guozhang Wang <wangguoz@gmail.com>
The topology optimization test was getting intermittent failures because of failures to create repartition topics on startup. This PR Increased admin client retries I kicked off the system test with 25 repeats, all passed http://confluent-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/2018-12-21--001.1545436859--bbejeck--MINOR_flaky_optimization_test_create_repartition_fails--6cd55e2/report.html Reviewers: Guozhang Wang <wangguoz@gmail.com>
…ions enabled (#6050) Right now if a repartition is required and users choose to name the repartition topic for an aggregation i.e. kGroupedStream = builder.<String, String>stream("topic").selectKey((k, v) -> k).groupByKey(Grouped.as("grouping")); The resulting KGroupedStream can't be reused with optimizations are disabled, as Streams will attempt to create two repartiton topics with the same name. However, if optimizations are enabled then the resulting KGroupedStream can be re-used For example the following will work if optimizations are enabled. This PR provides a unit test proving as much. Reviewers: Matthias J. Sax <mjsax@apache.org>, Guozhang Wang <wangguoz@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
19 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
More detailed description of your change,
if necessary. The PR title and PR message become
the squashed commit message, so use a separate
comment to ping reviewers.
Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.
Committer Checklist (excluded from commit message)