Merge trunk, refactor consumer/producer tests and other fixes and improvements#6
Merged
Merged
Conversation
This also adds some other needed infrastructure for distributed Copycat, most importantly the DistributedHerder, and refactors some code for handling Kafka-backed logs into KafkaBasedLog since this is shared betweeen offset and config storage. Author: Ewen Cheslack-Postava <me@ewencp.org> Reviewers: Gwen Shapira, James Cheng Closes apache#241 from ewencp/kafka-2372-copycat-distributed-config
granders Can you take a look at this quota system test? Author: Dong Lin <lindong28@gmail.com> Reviewers: Geoff Anderson, Ewen Cheslack-Postava Closes apache#275 from lindong28/KAFKA-2527
guozhangwang Fix the order of flushing. Undoing the change I did sometime ago. Author: Yasuhiro Matsuda <yasuhiro@confluent.io> Reviewers: Guozhang Wang Closes apache#304 from ymatsuda/flush_order
Author: Guozhang Wang <wangguoz@gmail.com> Reviewers: Jason Gustafson Closes apache#295 from guozhangwang/K2632
Author: Jason Gustafson <jason@confluent.io> Reviewers: Gwen Shapira Closes apache#306 from hachikuji/handle-wakeup-in-consumer-close
Added --timeout-ms argument to ConsoleConsumer that works with both old and new consumer. Also modified ducktape ConsoleConsumer service to use this arg instead of consumer.timeout.ms config that works only with the old consumer. Author: Rajini Sivaram <rajinisivaram@googlemail.com> Reviewers: Aditya Auradkar, Ismael Juma, Guozhang Wang Closes apache#274 from rajinisivaram/KAFKA-2603
…alizers Add support for the key value stores to use specified serializers and deserializers (aka, "serdes"). Prior to this change, the stores were limited to only the default serdes specified in the topology's configuration and exposed to the processors via the ProcessorContext. Now, using InMemoryKeyValueStore and RocksDBKeyValueStore are similar: both are parameterized on the key and value types, and both have similar multiple static factory methods. The static factory methods either take explicit key and value serdes, take key and value class types so the serdes can be inferred (only for the built-in serdes for string, integer, long, and byte array types), or use the default serdes on the ProcessorContext. Author: Randall Hauch <rhauch@gmail.com> Reviewers: Guozhang Wang Closes apache#255 from rhauch/kafka-2593
…h TopicCommand Author: Grant Henke <granthenke@gmail.com> Reviewers: Gwen Shapira Closes apache#308 from granthenke/configcommand
This is a minimal revert of some backward incompatible changes made in KAFKA-2205, with the addition of the deprecation logging message. Author: Grant Henke <granthenke@gmail.com> Reviewers: Gwen Shapira Closes apache#305 from granthenke/topic-configs
Author: Guozhang Wang <wangguoz@gmail.com> Reviewers: Gwen Shapira Closes apache#311 from guozhangwang/wakeupComments
guozhangwang This change aims to remove unnecessary ```consumer.poll(0)``` calls. * ```once``` after some partition is resumed * whenever the size of the top queue in any task is below ```BUFFERED_RECORDS_PER_PARTITION_CONFIG``` Author: Yasuhiro Matsuda <yasuhiro@confluent.io> Reviewers: Guozhang Wang Closes apache#315 from ymatsuda/less_poll_zero
Rebased code.. Author: Manikumar reddy O <manikumar.reddy@gmail.com> Reviewers: Guozhang Wang Closes apache#314 from omkreddy/KAFKA-2295
guozhangwang StreamTaskTest did not set up a temp directory for each test. This occasionally caused interference between tests through state directory locking. Author: Yasuhiro Matsuda <yasuhiro@confluent.io> Reviewers: Guozhang Wang Closes apache#317 from ymatsuda/fix_StreamTaskTest
Author: Ashish Singh <asingh@cloudera.com> Reviewers: Guozhang Wang Closes apache#297 from SinghAsDev/KAFKA-2487
Trivial fix to get rid of unused statements in kafkaProducer. Author: Mayuresh Gharat <mgharat@mgharat-ld1.linkedin.biz> Reviewers: Edward Ribeiro, Guozhang Wang Closes apache#320 from MayureshGharat/kafka-2120-followup
…en not required See here for more discussion: https://issues.apache.org/jira/browse/KAFKA-2419 Basically, the fix involves adding a param to Metrics to indicate if it is capable of metric cleanup or not. Author: Aditya Auradkar <aauradka@aauradka-mn1.linkedin.biz> Reviewers: Jun Rao <junrao@gmail.com> Closes apache#323 from auradkar/KAFKA-2419-fix
…nding in-memory stores Added a new `KeyValueStore` implementation called `InMemoryLRUCacheStore` that keeps a maximum number of entries in-memory, and as the size exceeds the capacity the least-recently used entry is removed from the store and the backing topic. Also added unit tests for this new store and the existing `InMemoryKeyValueStore` and `RocksDBKeyValueStore` implementations. A new `KeyValueStoreTestDriver` class simplifies all of the other tests, and can be used by other libraries to help test their own custom implementations. This PR depends upon [KAFKA-2593](https://issues.apache.org/jira/browse/KAFKA-2593) and its PR at apache#255. Once that PR is merged, I can rebase this PR if desired. Two issues were uncovered when creating these new unit tests, and both are also addressed as separate (small) commits in this PR: * The `RocksDBKeyValueStore` initialization was not creating the file system directory if missing. * `MeteredKeyValueStore` was casting to `ProcessorContextImpl` to access the `RecordCollector`, which prevent using `MeteredKeyValueStore` implementations in tests where something other than `ProcessorContextImpl` was used. The fix was to introduce a `RecordCollector.Supplier` interface to define this `recordCollector()` method, and change `ProcessorContextImpl` and `MockProcessorContext` to both implement this interface. Now, `MeteredKeyValueStore` can cast to the new interface to access the record collector rather than to a single concrete implementation, making it possible to use any and all current stores inside unit tests. Author: Randall Hauch <rhauch@gmail.com> Reviewers: Edward Ribeiro, Guozhang Wang Closes apache#256 from rhauch/kafka-2594
This PR adds schema projection utilities to copycat. Author: Liquan Pei <liquanpei@gmail.com> Reviewers: Ewen Cheslack-Postava Closes apache#307 from Ishiihara/schema-projection
Author: Guozhang Wang <wangguoz@gmail.com> Reviewers: Dong Lin, Jun Rao Closes apache#318 from guozhangwang/K2515
…lance Let's say every consumer in a group has session timeout s. Currently, if a consumer leaves the group, the worst case time to stabilize the group is 2s (s to detect the consumer failure + s for the rebalance window). If a consumer instead can declare they are leaving the group, the worst case time to stabilize the group would just be the s associated with the rebalance window. This is a low priority optimization! Author: Onur Karaman <okaraman@linkedin.com> Reviewers: Jason Gustafson, Guozhang Wang Closes apache#103 from onurkaraman/leave-group
…art of the code github Author: Gwen Shapira <cshapi@gmail.com> Reviewers: Guozhang Wang Closes apache#325 from gwenshap/KAFKA-2665
* apache/trunk: (21 commits) KAFKA-2665: Add images to code github KAFKA-2397: add leave group request to force coordinator trigger rebalance KAFKA-2515: Handle oversized messages properly in new consumer KAFKA-2484: Add schema projection utilities KAFKA-2594: Add InMemoryLRUCacheStore as a preliminary method for bounding in-memory stores KAFKA-2419 - Fix to prevent background thread from getting created when not required KAFKA-2120: cleaning follow-up KAFKA-2487: change kafka.examples.Consumer to use the new java consumer MINOR: set up temp directories properly in StreamTaskTest KAFKA-2295; Support dynamically loaded classes from context class loader KAFKA-2654: optimize unnecessary poll(0) away in StreamTask TRIVIAL: add @throws ConsumerWakeupException in KafkaConsumer KAFKA-2536: topics tool should allow users to alter topic configuration KAFKA-2650: Change ConfigCommand --deleted-config option to align wit… KAFKA-2593: Key value stores can use specified serializers and deserializers KAFKA-2603: Add timeout arg to ConsoleConsumer for new consumer MINOR: ignore wakeups when committing offsets on consumer close KAFKA-2632: move fetchable check ahead in handleFetchResponse MINOR: flush record collector after local state flush KAFKA-2527; System Test for Quotas in Ducktape ...
…d fix bug in `ClientUtils.createChannelBuilder`
I've split the work of KAFKA-1695 because this refactoring touches a large number of files. Most of the changes are trivial, but I feel it will be easier to review this way. This pull request includes the one Parth-Brahmbhatt started to address KAFKA-1695. Author: flavio junqueira <fpj@apache.org> Author: Flavio Junqueira <fpj@apache.org> Reviewers: Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com> Closes apache#303 from fpj/KAFKA-2639
Author: GabrielNicolasAvellaneda <avellaneda.gabriel@gmail.com> Reviewers: Ismael Juma, Ewen Cheslack-Postava Closes apache#268 from GabrielNicolasAvellaneda/readme-updates
…ocols Include improvements from `SaslIntegrationTest` and remove duplicated code.
As pointed out by Jun
* apache/trunk: MINOR: Fixed README examples on running specific tests. KAFKA-2639: Refactoring of ZkUtils
…error` to `warn` Before we switched from `BlockingChannel` to `NetworkClient`, we were always reporting a successful connection due to the fact that `BlockingChannel.connect` catches and swallows all exceptions. We are now reporting failures (which is better), but `error` seems too noisy (as can be seen in our tests). Author: Ismael Juma <ismael@juma.me.uk> Reviewers: Jun Rao <junrao@gmail.com> Closes apache#280 from ijuma/reduce-connection-failure-logging-level
Removed default hardcoded keystore and truststore in /tmp so that default JVM keystore/truststore may be used when keystore/truststore is not specified in Kafka server or client properties Author: Rajini Sivaram <rajinisivaram@googlemail.com> Reviewers: Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com> Closes apache#312 from rajinisivaram/KAFKA-2656
LogCleanerIntegrationTest calls LogCleaner.awaitCleaned() to wait until cleaner has processed up to given offset. However, existing awaitCleaned() implementation doesn't wait for this. This patch fix the problem. Author: Dong Lin <lindong@cis.upenn.edu> Author: Dong Lin <lindong28@gmail.com> Reviewers: Ismael Juma, Guozhang Wang Closes apache#327 from lindong28/KAFKA-2669
Author: Ewen Cheslack-Postava <me@ewencp.org> Reviewers: Guozhang Wang Closes apache#331 from ewencp/minor-capture-consumer-performance-stderr
Author: Guozhang Wang <wangguoz@gmail.com> Reviewers: Ewen Cheslack-Postava, Guozhang Wang Closes apache#332 from guozhangwang/K2515-hotfix
* apache/trunk: HOTFIX: check logic of KAFKA-2515 should be on buffer.limit() MINOR: Capture stderr in ConsumerPerformanceService. KAFKA-2669; Fix LogCleaner.awaitCleaned for LogCleanerIntegrationTest KAFKA-2656; Remove hardcoded default key and truststores MINOR: Reduce logging level for controller connection failures from `error` to `warn`
This seems to be necessary for our tests to pass consistently.
`Sasl*ConsumerTest` already tests similar scenarios.
harshach
added a commit
that referenced
this pull request
Oct 21, 2015
Merge trunk, refactor consumer/producer tests and other fixes and improvements
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
16 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.