Rack aware replication#15

Closed

jmlvanre wants to merge 171 commits into

jmlvanre:rack-aware-replication

Commits on Jan 13, 2012

KAFKA 244 Improve log4j appender to use kafka.producer.Producer; patched by vtkstef; reviewed by nehanarkhede
Neha Narkhede
committed

Commits on Jan 17, 2012

max.message.size and fetch.size defaults should be consistent; patched by Pierre-Yves Ritschard; reviewed by Jun Rao; KAFKA-247
Jun Rao
committed

Commits on Jan 19, 2012

ConsumerIterator throws a IllegalStateException after a ConsumerTimeout occurs; patched by Jun Rao; reviewed by Joel Koshy; KAFKA-241
Jun Rao
committed

Commits on Feb 2, 2012

Commits on Feb 6, 2012

KAFKA-257 Hadoop producer should use software load balancer; patched by Sam Shah; reviewed by nehanarkhede
Neha Narkhede
committed

Commits on Feb 7, 2012

KAFKA-263 Enhance single host broker failure test to have 2 topics with uneven distribution on the source brokers; patched by John Fung; reviewed by Neha Narkhede
Neha Narkhede
committed

Commits on Feb 9, 2012

KAFKA-262 Bug in the consumer rebalancing logic causes one consumer to release partitions that it does not own; patched by Neha Narkhede; reviewed by Jun Rao
Neha Narkhede
committed

Commits on Feb 10, 2012

trivial change to reduce default fetcher queue size
Jun Rao
committed

Commits on Feb 13, 2012

Add a queue of zookeeper notifications in the zookeeper consumer to reduce the number of rebalancing attempts; patched by Jun Rao; reviewed by Neha Narkhede; KAFKA-265
Jun Rao
committed

Commits on Feb 14, 2012

time-based reconnect in producer; patched by Yang Ye; reviewed by Jun Rao; KAFKA-268
Jun Rao
committed

Commits on Feb 15, 2012

Commits on Feb 17, 2012

shutdown watch executor thread properly; patched by Jun Rao; reviewed by Neha Narkhede; KAFKA-265
Jun Rao
committed
KAFKA-274 Handle corrupted messages cleanly; patched by nehanarkhede; reviewed by junrao
Neha Narkhede
committed
test access, fixed space
Joe Stein
committed
trivial fix for kafka-producer-perf-test.sh
Jun Rao
committed

Commits on Feb 20, 2012

make time-based reconnect starting at a random time; patched by Yang Ye; reviewed by Jun Rao; KAFKA-268
Jun Rao
committed

Commits on Feb 21, 2012

Commits on Feb 23, 2012

Add a shallow iterator to the ByteBufferMessageSet; patched by Yang Ye; reviewed by Jun Rao; KAFKA-277
Jun Rao
committed

Commits on Feb 25, 2012

add jmx beans in broker to track # of failed requests; patched by Jun Rao; reviewed by Neha Narkhede; KAFKA-283
Jun Rao
committed

Commits on Feb 27, 2012

consumer sometimes don't release partition ownership properly in ZK during rebalance; patched by Jun Rao; reviewed by Neha Narkhede; KAFKA-286
Jun Rao
committed

Commits on Feb 28, 2012

Increase maximum value of log.retention.size; patched by Elben Shira; reviewed by Jun Rao; KAFKA-285
Jun Rao
committed

Commits on Feb 29, 2012

use propertyExists to test if both broker.list and zk.connect are present; patched by Jun Rao; reviewed by Neha Narkhede; KAFKA-290
Jun Rao
committed

Commits on Mar 1, 2012

Niek Sanders - KAFKA-284 fixed compilation issue for cpp client
Joe Stein
committed

Commits on Mar 6, 2012

broker deletes all file segments when cleaning up an empty log segment; patched by Jun Rao; reviewed by Neha Narkhede; KAFKA-292
Jun Rao
committed

Commits on Mar 7, 2012

trivial fix to include violated payload size in MessageSizeTooLargeException; patched by Jun Rao
Jun Rao
committed

Commits on Mar 8, 2012

LogManager test fails on linux; patched by Jun Rao; reviewed by Neha Narkhede; KAFKA-220
Jun Rao
committed

Commits on Mar 14, 2012

Update Go Client to new version of Go; patched by AaronR; KAFKA-296
Jun Rao
committed

Commits on Mar 15, 2012

reverting previous commit for KAFKA-296 because patch didn't apply cleanly
Jun Rao
committed

Commits on Mar 21, 2012

Commits on Mar 22, 2012

Commits on Apr 6, 2012

Commits on Apr 7, 2012

Separate out Kafka mirroring into a stand-alone app; patched by Joel Koshy; reviewed by Jun Rao and Neha Narkhede; KAFKA-249
Jun Rao
committed

Commits on Apr 8, 2012

KAFKA-320 testZKSendWithDeadBroker fails intermittently due to ZKNodeExistsException; patched by nehanarkhede; reviewed by junrao and prashanth menon
Neha Narkhede
committed

Commits on Apr 25, 2012

Mirroring should use multiple producers; add producer retries to DefaultEventHandler; patched by Joel Koshy; reviewed by Jun Rao; KAFKA-332
Jun Rao
committed

Commits on May 18, 2012

Minor fix: Turning on TCP NODELAY in the Simple Consumer. This fix has a significant impact on single fetch request performance from a latency standpoint
Neha Narkhede
committed

Commits on Jun 14, 2012

KAFKA-365 change copyright in NOTICE to current year, reviewed by Jun Rao
Joe Stein
committed

Commits on Jun 19, 2012

add jmx beans in broker to track # bytes in consumer; patched by Jun Rao; reviewed by Joel Koshy; KAFKA-336
Jun Rao
committed

Commits on Jun 20, 2012

KAFKA-368 use the pig core jar from maven instead of distributing it patch by Joe Stein reviewed by Jun Rao and Neha Narkhede
Joe Stein
committed

Commits on Jul 6, 2012

KAFKA-373 Fix trunk broker failure test to work with mirror maker; patched by John Fung; reviewed by Joel Koshy
Joel Jacob Koshy
committed

Commits on Aug 24, 2012

Commits on Aug 25, 2012

Commits on Aug 29, 2012

Message size not checked at the server (patch v3); patched by Swapnil Ghike; reviewed by Jun Rao; KAFKA-469
Jun Rao
committed

Commits on Sep 7, 2012

Handle topic names with / on Kafka server; patched by Swapnil Ghike; reviewed by Jay Kreps and Jun Rao; kafka-495
Jun Rao
committed

Commits on Sep 18, 2012

TopicCount.constructTopicCount isn't thread-safe; patched by Jun Rao; reviewed by Joel Koshy; KAFKA-379
Jun Rao
committed

Commits on Sep 27, 2012

KAFKA-534 remove client library directory
Joe Stein
committed
KAFKA-533 changes to NOTICE and LICENSE related to KAFKA-534 removing client libraries from repo
Joe Stein
committed

Commits on Sep 29, 2012

KAFKA-538 missing apache license header in sbt and util.h
Joe Stein
committed

Commits on Oct 8, 2012

ConsumerOffsetChecker now works with hostnames (in addition to IP) in the brokers/ids zk path; KAFKA-549; patched by Bob Cotton; reviewed by Joel Koshy
Joel Jacob Koshy
committed

Commits on Oct 9, 2012

The ConsumerStats MBean name should include the groupid; patched by Michael Tamm; reviewed by Jun Rao; kafka-547
Jun Rao
committed

Commits on Oct 10, 2012

Fix wildcard consumption to work with greater than one stream; KAFKA-550; patched by Joel Koshy; reviewed by Jun Rao and Neha Narkhede.
Joel Jacob Koshy
committed

Commits on Nov 26, 2012

copy branch 0.8 to trunk
Jun Rao
committed

Commits on Nov 28, 2012

Kafka has become a TLP
gmcdonald
committed

Commits on Dec 2, 2012

KAFKA-521 Refactor the log subsystem. Patch reviewed by Neha.
Edward Jay Kreps
committed
KAFKA-521 Missing files on last commit.
Edward Jay Kreps
committed

Commits on Dec 10, 2012

KAFKA-597 Refactor scheduler. Fixes a couple of bugs, and adds the ability to mock scheduled tasks.
Edward Jay Kreps
committed

Commits on Dec 11, 2012

KAFKA-636 Make log segment delete an asynchronous background action done by the scheduler. Patch reviewed by Jun and Neha.
Edward Jay Kreps
committed

Commits on Dec 16, 2012

KAFKA-374 Move off the default java crc implementation to the crc code borrowed from Hadoop. Patch reviewed by Joe.
jkreps
committed

Commits on Dec 18, 2012

Merge branch '0.8' into trunk
jkreps
committed

Commits on Jan 4, 2013

Commits on Jan 5, 2013

fixed incubation url
Joe Stein
committed

Commits on Jan 7, 2013

KAFKA-133 publish kafka to maven - patch by Maxime Brugidou with updates for depen by Otis Gospodnetic, reviewed by Joe Stein
Joe Stein
committed

Commits on Jan 12, 2013

merge from 0.8 and resolve conflicts
junrao
committed

Commits on Jan 13, 2013

Merge remote branch 'origin/0.8' into trunk
junrao
committed

Commits on Jan 14, 2013

Merge remote branch 'origin/0.8' into trunk
junrao
committed

Commits on Jan 15, 2013

Merge remote branch 'origin/0.8' into trunk
junrao
committed

Commits on Jan 17, 2013

merge from 0.8 and resolve conflict in Log
junrao
committed

Commits on Jan 21, 2013

Merge remote branch 'origin/0.8' into trunk
junrao
committed

Commits on Jan 26, 2013

merge 0.8 to trunk and resolve conflicts
junrao
committed

Commits on Jan 29, 2013

KAFKA-631 Implement a log cleaner for Kafka. Reviewed by Neha.
jkreps
committed

Commits on Feb 25, 2013

KAFKA-771 NPE in handleOffsetCommitRequest; reviewed by Neha Narkhede

mumrah
authored and
nehanarkhede
committed

Commits on Mar 1, 2013

KAFKA-759 Commit/FetchOffset APIs should not return versionId; reviewed by Neha Narkhede
nehanarkhede
committed

Commits on Mar 4, 2013

merge from 0.8 and resolve conflicts
junrao
committed

Commits on Mar 8, 2013

KAFKA-554 Dynamic per-topic configuration. This patch adds a mechanism for storing per-topic configurations in zookeeper and dynamically making config changes across the cluster. Reviewed by Neha a…
jkreps
committed

Commits on Mar 12, 2013

KAFKA-739 Handle null message payloads in messages and in the log cleaner. Reviewed by Jun and Neha.
jkreps
committed

Commits on Apr 18, 2013

kafka-823; merge 0.8 (51421fcc0111031bb77f779a6f6c00520d526a34) to trunk; patched by Jun Rao; reviewed by Jay Kreps
junrao
committed

Commits on May 30, 2013

kafka-891; NullPointerException in ConsoleConsumer; patched by Colin B.; reviewed by Neha Narkhede and Jun Rao

Colin B
authored and
junrao
committed

Commits on Jul 8, 2013

Commits on Jul 11, 2013

Commits on Jul 29, 2013

KAFKA-925 Add a partition key that overrides the message's stored key. Patch reviewed by Joel.
jkreps
committed

Commits on Aug 3, 2013

Commits on Aug 6, 2013

KAFKA-615 fsync asynchronous from log roll. Patch reviewed by Jun and Sriram.
jkreps
committed

Commits on Aug 19, 2013

KAFKA-1009 DumpLogSegments tool should return error on non-existing files; reviewed by Neha Narkhede

Guozhang Wang
authored and
nehanarkhede
committed

Commits on Sep 11, 2013

merge from 0.8 da4512174b6f7c395ffe053a86d2c6bb19d2538a and resolve conflicts
junrao
committed

Commits on Sep 13, 2013

Commits on Sep 15, 2013

Trivial commit: Fix whitespace in Utils.rm.
jkreps
committed

Commits on Sep 18, 2013

KAFKA-1053 Kafka patch review tool that integrates JIRA and reviewboard; reviewed by Joel Koshy, Swapnil Ghike and Guozhang Wang
nehanarkhede
committed

Commits on Oct 7, 2013

Merge remote branch 'origin/0.8' into trunk
junrao
committed

Commits on Oct 8, 2013

Merge remote-tracking branch 'origin/0.8' into trunk
junrao
committed

Commits on Oct 10, 2013

Commits on Oct 11, 2013

KAFKA-1018 tidy up the pom, 0.8.0 release, build the test jars now too
joestein
committed

Commits on Oct 14, 2013

KAFKA-1081 Clean up shell scripts. Removed re-run of failed command. Use exec instead of subprocess execution.
jkreps
committed

Commits on Oct 16, 2013

Commits on Oct 17, 2013

Commits on Oct 18, 2013

KAFKA-1042 logSegments(from, to) Misses first segment.
jkreps
committed

Commits on Oct 19, 2013

KAFKA-1094 Configure reviewboard url in kafka-patch-review tool; reviewed by Neha Narkhede

Swapnil Ghike
authored and
nehanarkhede
committed

Commits on Oct 21, 2013

KAFKA-1042 follow up compilation fix; reviewed by Neha Narkhede

jmlvanre
authored and
nehanarkhede
committed

Commits on Oct 25, 2013

Commits on Oct 29, 2013

KAFKA-1001; Handle follower transition in batch; patched by Guozhang Wang; reviewed by Jun Rao

Guozhang Wang
authored and
junrao
committed

Commits on Oct 30, 2013

KAFKA-1060 Break-down sendTime into responseQueueTime and the real sendTime; reviewed by Neha Narkhede and Jun Rao

Guozhang Wang
authored and
nehanarkhede
committed

Commits on Oct 31, 2013

kafka-1092; Add server config parameter to separate bind address and ZK hostname; patched by Roger Hoover; reviewed by Jun Rao

theduderog
authored and
junrao
committed

Commits on Nov 4, 2013

Minor logging improvement
nehanarkhede
committed

Commits on Nov 6, 2013

KAFKA-1121 DumpLogSegments tool should print absolute file name to report inconsistencies; reviewed by Neha Narkhede and Guozhang Wang

Swapnil Ghike
authored and
nehanarkhede
committed

Commits on Nov 8, 2013

Commits on Nov 19, 2013

Commits on Nov 20, 2013

Commits on Nov 21, 2013

fix for scala 2.10.1 compilation issued introduced in kafka-1117
junrao
committed

Commits on Nov 22, 2013

kafka-933; Hadoop example running DataGenerator causes kafka.message.Message cannot be cast to [B exception; patched by drunkedcat; reviewed by Jun Rao

whitelilis
authored and
junrao
committed
kafka-1136; Add subAppend in Log4jAppender for generic usage; patched by Jie Huang; reviewed by Joel Koshy and Jun Rao

Jie Huang
authored and
junrao
committed
kafka-1103; Consumer uses two zkclients; patched by Guozhang Wang; reviewed by Joel Koshy and Jun Rao

Guozhang Wang
authored and
junrao
committed

Commits on Nov 25, 2013

Commits on Nov 26, 2013

Update description of num.partitions config in sample server properties - issue reported by Vaibhav Puranik
jjkoshy
committed

Commits on Nov 28, 2013

kafka-1140; Move the decoding logic from ConsumerIterator.makeNext to next; patched by Guozhang Wang; reviewed by Jun Rao

Guozhang Wang
authored and
junrao
committed

Commits on Nov 29, 2013

kafka-1152; ReplicaManager's handling of the leaderAndIsrRequest should gracefully handle leader == -1; patched by Swapnil Ghike; reviewed by Jun Rao

Swapnil Ghike
authored and
junrao
committed

Commits on Dec 2, 2013

kafka-1154; replicas may not have consistent data after becoming follower; patched by Jun Rao; reviewed by Guozhang Wang and Neha Narkhede
junrao
committed

Commits on Dec 3, 2013

kafka-1157; Clean up Per-topic Configuration from Kafka properties; patched by Guozhang Wang; reviewed by Jun Rao

Guozhang Wang
authored and
junrao
committed

Commits on Dec 4, 2013

Commits on Dec 5, 2013

kafka-1169; missing synchronization in access to leaderCache in KafkaApis; patched by Jun Rao; reviewed by Neha Narkhede
junrao
committed

Commits on Dec 6, 2013

KAFKA-1167 Improve the kafka-topics tool to list the topics with overridden configs; reviewed by Jun Rao
nehanarkhede
committed

Commits on Dec 7, 2013

KAFKA-1170 ISR can be inconsistent during partition reassignment for low throughput partitions; reviewed by Jun Rao and Guozhang Wang
nehanarkhede
committed

Commits on Dec 11, 2013

KAFKA-1178 Replica fetcher thread dies while becoming follower; reviewed by Jun Rao and Guozhang Wang
nehanarkhede
committed

Commits on Dec 12, 2013

Commits on Dec 19, 2013

KAFKA-1139 Topic data change handling callback should not call syncedRebalance directly; reviewed by Guozhang Wang and Jun Rao
nehanarkhede
committed

Commits on Dec 20, 2013

Commits on Jan 3, 2014

Commits on Jan 8, 2014

kafka-1074; Reassign partitions should delete the old replicas from disk; patched by Jun Rao; reviewed by Jay Kreps, Neha Narkhede and Guozhang Wang
junrao
committed

Commits on Jan 13, 2014

KAFKA-1199 Add a reduced access log level; reviewed by Guozhang Wang and Jun Rao
nehanarkhede
committed

Commits on Jan 14, 2014

KAFKA-1200 inconsistent log levels when consumed offset is reset patch by Dima Pekar reviewed by Joe Stein
joestein
committed

Commits on Jan 15, 2014

kafka-1202; optimize ZK access in KafkaController; also incorporating fixes in kafka-1020; patched by Jun Rao and Guozhang Wang; reviewed by Neha Narkhede and Joel Koshy
junrao
committed

Commits on Jan 16, 2014

kafka-1208; Update system test still to use kafka-topics instead of kafka-create-topics shell; patched by Guozhang Wang; reviewed by Jun Rao

Guozhang Wang
authored and
junrao
committed

Commits on Jan 17, 2014

kafka-1205; README in examples not update; patched by Ailing Zhang; reviewed by Jun Rao

ailing-bot
authored and
junrao
committed

Commits on Feb 1, 2014

Adding rack-aware replication option. rack-id defaults to -1. use the max-rack-replication option when creating a topic to distribute replicas such that no more than max-rack-replication replicas a…
jmlvanre
committed