diff --git a/docs/design.html b/docs/design.html index b1e4387373381..f0ab6ca339b1a 100644 --- a/docs/design.html +++ b/docs/design.html @@ -306,7 +306,7 @@

Log Compaction Basics

Here is a high-level picture that shows the logical structure of a Kafka log with the offset for each message.

- +

The head of the log is identical to a traditional Kafka log. It has dense, sequential offsets and retains all messages. Log compaction adds an option for handling the tail of the log. The picture above shows a log with a compacted tail. Note that the messages in the tail of the log retain the original offset assigned when they were first written—that never changes. Note also that all offsets remain valid positions in the log, even if the message with that offset has been compacted away; in this case this position is indistinguishable from the next highest offset that does appear in the log. For example, in the picture above the offsets 36, 37, and 38 are all equivalent positions and a read beginning at any of these offsets would return a message set beginning with 38.

@@ -314,7 +314,7 @@

Log Compaction Basics

The compaction is done in the background by periodically recopying log segments. Cleaning does not block reads and can be throttled to use no more than a configurable amount of I/O throughput to avoid impacting producers and consumers. The actual process of compacting a log segment looks something like this:

- +

What guarantees does log compaction provide?

diff --git a/docs/images/consumer-groups.png b/docs/images/consumer-groups.png new file mode 100644 index 0000000000000..16fe2936cbc71 Binary files /dev/null and b/docs/images/consumer-groups.png differ diff --git a/docs/images/kafka_log.png b/docs/images/kafka_log.png new file mode 100644 index 0000000000000..75abd96babc17 Binary files /dev/null and b/docs/images/kafka_log.png differ diff --git a/docs/images/kafka_multidc.png b/docs/images/kafka_multidc.png new file mode 100644 index 0000000000000..7bc56f4a53258 Binary files /dev/null and b/docs/images/kafka_multidc.png differ diff --git a/docs/images/kafka_multidc_complex.png b/docs/images/kafka_multidc_complex.png new file mode 100644 index 0000000000000..ab88debde607d Binary files /dev/null and b/docs/images/kafka_multidc_complex.png differ diff --git a/docs/images/log_anatomy.png b/docs/images/log_anatomy.png new file mode 100644 index 0000000000000..a649499926e5d Binary files /dev/null and b/docs/images/log_anatomy.png differ diff --git a/docs/images/log_cleaner_anatomy.png b/docs/images/log_cleaner_anatomy.png new file mode 100644 index 0000000000000..fb425b0558460 Binary files /dev/null and b/docs/images/log_cleaner_anatomy.png differ diff --git a/docs/images/log_compaction.png b/docs/images/log_compaction.png new file mode 100644 index 0000000000000..4e4a8334ce117 Binary files /dev/null and b/docs/images/log_compaction.png differ diff --git a/docs/images/mirror-maker.png b/docs/images/mirror-maker.png new file mode 100644 index 0000000000000..b25e8cb78ed00 Binary files /dev/null and b/docs/images/mirror-maker.png differ diff --git a/docs/images/producer_consumer.png b/docs/images/producer_consumer.png new file mode 100644 index 0000000000000..4b10cc95dd51f Binary files /dev/null and b/docs/images/producer_consumer.png differ diff --git a/docs/images/tracking_high_level.png b/docs/images/tracking_high_level.png new file mode 100644 index 0000000000000..b6432306afd2c Binary files /dev/null and b/docs/images/tracking_high_level.png differ diff --git a/docs/implementation.html b/docs/implementation.html index 25f9b39843550..d9ffa4669c1e9 100644 --- a/docs/implementation.html +++ b/docs/implementation.html @@ -191,7 +191,7 @@

5.5 Log

The use of the message offset as the message id is unusual. Our original idea was to use a GUID generated by the producer, and maintain a mapping from GUID to offset on each broker. But since a consumer must maintain an ID for each server, the global uniqueness of the GUID provides no value. Furthermore the complexity of maintaining the mapping from a random id to an offset requires a heavy weight index structure which must be synchronized with disk, essentially requiring a full persistent random-access data structure. Thus to simplify the lookup structure we decided to use a simple per-partition atomic counter which could be coupled with the partition id and node id to uniquely identify a message; this makes the lookup structure simpler, though multiple seeks per consumer request are still likely. However once we settled on a counter, the jump to directly using the offset seemed natural—both after all are monotonically increasing integers unique to a partition. Since the offset is hidden from the consumer API this decision is ultimately an implementation detail and we went with the more efficient approach.

- +

Writes

The log allows serial appends which always go to the last file. This file is rolled over to a fresh file when it reaches a configurable size (say 1GB). The log takes two configuration parameter M which gives the number of messages to write before forcing the OS to flush the file to disk, and S which gives a number of seconds after which a flush is forced. This gives a durability guarantee of losing at most M messages or S seconds of data in the event of a system crash. diff --git a/docs/introduction.html b/docs/introduction.html index 92a78265bd4b1..7e0b150c4ded5 100644 --- a/docs/introduction.html +++ b/docs/introduction.html @@ -30,7 +30,7 @@

1.1 Introduction

So, at a high level, producers send messages over the network to the Kafka cluster which in turn serves them up to consumers like this:
- +
Communication between the clients and the servers is done with a simple, high-performance, language agnostic TCP protocol. We provide a Java client for Kafka, but clients are available in many languages. @@ -40,7 +40,7 @@

Topics and Logs

A topic is a category or feed name to which messages are published. For each topic, the Kafka cluster maintains a partitioned log that looks like this:

- +
Each partition is an ordered, immutable sequence of messages that is continually appended to—a commit log. The messages in the partitions are each assigned a sequential id number called the offset that uniquely identifies each message within the partition.

@@ -76,7 +76,7 @@

Consumers

-
+
A two server Kafka cluster hosting four partitions (P0-P3) with two consumer groups. Consumer group A has two consumer instances and group B has four.

diff --git a/docs/ops.html b/docs/ops.html index 2164ab7e31c31..0645d1c3a0cb6 100644 --- a/docs/ops.html +++ b/docs/ops.html @@ -98,7 +98,7 @@

Mirroring data between clusters

We refer to the process of replicating data between Kafka clusters "mirroring" to avoid confusion with the replication that happens amongst the nodes in a single cluster. Kafka comes with a tool for mirroring data between Kafka clusters. The tool reads from one or more source clusters and writes to a destination cluster, like this:

- +

A common use case for this kind of mirroring is to provide a replica in another datacenter. This scenario will be discussed in more detail in the next section.