Update docs for 1.0.2 release

10/generated/producer_config.html

@@ -110,7 +110,7 @@
 <tr>
 <td>ssl.trustmanager.algorithm</td><td>The algorithm used by trust manager factory for SSL connections. Default value is the trust manager factory algorithm configured for the Java Virtual Machine.</td><td>string</td><td>PKIX</td><td></td><td>low</td></tr>
 <tr>
-<td>transaction.timeout.ms</td><td>The maximum amount of time in ms that the transaction coordinator will wait for a transaction status update from the producer before proactively aborting the ongoing transaction.If this value is larger than the max.transaction.timeout.ms setting in the broker, the request will fail with a `InvalidTransactionTimeout` error.</td><td>int</td><td>60000</td><td></td><td>low</td></tr>
+<td>transaction.timeout.ms</td><td>The maximum amount of time in ms that the transaction coordinator will wait for a transaction status update from the producer before proactively aborting the ongoing transaction.If this value is larger than the transaction.max.timeout.ms setting in the broker, the request will fail with a `InvalidTransactionTimeout` error.</td><td>int</td><td>60000</td><td></td><td>low</td></tr>
 <tr>
 <td>transactional.id</td><td>The TransactionalId to use for transactional delivery. This enables reliability semantics which span multiple producer sessions since it allows the client to guarantee that transactions using the same TransactionalId have been completed prior to starting any new transactions. If no TransactionalId is provided, then the producer is limited to idempotent delivery. Note that enable.idempotence must be enabled if a TransactionalId is configured. The default is <code>null</code>, which means transactions cannot be used. Note that transactions requires a cluster of at least three brokers by default what is the recommended setting for production; for development you can change this, by adjusting broker setting `transaction.state.log.replication.factor`.</td><td>string</td><td>null</td><td>non-empty string</td><td>low</td></tr>
 </tbody></table>

10/generated/protocol_errors.html

@@ -55,7 +55,7 @@
 <tr><td>INVALID_PRODUCER_EPOCH</td><td>47</td><td>False</td><td>Producer attempted an operation with an old epoch. Either there is a newer producer with the same transactionalId, or the producer's transaction has been expired by the broker.</td></tr>
 <tr><td>INVALID_TXN_STATE</td><td>48</td><td>False</td><td>The producer attempted a transactional operation in an invalid state</td></tr>
 <tr><td>INVALID_PRODUCER_ID_MAPPING</td><td>49</td><td>False</td><td>The producer attempted to use a producer id which is not currently assigned to its transactional id</td></tr>
-<tr><td>INVALID_TRANSACTION_TIMEOUT</td><td>50</td><td>False</td><td>The transaction timeout is larger than the maximum value allowed by the broker (as configured by max.transaction.timeout.ms).</td></tr>
+<tr><td>INVALID_TRANSACTION_TIMEOUT</td><td>50</td><td>False</td><td>The transaction timeout is larger than the maximum value allowed by the broker (as configured by transaction.max.timeout.ms).</td></tr>
 <tr><td>CONCURRENT_TRANSACTIONS</td><td>51</td><td>False</td><td>The producer attempted to update a transaction while another concurrent operation on the same transaction was ongoing</td></tr>
 <tr><td>TRANSACTION_COORDINATOR_FENCED</td><td>52</td><td>False</td><td>Indicates that the transaction coordinator sending a WriteTxnMarker is no longer the current coordinator for a given producer</td></tr>
 <tr><td>TRANSACTIONAL_ID_AUTHORIZATION_FAILED</td><td>53</td><td>False</td><td>Transactional Id authorization failed</td></tr>

10/generated/streams_config.html

@@ -78,6 +78,8 @@
 <tr>
 <td>timestamp.extractor</td><td>Timestamp extractor class that implements the <code>org.apache.kafka.streams.processor.TimestampExtractor</code> interface. This config is deprecated, use <code>default.timestamp.extractor</code> instead</td><td>class</td><td>null</td><td></td><td>low</td></tr>
 <tr>
+<td>upgrade.from</td><td>Allows upgrading from version 0.10.0 to version 0.10.1 (or newer) in a backward compatible way. Default is null. Accepted values are "0.10.0" (for upgrading from 0.10.0.x).</td><td>string</td><td>null</td><td>[null, 0.10.0]</td><td>low</td></tr>
+<tr>
 <td>value.serde</td><td>Serializer / deserializer class for value that implements the <code>org.apache.kafka.common.serialization.Serde</code> interface. This config is deprecated, use <code>default.value.serde</code> instead</td><td>class</td><td>null</td><td></td><td>low</td></tr>
 <tr>
 <td>windowstore.changelog.additional.retention.ms</td><td>Added to a windows maintainMs to ensure data is not deleted from the log prematurely. Allows for clock drift. Default is 1 day</td><td>long</td><td>86400000</td><td></td><td>low</td></tr>

10/js/templateData.js

@@ -19,6 +19,6 @@ limitations under the License.
 var context={
     "version": "10",
     "dotVersion": "1.0",
-    "fullDotVersion": "1.0.1",
+    "fullDotVersion": "1.0.2",
     "scalaVersion": "2.11"
 };

10/protocol.html

@@ -148,10 +148,11 @@ <h5><a id="sasl_handshake" href="#sasl_handshake">SASL Authentication Sequence</
   <li>Kafka <code>ApiVersionsRequest</code> may be sent by the client to obtain the version ranges of requests supported by the broker. This is optional.</li>
   <li>Kafka <code>SaslHandshakeRequest</code> containing the SASL mechanism for authentication is sent by the client. If the requested mechanism is not enabled
     in the server, the server responds with the list of supported mechanisms and closes the client connection. If the mechanism is enabled
-    in the server, the server sends a successful response and continues with SASL authentication.
-  <li>The actual SASL authentication is now performed. A series of SASL client and server tokens corresponding to the mechanism are sent as opaque
-    packets. These packets contain a 32-bit size followed by the token as defined by the protocol for the SASL mechanism.
-  <li>If authentication succeeds, subsequent packets are handled as Kafka API requests. Otherwise, the client connection is closed.
+    in the server, the server sends a successful response and continues with SASL authentication.</li>
+  <li>The actual SASL authentication is now performed. If <code>SaslHandshakeRequest</code> version is v0, a series of SASL client and server tokens corresponding to the mechanism are sent
+     as opaque packets without wrapping the messages with Kafka protocol headers. If <code>SaslHandshakeRequest</code> version is v1, the <code>SaslAuthenticate</code>
+     request/response are used, where the actual SASL tokens are wrapped in the Kafka protocol. The error code in the final message from the broker will indicate if authentication succeeded or failed.</li>
+  <li>If authentication succeeds, subsequent packets are handled as Kafka API requests. Otherwise, the client connection is closed.</li>
 </ol>
 <p>For interoperability with 0.9.0.x clients, the first packet received by the server is handled as a SASL/GSSAPI client token if it is not a valid
 Kafka request. SASL/GSSAPI authentication is performed starting with this packet, skipping the first two steps above.</p>

10/streams/core-concepts.html

@@ -63,7 +63,7 @@ <h3><a id="streams_topology" href="#streams_topology">Stream Processing Topology
     <ul>
         <li>A <b>stream</b> is the most important abstraction provided by Kafka Streams: it represents an unbounded, continuously updating data set. A stream is an ordered, replayable, and fault-tolerant sequence of immutable data records, where a <b>data record</b> is defined as a key-value pair.</li>
         <li>A <b>stream processing application</b> is any program that makes use of the Kafka Streams library. It defines its computational logic through one or more <b>processor topologies</b>, where a processor topology is a graph of stream processors (nodes) that are connected by streams (edges).</li>
-        <li>A <b><a id="#streams_processor_node" href="#streams_processor_node">stream processor</a></b> is a node in the processor topology; it represents a processing step to transform data in streams by receiving one input record at a time from its upstream processors in the topology, applying its operation to it, and may subsequently produce one or more output records to its downstream processors. </li>
+        <li>A <b><a href="#streams_processor_node">stream processor</a></b> is a node in the processor topology; it represents a processing step to transform data in streams by receiving one input record at a time from its upstream processors in the topology, applying its operation to it, and may subsequently produce one or more output records to its downstream processors. </li>
     </ul>
 
     There are two special processors in the topology:
@@ -78,14 +78,14 @@ <h3><a id="streams_topology" href="#streams_topology">Stream Processing Topology
     <img class="centered" src="/{{version}}/images/streams-architecture-topology.jpg" style="width:400px">
 
     <p>
-        Kafka Streams offers two ways to define the stream processing topology: the <a href="/{{version}}/documentation/streams/developer-guide/dsl-api.html"><b>Kafka Streams DSL</b></a> provides
-        the most common data transformation operations such as <code>map</code>, <code>filter</code>, <code>join</code> and <code>aggregations</code> out of the box; the lower-level <a href="/{{version}}/documentation/streams/developer-guide/processor-api.html"><b>Processor API</b></a> allows
+        Kafka Streams offers two ways to define the stream processing topology: the <a href="/{{version}}/documentation/streams/developer-guide#streams_dsl"><b>Kafka Streams DSL</b></a> provides
+        the most common data transformation operations such as <code>map</code>, <code>filter</code>, <code>join</code> and <code>aggregations</code> out of the box; the lower-level <a href="/{{version}}/documentation/streams/developer-guide#streams_processor"><b>Processor API</b></a> allows
         developers define and connect custom processors as well as to interact with <a href="#streams_state">state stores</a>.
     </p>
 
     <p>
         A processor topology is merely a logical abstraction for your stream processing code.
-        At runtime, the logical topology is instantiated and replicated inside the application for parallel processing (see <a href="/{{version}}/documentation/streams/architecture#streams_architecture_tasks"><b>Stream Partitions and Tasks</b></a> for details).
+        At runtime, the logical topology is instantiated and replicated inside the application for parallel processing (see <a href="#streams_architecture_tasks"><b>Stream Partitions and Tasks</b></a> for details).
     </p>
 
     <h3><a id="streams_time" href="#streams_time">Time</a></h3>

10/streams/developer-guide/config-streams.html

@@ -75,6 +75,7 @@
             <li><a class="reference internal" href="#num-standby-replicas" id="id10">num.standby.replicas</a></li>
             <li><a class="reference internal" href="#num-stream-threads" id="id11">num.stream.threads</a></li>
             <li><a class="reference internal" href="#partition-grouper" id="id12">partition.grouper</a></li>
+            <li><a class="reference internal" href="#processing-guarantee" id="id25">processing.guarantee</a></li>
             <li><a class="reference internal" href="#replication-factor" id="id13">replication.factor</a></li>
             <li><a class="reference internal" href="#state-dir" id="id14">state.dir</a></li>
             <li><a class="reference internal" href="#timestamp-extractor" id="id15">timestamp.extractor</a></li>
@@ -98,6 +99,13 @@
         <h3><a class="toc-backref" href="#id3">Required configuration parameters</a><a class="headerlink" href="#required-configuration-parameters" title="Permalink to this headline"></a></h3>
         <p>Here are the required Streams configuration parameters.</p>
         <table border="1" class="non-scrolling-table docutils">
+          <colgroup>
+            <col width="20%" />
+            <col width="5%" />
+            <col width="7%" />
+            <col width="38%" />
+            <col width="31%" />
+          </colgroup>
           <thead valign="bottom">
           <tr class="row-odd"><th class="head">Parameter Name</th>
             <th class="head">Importance</th>
@@ -162,6 +170,13 @@ <h4><a class="toc-backref" href="#id5">bootstrap.servers</a><a class="headerlink
           </ul>
           </div></blockquote>
         <table border="1" class="non-scrolling-table docutils">
+          <colgroup>
+            <col width="20%" />
+            <col width="5%" />
+            <col width="7%" />
+            <col width="38%" />
+            <col width="31%" />
+          </colgroup>
           <thead valign="bottom">
           <tr class="row-odd"><th class="head">Parameter Name</th>
             <th class="head">Importance</th>
@@ -201,7 +216,7 @@ <h4><a class="toc-backref" href="#id5">bootstrap.servers</a><a class="headerlink
           <tr class="row-odd"><td>default.deserialization.exception.handler</td>
             <td>Medium</td>
             <td colspan="2">Exception handling class that implements the <code class="docutils literal"><span class="pre">DeserializationExceptionHandler</span></code> interface.</td>
-            <td><code class="docutils literal"><span class="pre">LogAndContinueExceptionHandler</span></code></td>
+            <td>30000 milliseconds</td>
           </tr>
           <tr class="row-even"><td>default.production.exception.handler</td>
             <td>Medium</td>
@@ -248,6 +263,11 @@ <h4><a class="toc-backref" href="#id5">bootstrap.servers</a><a class="headerlink
             <td colspan="2">Partition grouper class that implements the <code class="docutils literal"><span class="pre">PartitionGrouper</span></code> interface.</td>
             <td>See <a class="reference internal" href="#streams-developer-guide-partition-grouper"><span class="std std-ref">Partition Grouper</span></a></td>
           </tr>
+          <tr class="row-even"><td>processing.guarantee</td>
+            <td>Low</td>
+            <td colspan="2">The processing mode. Can be either <code class="docutils literal"><span class="pre">"at_least_once"</span></code> (default) or <code class="docutils literal"><span class="pre">"exactly_once"</span></code>.
+            <td>See <a class="reference internal" href="#streams-developer-guide-processing-guarantedd"><span class="std std-ref">Processing Guarantee</span></a></td>
+          </tr>
           <tr class="row-odd"><td>poll.ms</td>
             <td>Low</td>
             <td colspan="2">The amount of time in milliseconds to block waiting for input.</td>
@@ -389,6 +409,19 @@ <h4><a class="toc-backref" href="#id11">num.stream.threads</a><a class="headerli
               It assigns each task with one partition for each of the source topic partitions. The generated number of tasks equals the largest
               number of partitions among the input topics. Usually an application does not need to customize the partition grouper.</div></blockquote>
         </div>
+        <div class="section" id="processing-guarantee">
+          <span id="streams-developer-guide-processing-guarantee"></span><h4><a class="toc-backref" href="#id25">processing.guarantee</a><a class="headerlink" href="#processing-guarantee" title="Permalink to this headline"></a></h4>
+          <blockquote>
+            <div>The processing guarantee that should be used. Possible values are <code class="docutils literal"><span class="pre">"at_least_once"</span></code> (default) and <code class="docutils literal"><span class="pre">"exactly_once"</span></code>.
+                 Note that if exactly-once processing is enabled, the default for parameter <code class="docutils literal"><span class="pre">commit.interval.ms</span></code> changes to 100ms.
+                 Additionally, consumers are configured with <code class="docutils literal"><span class="pre">isolation.level="read_committed"</span></code>
+                 and producers are configured with <code class="docutils literal"><span class="pre">retries=Integer.MAX_VALUE</span></code>, <code class="docutils literal"><span class="pre">enable.idempotence=true</span></code>,
+                 and <code class="docutils literal"><span class="pre">max.in.flight.requests.per.connection=1</span></code> per default.
+                 Note that by default exactly-once processing requires a cluster of at least three brokers what is the recommended setting for production.
+                 For development you can change this, by adjusting broker setting <code class="docutils literal"><span class="pre">transaction.state.log.replication.factor</span></code> to the number of broker you want to use.
+                 For more details see <a href="../core-concepts#streams_processing_guarantee">Processing Guarantees</a>.
+            </div></blockquote>
+        </div>
         <div class="section" id="replication-factor">
           <span id="replication-factor-parm"></span><h4><a class="toc-backref" href="#id13">replication.factor</a><a class="headerlink" href="#replication-factor" title="Permalink to this headline"></a></h4>
           <blockquote>
@@ -535,6 +568,11 @@ <h4><a class="toc-backref" href="#id18">Default Values</a><a class="headerlink"
             of these configs, see <a class="reference external" href="http://kafka.apache.org/0100/documentation.html#producerconfigs">Producer Configs</a>
             and <a class="reference external" href="http://kafka.apache.org/0100/documentation.html#newconsumerconfigs">Consumer Configs</a>.</p>
           <table border="1" class="non-scrolling-table docutils">
+            <colgroup>
+              <col width="50%" />
+              <col width="19%" />
+              <col width="31%" />
+            </colgroup>
             <thead valign="bottom">
             <tr class="row-odd"><th class="head">Parameter Name</th>
               <th class="head">Corresponding Client</th>
@@ -624,6 +662,12 @@ <h4><a class="toc-backref" href="#id18">Default Values</a><a class="headerlink"
         <h3><a class="toc-backref" href="#id21">Recommended configuration parameters for resiliency</a><a class="headerlink" href="#recommended-configuration-parameters-for-resiliency" title="Permalink to this headline"></a></h3>
         <p>There are several Kafka and Kafka Streams configuration options that need to be configured explicitly for resiliency in face of broker failures:</p>
         <table border="1" class="non-scrolling-table docutils">
+          <colgroup>
+            <col width="22%" />
+            <col width="19%" />
+            <col width="10%" />
+            <col width="49%" />
+          </colgroup>
           <thead valign="bottom">
           <tr class="row-odd"><th class="head">Parameter Name</th>
             <th class="head">Corresponding Client</th>