From 7d3651d425f0aa6ffcdb18f8063b786f017a2d60 Mon Sep 17 00:00:00 2001 From: "Matthias J. Sax" Date: Sun, 8 Jul 2018 14:50:27 -0700 Subject: [PATCH] Update docs for 1.0.2 release --- 10/generated/producer_config.html | 2 +- 10/generated/protocol_errors.html | 2 +- 10/generated/streams_config.html | 2 + 10/js/templateData.js | 2 +- 10/protocol.html | 9 +- 10/streams/core-concepts.html | 8 +- .../developer-guide/config-streams.html | 46 ++++- 10/streams/developer-guide/dsl-api.html | 11 +- 10/streams/developer-guide/processor-api.html | 4 +- 10/streams/index.html | 29 +--- 10/streams/upgrade-guide.html | 46 +++-- 10/upgrade.html | 162 +++++++++++++++++- downloads.html | 156 +++++++++-------- 13 files changed, 351 insertions(+), 128 deletions(-) diff --git a/10/generated/producer_config.html b/10/generated/producer_config.html index 4545b0d703a..3ff65e4bfa6 100644 --- a/10/generated/producer_config.html +++ b/10/generated/producer_config.html @@ -110,7 +110,7 @@ ssl.trustmanager.algorithmThe algorithm used by trust manager factory for SSL connections. Default value is the trust manager factory algorithm configured for the Java Virtual Machine.stringPKIXlow -transaction.timeout.msThe maximum amount of time in ms that the transaction coordinator will wait for a transaction status update from the producer before proactively aborting the ongoing transaction.If this value is larger than the max.transaction.timeout.ms setting in the broker, the request will fail with a `InvalidTransactionTimeout` error.int60000low +transaction.timeout.msThe maximum amount of time in ms that the transaction coordinator will wait for a transaction status update from the producer before proactively aborting the ongoing transaction.If this value is larger than the transaction.max.timeout.ms setting in the broker, the request will fail with a `InvalidTransactionTimeout` error.int60000low transactional.idThe TransactionalId to use for transactional delivery. This enables reliability semantics which span multiple producer sessions since it allows the client to guarantee that transactions using the same TransactionalId have been completed prior to starting any new transactions. If no TransactionalId is provided, then the producer is limited to idempotent delivery. Note that enable.idempotence must be enabled if a TransactionalId is configured. The default is null, which means transactions cannot be used. Note that transactions requires a cluster of at least three brokers by default what is the recommended setting for production; for development you can change this, by adjusting broker setting `transaction.state.log.replication.factor`.stringnullnon-empty stringlow diff --git a/10/generated/protocol_errors.html b/10/generated/protocol_errors.html index 99d68b0c7ff..de38465f17b 100644 --- a/10/generated/protocol_errors.html +++ b/10/generated/protocol_errors.html @@ -55,7 +55,7 @@ INVALID_PRODUCER_EPOCH47FalseProducer attempted an operation with an old epoch. Either there is a newer producer with the same transactionalId, or the producer's transaction has been expired by the broker. INVALID_TXN_STATE48FalseThe producer attempted a transactional operation in an invalid state INVALID_PRODUCER_ID_MAPPING49FalseThe producer attempted to use a producer id which is not currently assigned to its transactional id -INVALID_TRANSACTION_TIMEOUT50FalseThe transaction timeout is larger than the maximum value allowed by the broker (as configured by max.transaction.timeout.ms). +INVALID_TRANSACTION_TIMEOUT50FalseThe transaction timeout is larger than the maximum value allowed by the broker (as configured by transaction.max.timeout.ms). CONCURRENT_TRANSACTIONS51FalseThe producer attempted to update a transaction while another concurrent operation on the same transaction was ongoing TRANSACTION_COORDINATOR_FENCED52FalseIndicates that the transaction coordinator sending a WriteTxnMarker is no longer the current coordinator for a given producer TRANSACTIONAL_ID_AUTHORIZATION_FAILED53FalseTransactional Id authorization failed diff --git a/10/generated/streams_config.html b/10/generated/streams_config.html index 93a5c04b707..10d99c153cc 100644 --- a/10/generated/streams_config.html +++ b/10/generated/streams_config.html @@ -78,6 +78,8 @@ timestamp.extractorTimestamp extractor class that implements the org.apache.kafka.streams.processor.TimestampExtractor interface. This config is deprecated, use default.timestamp.extractor insteadclassnulllow +upgrade.fromAllows upgrading from version 0.10.0 to version 0.10.1 (or newer) in a backward compatible way. Default is null. Accepted values are "0.10.0" (for upgrading from 0.10.0.x).stringnull[null, 0.10.0]low + value.serdeSerializer / deserializer class for value that implements the org.apache.kafka.common.serialization.Serde interface. This config is deprecated, use default.value.serde insteadclassnulllow windowstore.changelog.additional.retention.msAdded to a windows maintainMs to ensure data is not deleted from the log prematurely. Allows for clock drift. Default is 1 daylong86400000low diff --git a/10/js/templateData.js b/10/js/templateData.js index 2238b08d2ca..655f12702c8 100644 --- a/10/js/templateData.js +++ b/10/js/templateData.js @@ -19,6 +19,6 @@ limitations under the License. var context={ "version": "10", "dotVersion": "1.0", - "fullDotVersion": "1.0.1", + "fullDotVersion": "1.0.2", "scalaVersion": "2.11" }; diff --git a/10/protocol.html b/10/protocol.html index 40422237e1f..6e45b5ef15e 100644 --- a/10/protocol.html +++ b/10/protocol.html @@ -148,10 +148,11 @@
SASL Authentication SequenceKafka ApiVersionsRequest may be sent by the client to obtain the version ranges of requests supported by the broker. This is optional.
  • Kafka SaslHandshakeRequest containing the SASL mechanism for authentication is sent by the client. If the requested mechanism is not enabled in the server, the server responds with the list of supported mechanisms and closes the client connection. If the mechanism is enabled - in the server, the server sends a successful response and continues with SASL authentication. -
  • The actual SASL authentication is now performed. A series of SASL client and server tokens corresponding to the mechanism are sent as opaque - packets. These packets contain a 32-bit size followed by the token as defined by the protocol for the SASL mechanism. -
  • If authentication succeeds, subsequent packets are handled as Kafka API requests. Otherwise, the client connection is closed. + in the server, the server sends a successful response and continues with SASL authentication.
    • +
  • The actual SASL authentication is now performed. If SaslHandshakeRequest version is v0, a series of SASL client and server tokens corresponding to the mechanism are sent + as opaque packets without wrapping the messages with Kafka protocol headers. If SaslHandshakeRequest version is v1, the SaslAuthenticate + request/response are used, where the actual SASL tokens are wrapped in the Kafka protocol. The error code in the final message from the broker will indicate if authentication succeeded or failed.
    • +
  • If authentication succeeds, subsequent packets are handled as Kafka API requests. Otherwise, the client connection is closed.

    • For interoperability with 0.9.0.x clients, the first packet received by the server is handled as a SASL/GSSAPI client token if it is not a valid Kafka request. SASL/GSSAPI authentication is performed starting with this packet, skipping the first two steps above.

    diff --git a/10/streams/core-concepts.html b/10/streams/core-concepts.html index 1e48805144b..f2f32ad82be 100644 --- a/10/streams/core-concepts.html +++ b/10/streams/core-concepts.html @@ -63,7 +63,7 @@

    Stream Processing Topology There are two special processors in the topology: @@ -78,14 +78,14 @@

    Stream Processing Topology

    - Kafka Streams offers two ways to define the stream processing topology: the Kafka Streams DSL provides - the most common data transformation operations such as map, filter, join and aggregations out of the box; the lower-level Processor API allows + Kafka Streams offers two ways to define the stream processing topology: the Kafka Streams DSL provides + the most common data transformation operations such as map, filter, join and aggregations out of the box; the lower-level Processor API allows developers define and connect custom processors as well as to interact with state stores.

    A processor topology is merely a logical abstraction for your stream processing code. - At runtime, the logical topology is instantiated and replicated inside the application for parallel processing (see Stream Partitions and Tasks for details). + At runtime, the logical topology is instantiated and replicated inside the application for parallel processing (see Stream Partitions and Tasks for details).

    Time

    diff --git a/10/streams/developer-guide/config-streams.html b/10/streams/developer-guide/config-streams.html index fd0cbcace7e..7b029a76978 100644 --- a/10/streams/developer-guide/config-streams.html +++ b/10/streams/developer-guide/config-streams.html @@ -75,6 +75,7 @@
  • num.standby.replicas
  • num.stream.threads
  • partition.grouper
    • +
  • processing.guarantee
  • replication.factor
  • state.dir
  • timestamp.extractor
    • @@ -98,6 +99,13 @@

    Required configuration parameters

    Here are the required Streams configuration parameters.

    + + + + + + + @@ -162,6 +170,13 @@

    bootstrap.servers +

    + + + + + + @@ -201,7 +216,7 @@

    bootstrap.servers

    - + @@ -248,6 +263,11 @@

    bootstrap.serversPartition grouper class that implements the PartitionGrouper interface.

    + + + + @@ -389,6 +409,19 @@

    num.stream.threads +

    processing.guarantee

    +
    +
    The processing guarantee that should be used. Possible values are "at_least_once" (default) and "exactly_once". + Note that if exactly-once processing is enabled, the default for parameter commit.interval.ms changes to 100ms. + Additionally, consumers are configured with isolation.level="read_committed" + and producers are configured with retries=Integer.MAX_VALUE, enable.idempotence=true, + and max.in.flight.requests.per.connection=1 per default. + Note that by default exactly-once processing requires a cluster of at least three brokers what is the recommended setting for production. + For development you can change this, by adjusting broker setting transaction.state.log.replication.factor to the number of broker you want to use. + For more details see Processing Guarantees. +
    +

    Parameter Name Importance
    Parameter Name Importancedefault.deserialization.exception.handler Medium Exception handling class that implements the DeserializationExceptionHandler interface.LogAndContinueExceptionHandler30000 milliseconds
    default.production.exception.handler MediumSee Partition Grouper
    processing.guaranteeLowThe processing mode. Can be either "at_least_once" (default) or "exactly_once". + See Processing Guarantee
    poll.ms Low The amount of time in milliseconds to block waiting for input.
    + + + + + @@ -624,6 +662,12 @@

    Default ValuesRecommended configuration parameters for resiliency

    There are several Kafka and Kafka Streams configuration options that need to be configured explicitly for resiliency in face of broker failures:

    Parameter Name Corresponding Client
    + + + + + + diff --git a/10/streams/developer-guide/dsl-api.html b/10/streams/developer-guide/dsl-api.html index 52135ef2ab8..112746b2a7f 100644 --- a/10/streams/developer-guide/dsl-api.html +++ b/10/streams/developer-guide/dsl-api.html @@ -180,8 +180,7 @@

    Overviewnull) or as DELETE (when the value is null) for that key. (details)

    In the case of a GlobalKTable, the local GlobalKTable instance of every application instance will - be populated with data from only a subset of the partitions of the input topic. Collectively, across - all application instances, all input topic partitions are read and processed.

    + be populated with data from all the partitions of the input topic.

    You must provide a name for the table (more precisely, for the internal state store that backs the table). This is required for supporting interactive queries against the table. When a @@ -1485,6 +1484,14 @@

    Overviewwindowed joins or non-windowed joins.

    Parameter Name Corresponding Client
    + + + + + + + + diff --git a/10/streams/developer-guide/processor-api.html b/10/streams/developer-guide/processor-api.html index 6719ad10189..fdf6c86b4cd 100644 --- a/10/streams/developer-guide/processor-api.html +++ b/10/streams/developer-guide/processor-api.html @@ -140,8 +140,8 @@

    Overview@Override public void close() { - // close the key-value store - kvStore.close(); + // close any resources managed by this processor + // Note: Do not close any StateStores as these are managed by the library } } diff --git a/10/streams/index.html b/10/streams/index.html index 1a08b690881..b39a7d363d6 100644 --- a/10/streams/index.html +++ b/10/streams/index.html @@ -15,7 +15,6 @@ -

    Join operands Type