From 7b38d43fbb78ef7f7c0a31dfaabdf681012d0555 Mon Sep 17 00:00:00 2001 From: Victoria Bialas Date: Tue, 23 Apr 2019 15:11:13 -0700 Subject: [PATCH 1/8] KAFKA-8227 - Add missing links Core Concepts duality of streams tables Signed-off-by: Victoria Bialas --- 22/streams/core-concepts.html | 27 ++++++++++++++------- 22/streams/developer-guide/running-app.html | 3 ++- 2 files changed, 20 insertions(+), 10 deletions(-) diff --git a/22/streams/core-concepts.html b/22/streams/core-concepts.html index 1e1aeb7b1aa..8e87397cb7d 100644 --- a/22/streams/core-concepts.html +++ b/22/streams/core-concepts.html @@ -157,24 +157,33 @@

Duality of An example use case that is very common in practice is an e-commerce application that enriches an incoming stream of customer transactions with the latest customer information from a database table. In other words, streams are everywhere, but databases are everywhere, too.

+ + EXAMPLEs REMOVE WHEN DONE: + quickstart + Step 8: Use Kafka Streams to process data

Any stream processing technology must therefore provide first-class support for streams and tables. - Kafka's Streams API provides such functionality through its core abstractions for - streams <streams_concepts_kstream> and - tables <streams_concepts_ktable>, which we will talk about in a minute. - Now, an interesting observation is that there is actually a close relationship between streams and tables, - the so-called stream-table duality. - And Kafka exploits this duality in many ways: for example, to make your applications + Kafka's Streams API provides such functionality through its core abstractions for + streams + <streams_concepts_kstreams> + and tables + <streams_concepts_kstreams>, + which we will talk about in a minute. Now, an interesting observation is that there is actually a close relationship between streams and tables, + the so-called stream-table duality. And Kafka exploits this duality in many ways: for example, to make your applications + elastic elastic <streams_developer-guide_execution-scaling>, - to support fault-tolerant stateful processing <streams_developer-guide_state-store_fault-tolerance>, - or to run interactive queries <streams_concepts_interactive-queries> + to support fault-tolerant stateful processing + <streams_developer-guide_state-store_fault-tolerance>, + or to run interactive queries + <streams_concepts_interactive-queries> against your application's latest processing results. And, beyond its internal usage, the Kafka Streams API also allows developers to exploit this duality in their own applications.

- Before we discuss concepts such as aggregations <streams_concepts_aggregations> + Before we discuss concepts such as aggregations + <streams_concepts_aggregations>, in Kafka Streams we must first introduce tables in more detail, and talk about the aforementioned stream-table duality. Essentially, this duality means that a stream can be viewed as a table, and a table can be viewed as a stream.

diff --git a/22/streams/developer-guide/running-app.html b/22/streams/developer-guide/running-app.html index f83210dd170..14ba34f9123 100644 --- a/22/streams/developer-guide/running-app.html +++ b/22/streams/developer-guide/running-app.html @@ -83,7 +83,8 @@ more information, see the State restoration during workload rebalance section).

-

Elastic scaling of your application

+ +

Elastic scaling of your application

Kafka Streams makes your stream processing applications elastic and scalable. You can add and remove processing capacity dynamically during application runtime without any downtime or data loss. This makes your applications resilient in the face of failures and for allows you to perform maintenance as needed (e.g. rolling upgrades).

From dd1eea73a65e20c367ef5b60bd2f66440722dfab Mon Sep 17 00:00:00 2001 From: Victoria Bialas Date: Tue, 23 Apr 2019 17:39:24 -0700 Subject: [PATCH 2/8] removed TBD notes Signed-off-by: Victoria Bialas --- 22/streams/core-concepts.html | 4 ---- 1 file changed, 4 deletions(-) diff --git a/22/streams/core-concepts.html b/22/streams/core-concepts.html index 8e87397cb7d..42d1ae36754 100644 --- a/22/streams/core-concepts.html +++ b/22/streams/core-concepts.html @@ -157,10 +157,6 @@

Duality of An example use case that is very common in practice is an e-commerce application that enriches an incoming stream of customer transactions with the latest customer information from a database table. In other words, streams are everywhere, but databases are everywhere, too.

- - EXAMPLEs REMOVE WHEN DONE: - quickstart - Step 8: Use Kafka Streams to process data

Any stream processing technology must therefore provide first-class support for streams and tables. From 4ace72ffe1504762e555acfe298583ed1b5d63e3 Mon Sep 17 00:00:00 2001 From: Victoria Bialas Date: Tue, 23 Apr 2019 18:08:09 -0700 Subject: [PATCH 3/8] added correct x-ref for stream processor, which was linking to itself Signed-off-by: Victoria Bialas --- 22/streams/core-concepts.html | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/22/streams/core-concepts.html b/22/streams/core-concepts.html index 42d1ae36754..e60efd6eb94 100644 --- a/22/streams/core-concepts.html +++ b/22/streams/core-concepts.html @@ -63,7 +63,9 @@

Stream Processing Topology There are two special processors in the topology: From beeabda7ccb10055ff992bd00673c5c781faf82c Mon Sep 17 00:00:00 2001 From: Victoria Bialas Date: Tue, 23 Apr 2019 18:16:11 -0700 Subject: [PATCH 4/8] replaced self-referential link with link to processor API doc Signed-off-by: Victoria Bialas --- 22/streams/core-concepts.html | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/22/streams/core-concepts.html b/22/streams/core-concepts.html index e60efd6eb94..670d35fa505 100644 --- a/22/streams/core-concepts.html +++ b/22/streams/core-concepts.html @@ -57,14 +57,12 @@

Core Concepts

We first summarize the key concepts of Kafka Streams.

- +

Stream Processing Topology

-
  • A stream is the most important abstraction provided by Kafka Streams: it represents an unbounded, continuously updating data set. A stream is an ordered, replayable, and fault-tolerant sequence of immutable data records, where a data record is defined as a key-value pair.
  • A stream processing application is any program that makes use of the Kafka Streams library. It defines its computational logic through one or more processor topologies, where a processor topology is a graph of stream processors (nodes) that are connected by streams (edges).
    • -
  • A stream processor is a node in the processor topology; it represents a processing step to transform data in streams by receiving one input record at a time from its upstream processors in the topology, applying its operation to it, and may subsequently produce one or more output records to its downstream processors. - (See Defining a Stream Processor.) +
  • A stream processor is a node in the processor topology; it represents a processing step to transform data in streams by receiving one input record at a time from its upstream processors in the topology, applying its operation to it, and may subsequently produce one or more output records to its downstream processors.
From ded5ab04d9effb47b9ec6d9d0a1b0f22bead982f Mon Sep 17 00:00:00 2001 From: Victoria Bialas Date: Tue, 23 Apr 2019 18:25:03 -0700 Subject: [PATCH 5/8] fix all same links on v21 Signed-off-by: Victoria Bialas --- 21/streams/core-concepts.html | 27 ++++++++++++++++----------- 1 file changed, 16 insertions(+), 11 deletions(-) diff --git a/21/streams/core-concepts.html b/21/streams/core-concepts.html index 1e1aeb7b1aa..cc093b5c63f 100644 --- a/21/streams/core-concepts.html +++ b/21/streams/core-concepts.html @@ -57,13 +57,13 @@

Core Concepts

We first summarize the key concepts of Kafka Streams.

- +

Stream Processing Topology

  • A stream is the most important abstraction provided by Kafka Streams: it represents an unbounded, continuously updating data set. A stream is an ordered, replayable, and fault-tolerant sequence of immutable data records, where a data record is defined as a key-value pair.
  • A stream processing application is any program that makes use of the Kafka Streams library. It defines its computational logic through one or more processor topologies, where a processor topology is a graph of stream processors (nodes) that are connected by streams (edges).
    • -
  • A stream processor is a node in the processor topology; it represents a processing step to transform data in streams by receiving one input record at a time from its upstream processors in the topology, applying its operation to it, and may subsequently produce one or more output records to its downstream processors.
    • +
  • A stream processor is a node in the processor topology; it represents a processing step to transform data in streams by receiving one input record at a time from its upstream processors in the topology, applying its operation to it, and may subsequently produce one or more output records to its downstream processors.
There are two special processors in the topology: @@ -160,21 +160,26 @@

Duality of

Any stream processing technology must therefore provide first-class support for streams and tables. - Kafka's Streams API provides such functionality through its core abstractions for - streams <streams_concepts_kstream> and - tables <streams_concepts_ktable>, which we will talk about in a minute. - Now, an interesting observation is that there is actually a close relationship between streams and tables, - the so-called stream-table duality. - And Kafka exploits this duality in many ways: for example, to make your applications + Kafka's Streams API provides such functionality through its core abstractions for + streams + <streams_concepts_kstreams> + and tables + <streams_concepts_kstreams>, + which we will talk about in a minute. Now, an interesting observation is that there is actually a close relationship between streams and tables, + the so-called stream-table duality. And Kafka exploits this duality in many ways: for example, to make your applications + elastic elastic <streams_developer-guide_execution-scaling>, - to support fault-tolerant stateful processing <streams_developer-guide_state-store_fault-tolerance>, - or to run interactive queries <streams_concepts_interactive-queries> + to support fault-tolerant stateful processing + <streams_developer-guide_state-store_fault-tolerance>, + or to run interactive queries + <streams_concepts_interactive-queries> against your application's latest processing results. And, beyond its internal usage, the Kafka Streams API also allows developers to exploit this duality in their own applications.

- Before we discuss concepts such as aggregations <streams_concepts_aggregations> + Before we discuss concepts such as aggregations + <streams_concepts_aggregations>, in Kafka Streams we must first introduce tables in more detail, and talk about the aforementioned stream-table duality. Essentially, this duality means that a stream can be viewed as a table, and a table can be viewed as a stream.

From 5b0ee587e46a0195fe9a371cd22d0ce44657ef63 Mon Sep 17 00:00:00 2001 From: Victoria Bialas Date: Tue, 23 Apr 2019 18:36:07 -0700 Subject: [PATCH 6/8] same links on v20 Signed-off-by: Victoria Bialas --- 20/streams/core-concepts.html | 26 ++++++++++++++++---------- 1 file changed, 16 insertions(+), 10 deletions(-) diff --git a/20/streams/core-concepts.html b/20/streams/core-concepts.html index 594efaa5ed9..e522f3041c3 100644 --- a/20/streams/core-concepts.html +++ b/20/streams/core-concepts.html @@ -57,13 +57,14 @@

Core Concepts

We first summarize the key concepts of Kafka Streams.

+

Stream Processing Topology

  • A stream is the most important abstraction provided by Kafka Streams: it represents an unbounded, continuously updating data set. A stream is an ordered, replayable, and fault-tolerant sequence of immutable data records, where a data record is defined as a key-value pair.
  • A stream processing application is any program that makes use of the Kafka Streams library. It defines its computational logic through one or more processor topologies, where a processor topology is a graph of stream processors (nodes) that are connected by streams (edges).
    • -
  • A stream processor is a node in the processor topology; it represents a processing step to transform data in streams by receiving one input record at a time from its upstream processors in the topology, applying its operation to it, and may subsequently produce one or more output records to its downstream processors.
    • +
  • A stream processor is a node in the processor topology; it represents a processing step to transform data in streams by receiving one input record at a time from its upstream processors in the topology, applying its operation to it, and may subsequently produce one or more output records to its downstream processors.
There are two special processors in the topology: @@ -160,21 +161,26 @@

Duality of

Any stream processing technology must therefore provide first-class support for streams and tables. - Kafka's Streams API provides such functionality through its core abstractions for - streams <streams_concepts_kstream> and - tables <streams_concepts_ktable>, which we will talk about in a minute. - Now, an interesting observation is that there is actually a close relationship between streams and tables, - the so-called stream-table duality. - And Kafka exploits this duality in many ways: for example, to make your applications + Kafka's Streams API provides such functionality through its core abstractions for + streams + <streams_concepts_kstreams> + and tables + <streams_concepts_kstreams>, + which we will talk about in a minute. Now, an interesting observation is that there is actually a close relationship between streams and tables, + the so-called stream-table duality. And Kafka exploits this duality in many ways: for example, to make your applications + elastic elastic <streams_developer-guide_execution-scaling>, - to support fault-tolerant stateful processing <streams_developer-guide_state-store_fault-tolerance>, - or to run interactive queries <streams_concepts_interactive-queries> + to support fault-tolerant stateful processing + <streams_developer-guide_state-store_fault-tolerance>, + or to run interactive queries + <streams_concepts_interactive-queries> against your application's latest processing results. And, beyond its internal usage, the Kafka Streams API also allows developers to exploit this duality in their own applications.

- Before we discuss concepts such as aggregations <streams_concepts_aggregations> + Before we discuss concepts such as aggregations + <streams_concepts_aggregations>, in Kafka Streams we must first introduce tables in more detail, and talk about the aforementioned stream-table duality. Essentially, this duality means that a stream can be viewed as a table, and a table can be viewed as a stream.

From 94756e97c821e30ffbd002ecb9caf68de58ac12a Mon Sep 17 00:00:00 2001 From: Victoria Bialas Date: Tue, 23 Apr 2019 19:04:08 -0700 Subject: [PATCH 7/8] fixed self-referential links to stream processor Signed-off-by: Victoria Bialas --- 10/streams/core-concepts.html | 3 ++- 10/streams/developer-guide/processor-api.html | 2 +- 11/streams/core-concepts.html | 5 +++-- 11/streams/developer-guide/processor-api.html | 2 +- 4 files changed, 7 insertions(+), 5 deletions(-) diff --git a/10/streams/core-concepts.html b/10/streams/core-concepts.html index 81bfdf6a1f0..13a8b3ee8d8 100644 --- a/10/streams/core-concepts.html +++ b/10/streams/core-concepts.html @@ -57,13 +57,14 @@

Core Concepts

We first summarize the key concepts of Kafka Streams.

+

Stream Processing Topology

  • A stream is the most important abstraction provided by Kafka Streams: it represents an unbounded, continuously updating data set. A stream is an ordered, replayable, and fault-tolerant sequence of immutable data records, where a data record is defined as a key-value pair.
  • A stream processing application is any program that makes use of the Kafka Streams library. It defines its computational logic through one or more processor topologies, where a processor topology is a graph of stream processors (nodes) that are connected by streams (edges).
    • -
  • A stream processor is a node in the processor topology; it represents a processing step to transform data in streams by receiving one input record at a time from its upstream processors in the topology, applying its operation to it, and may subsequently produce one or more output records to its downstream processors.
    • +
  • A stream processor is a node in the processor topology; it represents a processing step to transform data in streams by receiving one input record at a time from its upstream processors in the topology, applying its operation to it, and may subsequently produce one or more output records to its downstream processors.
There are two special processors in the topology: diff --git a/10/streams/developer-guide/processor-api.html b/10/streams/developer-guide/processor-api.html index fdf6c86b4cd..1e1df6502df 100644 --- a/10/streams/developer-guide/processor-api.html +++ b/10/streams/developer-guide/processor-api.html @@ -66,7 +66,7 @@

Overview

Defining a Stream Processor

-

A stream processor is a node in the processor topology that represents a single processing step. +

A stream processor is a node in the processor topology that represents a single processing step. With the Processor API, you can define arbitrary stream processors that processes one received record at a time, and connect these processors with their associated state stores to compose the processor topology.

You can define a customized stream processor by implementing the Processor interface, which provides the process() API method. diff --git a/11/streams/core-concepts.html b/11/streams/core-concepts.html index 473a268b7a5..bd930de910a 100644 --- a/11/streams/core-concepts.html +++ b/11/streams/core-concepts.html @@ -57,13 +57,14 @@

Core Concepts

We first summarize the key concepts of Kafka Streams.

+

Stream Processing Topology

- +
  • A stream is the most important abstraction provided by Kafka Streams: it represents an unbounded, continuously updating data set. A stream is an ordered, replayable, and fault-tolerant sequence of immutable data records, where a data record is defined as a key-value pair.
  • A stream processing application is any program that makes use of the Kafka Streams library. It defines its computational logic through one or more processor topologies, where a processor topology is a graph of stream processors (nodes) that are connected by streams (edges).
    • -
  • A stream processor is a node in the processor topology; it represents a processing step to transform data in streams by receiving one input record at a time from its upstream processors in the topology, applying its operation to it, and may subsequently produce one or more output records to its downstream processors.
    • +
  • A stream processor is a node in the processor topology; it represents a processing step to transform data in streams by receiving one input record at a time from its upstream processors in the topology, applying its operation to it, and may subsequently produce one or more output records to its downstream processors.
There are two special processors in the topology: diff --git a/11/streams/developer-guide/processor-api.html b/11/streams/developer-guide/processor-api.html index fdf6c86b4cd..1e1df6502df 100644 --- a/11/streams/developer-guide/processor-api.html +++ b/11/streams/developer-guide/processor-api.html @@ -66,7 +66,7 @@

Overview

Defining a Stream Processor

-

A stream processor is a node in the processor topology that represents a single processing step. +

A stream processor is a node in the processor topology that represents a single processing step. With the Processor API, you can define arbitrary stream processors that processes one received record at a time, and connect these processors with their associated state stores to compose the processor topology.

You can define a customized stream processor by implementing the Processor interface, which provides the process() API method. From eb6b222a764605b796f660f4f7112a101e262c42 Mon Sep 17 00:00:00 2001 From: Victoria Bialas Date: Wed, 24 Apr 2019 11:39:00 -0700 Subject: [PATCH 8/8] Bill's review comments, removed placeholder link targets Signed-off-by: Victoria Bialas --- 21/streams/core-concepts.html | 14 ++++---------- 22/streams/core-concepts.html | 14 ++++---------- 2 files changed, 8 insertions(+), 20 deletions(-) diff --git a/21/streams/core-concepts.html b/21/streams/core-concepts.html index cc093b5c63f..29080221b4a 100644 --- a/21/streams/core-concepts.html +++ b/21/streams/core-concepts.html @@ -162,25 +162,19 @@

Duality of Any stream processing technology must therefore provide first-class support for streams and tables. Kafka's Streams API provides such functionality through its core abstractions for streams - <streams_concepts_kstreams> - and tables - <streams_concepts_kstreams>, + and tables, which we will talk about in a minute. Now, an interesting observation is that there is actually a close relationship between streams and tables, the so-called stream-table duality. And Kafka exploits this duality in many ways: for example, to make your applications - elastic - elastic <streams_developer-guide_execution-scaling>, - to support fault-tolerant stateful processing - <streams_developer-guide_state-store_fault-tolerance>, + elastic, + to support fault-tolerant stateful processing, or to run interactive queries - <streams_concepts_interactive-queries> against your application's latest processing results. And, beyond its internal usage, the Kafka Streams API also allows developers to exploit this duality in their own applications.

Before we discuss concepts such as aggregations - <streams_concepts_aggregations>, - in Kafka Streams we must first introduce tables in more detail, and talk about the aforementioned stream-table duality. + in Kafka Streams, we must first introduce tables in more detail, and talk about the aforementioned stream-table duality. Essentially, this duality means that a stream can be viewed as a table, and a table can be viewed as a stream.

diff --git a/22/streams/core-concepts.html b/22/streams/core-concepts.html index 670d35fa505..13c2a31b695 100644 --- a/22/streams/core-concepts.html +++ b/22/streams/core-concepts.html @@ -162,25 +162,19 @@

Duality of Any stream processing technology must therefore provide first-class support for streams and tables. Kafka's Streams API provides such functionality through its core abstractions for streams - <streams_concepts_kstreams> - and tables - <streams_concepts_kstreams>, + and tables, which we will talk about in a minute. Now, an interesting observation is that there is actually a close relationship between streams and tables, the so-called stream-table duality. And Kafka exploits this duality in many ways: for example, to make your applications - elastic - elastic <streams_developer-guide_execution-scaling>, - to support fault-tolerant stateful processing - <streams_developer-guide_state-store_fault-tolerance>, + elastic, + to support fault-tolerant stateful processing, or to run interactive queries - <streams_concepts_interactive-queries> against your application's latest processing results. And, beyond its internal usage, the Kafka Streams API also allows developers to exploit this duality in their own applications.

Before we discuss concepts such as aggregations - <streams_concepts_aggregations>, - in Kafka Streams we must first introduce tables in more detail, and talk about the aforementioned stream-table duality. + in Kafka Streams, we must first introduce tables in more detail, and talk about the aforementioned stream-table duality. Essentially, this duality means that a stream can be viewed as a table, and a table can be viewed as a stream.