From 9ed5e7ecf2b74d55a940dda60f1f911564af1e85 Mon Sep 17 00:00:00 2001
From: "Matthias J. Sax" <matthias@confluent.io>
Date: Fri, 17 May 2019 20:18:39 +0200
Subject: [PATCH 1/4] MINOR: add Kafka Streams upgrade notes for 2.3 release

---
 docs/streams/upgrade-guide.html | 42 +++++++++++++++++++++++++++++++++
 docs/upgrade.html               | 10 ++++++++
 2 files changed, 52 insertions(+)

diff --git a/docs/streams/upgrade-guide.html b/docs/streams/upgrade-guide.html
index e07b5a9acd844..68826f504f977 100644
--- a/docs/streams/upgrade-guide.html
+++ b/docs/streams/upgrade-guide.html
@@ -70,6 +70,48 @@ <h1>Upgrade Guide and API Changes</h1>
         More details about the new config <code>StreamsConfig#TOPOLOGY_OPTIMIZATION</code> can be found in <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-295%3A+Add+Streams+Configuration+Allowing+for+Optional+Topology+Optimization">KIP-295</a>.
     </p>
 
+    <h3><a id="streams_api_changes_230" href="#streams_api_changes_230">Streams API changes in 2.3.0</a></h3>
+    <p>
+        To improve operator semantics, new store types are added that allow storing an additional timestamp per key-value pair or window.
+        Some DSL operators (for example KTables) are using those new stores.
+        Hence, you can now receive the last update timestamp via Interactive Queries if you change the <code>QueryableStoreType</code>.
+        While this change is mainly transparent to the user, there are some corner cases that may require code changes:
+        <strong>Caution: If you receive an untyped store and use a cast, you might need to update your code to cast to the correct type.</strong>
+        Additionally, <code>TopologyTestDriver#getStateStore()</code> only returns non-built-in stores and throws an exception if a built-in store is accessed.
+        For more details please read <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-258%3A+Allow+to+Store+Record+Timestamps+in+RocksDB">KIP-258</a>.
+    </p>
+
+    <p>
+        To improve type safety, a new operator <code>KStream#flatTransformValues</code> is added.
+        For more details please read <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-313%3A+Add+KStream.flatTransform+and+KStream.flatTransformValues">KIP-313</a>.
+    </p>
+
+    <p>
+        In previous release, Kafka Streams only shipped with a built-in in-memory key-value store.
+        In this release, in-memory window store (<a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-428%3A+Add+in-memory+window+store">KIP_428</a>)
+        and in-memory session store (<a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-445%3A+In-memory+Session+Store">KIP-445</a>) are added.
+    </p>
+
+    <p>
+	Kafka Streams used to set the configuration parameter <code>max.poll.interval.ms</code> to <code>Interger.MAX_VALUE</code>.
+        In default value is removed and Kafka Streams uses the consumer default value now.
+        For more details please read <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-442%3A+Return+to+default+max+poll+interval+in+Streams">KIP-442</a>.
+    </p>
+
+    <p>
+        Default configuration for repartition topic was changed:
+	The segment size for index files (<code>segment.index.bytes</code>) is no longer 50MB, but uses the cluster default.
+        Similarly, the configuration <code>segment.ms</code> in no longer 10 minutes, but uses the cluster default configuration.
+        Lastly, the retention period (<code>retention.ms</code>) is changed from <code>Long.MAX_VALUE</code> to <code>-1</code> (infinite).
+        For more details please read <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-443%3A+Return+to+default+segment.ms+and+segment.index.bytes+in+Streams+repartition+topics">KIP-443</a>.
+    </p>
+
+    <p>
+        To avoid memory leaks, <code>RocksDBConfigSetter</code> has a new <code>close()</code> method that is called on shutdown.
+        Users should implement this method to release any memory used by RocksDB config objects, by closing those objects.
+        For more details please read <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-453%3A+Add+close%28%29+method+to+RocksDBConfigSetter">KIP-453</a>.
+    </p>
+
     <h3><a id="streams_api_changes_230" href="#streams_api_changes_230">Streams API changes in 2.3.0</a></h3>
     <p>Version 2.3.0 adds the Suppress operator to the <code>kafka-streams-scala</code> Ktable API.</p>
     <p>
diff --git a/docs/upgrade.html b/docs/upgrade.html
index 1828ab7813991..be9886cb990e2 100644
--- a/docs/upgrade.html
+++ b/docs/upgrade.html
@@ -19,6 +19,16 @@
 
 <script id="upgrade-template" type="text/x-handlebars-template">
 
+<h4><a id="upgrade_2_3_0" href="#upgrade_2_3_0">Upgrading from 0.8.x, 0.9.x, 0.10.0.x, 0.10.1.x, 0.10.2.x, 0.11.0.x, 1.0.x, 1.1.x, 2.0.x or 2.1.x or 2.2.x to 2.3.0</a></h4>
+
+<!-- TODO core-team -->
+
+<h5><a id="upgrade_230_notable" href="#upgrade_230_notable">Notable changes in 2.3.0</a></h5>
+<ul>
+    <li> Kafka Streams DSL switches its used store types. While this change is mainly transparent to users, there are some corner cases that may require code changes.
+	See the <a href="/{{version}}/documentation/streams/upgrade-guide#streams_api_changes_230">Kafka Streams upgrade section</a> for more details.</li>
+</ul>
+
 <h4><a id="upgrade_2_2_0" href="#upgrade_2_2_0">Upgrading from 0.8.x, 0.9.x, 0.10.0.x, 0.10.1.x, 0.10.2.x, 0.11.0.x, 1.0.x, 1.1.x, 2.0.x or 2.1.x to 2.2.0</a></h4>
 
 <p><b>If you are upgrading from a version prior to 2.1.x, please see the note below about the change to the schema used to store consumer offsets.

From 519d18be575899ecba00fa86ee05a2c055929fe8 Mon Sep 17 00:00:00 2001
From: "Matthias J. Sax" <matthias@confluent.io>
Date: Sun, 19 May 2019 14:03:43 +0200
Subject: [PATCH 2/4] Github comments

---
 docs/streams/upgrade-guide.html | 36 ++++++++++++++++++---------------
 1 file changed, 20 insertions(+), 16 deletions(-)

diff --git a/docs/streams/upgrade-guide.html b/docs/streams/upgrade-guide.html
index 68826f504f977..ed5ef1400bff1 100644
--- a/docs/streams/upgrade-guide.html
+++ b/docs/streams/upgrade-guide.html
@@ -71,6 +71,21 @@ <h1>Upgrade Guide and API Changes</h1>
     </p>
 
     <h3><a id="streams_api_changes_230" href="#streams_api_changes_230">Streams API changes in 2.3.0</a></h3>
+
+    <p>Version 2.3.0 adds the Suppress operator to the <code>kafka-streams-scala</code> Ktable API.</p>
+
+    <p>
+        As of 2.3.0 Streams now offers an in-memory version of the window (<a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-428%3A+Add+in-memory+window+store">KIP-428</a>)
+        and the session (<a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-445%3A+In-memory+Session+Store">KIP-445</a>) store, in addition to the persistent ones based on RocksDB.
+        The new public interfaces <code>inMemoryWindowStore()</code> and <code>inMemorySessionStore()</code> are added to <code>Stores</code> and provide the built-in in-memory window or session store.
+    </p>
+
+    <p>
+        In 2.3.0 we have added default implementation to <code>close()</code> and <code>configure()</code> for <code>Serializer</code>,
+        <code>Deserializer</code> and <code>Serde</code> so that they can be implemented by lambda expression.
+        For more details please read <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-331+Add+default+implementation+to+close%28%29+and+configure%28%29+for+Serializer%2C+Deserializer+and+Serde">KIP-331</a>.
+    </p>
+
     <p>
         To improve operator semantics, new store types are added that allow storing an additional timestamp per key-value pair or window.
         Some DSL operators (for example KTables) are using those new stores.
@@ -87,20 +102,14 @@ <h3><a id="streams_api_changes_230" href="#streams_api_changes_230">Streams API
     </p>
 
     <p>
-        In previous release, Kafka Streams only shipped with a built-in in-memory key-value store.
-        In this release, in-memory window store (<a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-428%3A+Add+in-memory+window+store">KIP_428</a>)
-        and in-memory session store (<a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-445%3A+In-memory+Session+Store">KIP-445</a>) are added.
-    </p>
-
-    <p>
-	Kafka Streams used to set the configuration parameter <code>max.poll.interval.ms</code> to <code>Interger.MAX_VALUE</code>.
+        Kafka Streams used to set the configuration parameter <code>max.poll.interval.ms</code> to <code>Interger.MAX_VALUE</code>.
         In default value is removed and Kafka Streams uses the consumer default value now.
         For more details please read <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-442%3A+Return+to+default+max+poll+interval+in+Streams">KIP-442</a>.
     </p>
 
     <p>
         Default configuration for repartition topic was changed:
-	The segment size for index files (<code>segment.index.bytes</code>) is no longer 50MB, but uses the cluster default.
+        The segment size for index files (<code>segment.index.bytes</code>) is no longer 50MB, but uses the cluster default.
         Similarly, the configuration <code>segment.ms</code> in no longer 10 minutes, but uses the cluster default configuration.
         Lastly, the retention period (<code>retention.ms</code>) is changed from <code>Long.MAX_VALUE</code> to <code>-1</code> (infinite).
         For more details please read <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-443%3A+Return+to+default+segment.ms+and+segment.index.bytes+in+Streams+repartition+topics">KIP-443</a>.
@@ -112,15 +121,10 @@ <h3><a id="streams_api_changes_230" href="#streams_api_changes_230">Streams API
         For more details please read <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-453%3A+Add+close%28%29+method+to+RocksDBConfigSetter">KIP-453</a>.
     </p>
 
-    <h3><a id="streams_api_changes_230" href="#streams_api_changes_230">Streams API changes in 2.3.0</a></h3>
-    <p>Version 2.3.0 adds the Suppress operator to the <code>kafka-streams-scala</code> Ktable API.</p>
-    <p>
-        As of 2.3.0 Streams now offers an in-memory version of the window and the session store, in addition to the persistent ones based on RocksDB. The new public interfaces <code>inMemoryWindowStore()</code> and <code>inMemorySessionStore()</code> are added to Stores and provide the built-in in-memory window or session store.
-    </p>
-
     <p>
-        In 2.3.0 we have added default implementation to close() and configure() for <code>Serializer</code>, <code>Deserializer</code> and <code>Serde</code> so that they can be
-        implemented by lambda expression. For more details please read <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-331+Add+default+implementation+to+close%28%29+and+configure%28%29+for+Serializer%2C+Deserializer+and+Serde">KIP-331</a>.
+        RocksDB dependency was updated to version <code>5.18.3</code>.
+        The new version allows to specify more RocksDB configurations, including <code>WriteBufferManager</code> that help to limit RocksDB off-head memory usage.
+        For more details please read <a href="https://issues.apache.org/jira/browse/KAFKA-8215">KAFKA-8215</a>.
     </p>
 
     <h3><a id="streams_api_changes_220" href="#streams_api_changes_220">Streams API changes in 2.2.0</a></h3>

From 738e3b4c98f94684211afea67c68be9990b8cdf4 Mon Sep 17 00:00:00 2001
From: "Matthias J. Sax" <matthias@confluent.io>
Date: Tue, 21 May 2019 22:18:00 +0200
Subject: [PATCH 3/4] Github comments

---
 docs/streams/upgrade-guide.html | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/docs/streams/upgrade-guide.html b/docs/streams/upgrade-guide.html
index ed5ef1400bff1..e3b9c4efe5476 100644
--- a/docs/streams/upgrade-guide.html
+++ b/docs/streams/upgrade-guide.html
@@ -89,9 +89,13 @@ <h3><a id="streams_api_changes_230" href="#streams_api_changes_230">Streams API
     <p>
         To improve operator semantics, new store types are added that allow storing an additional timestamp per key-value pair or window.
         Some DSL operators (for example KTables) are using those new stores.
-        Hence, you can now receive the last update timestamp via Interactive Queries if you change the <code>QueryableStoreType</code>.
-        While this change is mainly transparent to the user, there are some corner cases that may require code changes:
-        <strong>Caution: If you receive an untyped store and use a cast, you might need to update your code to cast to the correct type.</strong>
+        Hence, you can now retrieve the last update timestamp via Interactive Queries if you specify
+        <code>TimestampedKeyValueStoreType</code> or <code>TimestampedWindowStoreType</code> as your <code>QueryableStoreType</code>.
+        While this change is mainly transparent, there are some corner cases that may require code changes:
+        <strong>Caution: If you receive an untyped store and use a cast, you might need to update your code to cast to the correct type.
+        Otherwise, you might get an exception similar to
+        <code>java.lang.ClassCastException: class org.apache.kafka.streams.state.ValueAndTimestamp cannot be cast to class YOUR-VALUE-TYPE</code>
+        upon getting a value from the store.</strong>
         Additionally, <code>TopologyTestDriver#getStateStore()</code> only returns non-built-in stores and throws an exception if a built-in store is accessed.
         For more details please read <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-258%3A+Allow+to+Store+Record+Timestamps+in+RocksDB">KIP-258</a>.
     </p>

From bef0240ac300e4758943a7528773d138edc55589 Mon Sep 17 00:00:00 2001
From: "Matthias J. Sax" <matthias@confluent.io>
Date: Thu, 23 May 2019 23:17:31 -0700
Subject: [PATCH 4/4] Github comment

---
 docs/streams/upgrade-guide.html | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/docs/streams/upgrade-guide.html b/docs/streams/upgrade-guide.html
index e3b9c4efe5476..f4c5d48c4cd48 100644
--- a/docs/streams/upgrade-guide.html
+++ b/docs/streams/upgrade-guide.html
@@ -106,8 +106,8 @@ <h3><a id="streams_api_changes_230" href="#streams_api_changes_230">Streams API
     </p>
 
     <p>
-        Kafka Streams used to set the configuration parameter <code>max.poll.interval.ms</code> to <code>Interger.MAX_VALUE</code>.
-        In default value is removed and Kafka Streams uses the consumer default value now.
+        Kafka Streams used to set the configuration parameter <code>max.poll.interval.ms</code> to <code>Integer.MAX_VALUE</code>.
+        This default value is removed and Kafka Streams uses the consumer default value now.
         For more details please read <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-442%3A+Return+to+default+max+poll+interval+in+Streams">KIP-442</a>.
     </p>