Skip to content

MINOR: updated KS release notes for 3.5 #13577

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mjsax merged 1 commit into from
Apr 21, 2023
Merged

MINOR: updated KS release notes for 3.5 #13577

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 41 additions & 15 deletions docs/streams/upgrade-guide.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,16 +65,6 @@ <h1>Upgrade Guide and API Changes</h1>
</p>

<h3 class="anchor-heading"><a id="streams_notable_changes" class="anchor-link"></a><a href="#streams_notable_changes">Notable compatibility changes in past releases</a></h3>
<p>
Downgrading from 3.0.x or newer version to 2.8.x or older version needs special attention:
Since 3.0.0 release, Kafka Streams uses a newer RocksDB version whose on-disk format changed.
This means that old versioned RocksDB would not be able to recognize the bytes written by that newer versioned RocksDB,
and hence it is harder to downgrade Kafka Streams with version 3.0.0 or newer to older versions in-flight.
Users need to wipe out the local RocksDB state stores written by the new versioned Kafka Streams before swapping in the
older versioned Kafka Streams bytecode, which would then restore the state stores with the old on-disk format from the
changelogs.
</p>

<p>
Downgrading from 3.5.x or newer version to 3.4.x or older version needs special attention:
Since 3.5.0 release, Kafka Streams uses a new serialization format for repartition topics.
Expand All @@ -89,6 +79,16 @@ <h3 class="anchor-heading"><a id="streams_notable_changes" class="anchor-link"><
pre-3.5.x version.
</p>

<p>
Downgrading from 3.0.x or newer version to 2.8.x or older version needs special attention:
Since 3.0.0 release, Kafka Streams uses a newer RocksDB version whose on-disk format changed.
This means that old versioned RocksDB would not be able to recognize the bytes written by that newer versioned RocksDB,
and hence it is harder to downgrade Kafka Streams with version 3.0.0 or newer to older versions in-flight.
Users need to wipe out the local RocksDB state stores written by the new versioned Kafka Streams before swapping in the
older versioned Kafka Streams bytecode, which would then restore the state stores with the old on-disk format from the
changelogs.
</p>

<p>
Kafka Streams does not support running multiple instances of the same application as different processes on the same physical state directory. Starting in 2.8.0 (as well as 2.7.1 and 2.6.2),
this restriction will be enforced. If you wish to run more than one instance of Kafka Streams, you must configure them with different values for <code>state.dir</code>.
Expand Down Expand Up @@ -132,11 +132,6 @@ <h3 class="anchor-heading"><a id="streams_notable_changes" class="anchor-link"><
</p>

<h3><a id="streams_api_changes_350" href="#streams_api_changes_350">Streams API changes in 3.5.0</a></h3>
<p>
We added a new Serde type <code>Boolean</code> in
<a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-907%3A+Add+Boolean+Serde+to+public+interface">KIP-907</a>
</p>

<p>
A new state store type, versioned key-value stores, was introduced in
<a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-889%3A+Versioned+State+Stores">KIP-889</a>.
Expand All @@ -151,6 +146,37 @@ <h3><a id="streams_api_changes_350" href="#streams_api_changes_350">Streams API
affected upon upgrading to 3.5 without explicit code changes.
</p>

<p>
In addition to KIP-899, <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-914%3A+DSL+Processor+Semantics+for+Versioned+Stores">KIP-914</a>
updates DSL processing semantics if a user opts-in to use the new versioned key-value stores.
Using the new versioned key-value stores, DSL processing are able to handle out-of-order data better:
For example, late record may be dropped and stream-table joins do a timestamped based lookup into the table.
Table aggregations and primary/foreign-key table-table joins are also improved.
Note: versioned key-value stores are not supported for global-KTable and don't work with <code>suppress()</code>.
</p>

<p>
<a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-904%3A+Kafka+Streams+-+Guarantee+subtractor+is+called+before+adder+if+key+has+not+changed">KIP-904</a>
improves the implemenation of KTable aggregations. In general, an input KTable update triggers a result refinent for two rows;
however, prior to KIP-904, if both refinements happend to the same result row, two independent updates to the same row are applied, resulting in spurious itermediate results.
KIP-904 allows us to detect this case, and to only apply a single update avoiding spurious intermediate results.
</p>

<p>
Error handling is improved via <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-399%3A+Extend+ProductionExceptionHandler+to+cover+serialization+exceptions">KIP-399</a>.
The existing <code>ProductionExceptionHandler</code> now also covers serialization errors.
</p>

<p>
We added a new Serde type <code>Boolean</code> in
<a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-907%3A+Add+Boolean+Serde+to+public+interface">KIP-907</a>
</p>

<p>
<a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-884%3A+Add+config+to+configure+KafkaClientSupplier+in+Kafka+Streams">KIP-884</a>
adds a new config <code>default.client.supplier</code> that allows to use a custom <code>KafkaClientSupplier</code> without any code changes.
</p>

<h3><a id="streams_api_changes_310" href="#streams_api_changes_310">Streams API changes in 3.1.0</a></h3>
<p>
The semantics of left/outer stream-stream join got improved via
Expand Down
10 changes: 10 additions & 0 deletions docs/upgrade.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,16 @@ <h5><a id="upgrade_350_notable" href="#upgrade_350_notable">Notable changes in 3
operations to return the latest record (per key) as of a specified timestamp.
See <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-889%3A+Versioned+State+Stores">KIP-889</a>
for more details.
If the new store typed is used in the DSL, improved processing semantics are applied as described in
<a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-914%3A+DSL+Processor+Semantics+for+Versioned+Stores">KIP-914</a>.
</li>
<li>KTable aggregation semantics got further improved via
<a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-904%3A+Kafka+Streams+-+Guarantee+subtractor+is+called+before+adder+if+key+has+not+changed">KIP-904</a>,
now avoiding spurious itermedite results.
</li>
<li>Kafka Streams' <code>ProductionExceptionHandler</code> is improved via
<a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-399%3A+Extend+ProductionExceptionHandler+to+cover+serialization+exceptions">KIP-399</a>,
now also covering serialiation errors.
</li>
<li>MirrorMaker now uses incrementalAlterConfigs API by default to synchronize topic configurations instead of the deprecated alterConfigs API.
A new settings called <code>use.incremental.alter.configs</code> is introduced to allow users to control which API to use.
Expand Down