Tombstone retention #829

kbatuigas · 2024-10-28T14:11:50Z

Description

Support for topic level delete.retention.ms and cluster level tombstone_retention_ms properties, which define the retention limit for tombstone records in compacted topics.

TODO:

A separate PR will add the tombstone_retention_ms reference entry: add Tombstone property #847
We will also have to update https://docs.redpanda.com/current/develop/kafka-clients/#unsupported-kafka-features when this is available.

Resolves https://github.com/redpanda-data/documentation-private/issues/2600
Review deadline: Nov 12

Page previews

24.3 Beta > Compaction settings > Tombstone record removal
Topic properties > delete.retention.ms
What's new

Checks

New feature
Content gap
Support Follow-up
Small fix (typos, links, copyedits, etc)

netlify · 2024-10-28T14:12:05Z

✅ Deploy Preview for redpanda-docs-preview ready!

Name	Link
🔨 Latest commit	`12ee583`
🔍 Latest deploy log	https://app.netlify.com/sites/redpanda-docs-preview/deploys/6733a6f991973d0008c2f54a
😎 Deploy Preview	https://deploy-preview-829--redpanda-docs-preview.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

modules/manage/pages/cluster-maintenance/compaction-settings.adoc

modules/reference/pages/properties/topic-properties.adoc

modules/manage/pages/cluster-maintenance/compaction-settings.adoc

WillemKauf

It's looking good! A small number of comments on my end.

modules/get-started/pages/whats-new.adoc

modules/manage/pages/cluster-maintenance/compaction-settings.adoc

Deflaimun · 2024-11-12T17:12:05Z

@kbatuigas feel free to merge this one when you merge yours
#847

modules/manage/pages/cluster-maintenance/compaction-settings.adoc

Deflaimun · 2024-11-12T18:19:30Z

modules/manage/pages/cluster-maintenance/compaction-settings.adoc

+
+== Tombstone record removal
+
+Compaction also enables deletion of existing records through tombstones. For example, as data is deleted from a source system, clients produce a tombstone record to the log. A tombstone record contains the key and the value `null`. Tombstones signal to brokers and consumers that records with the same key prior to it in the log should be deleted. 


maybe we can add somewhere here what a tombstone is

Can link to other article in compaction which describes it

Deflaimun · 2024-11-12T18:20:18Z

modules/reference/pages/properties/topic-properties.adoc

+[[deleteretentionms]]
+==== delete.retention.ms
+
+The retention time (in ms) for tombstone records in a compacted topic. Redpanda removes tombstone records after the retention limit is exceeded.


I would remove (in ms) from the description and add a new field "Units: milliseconds " like we have on cluster properties.

Deflaimun · 2024-11-12T18:21:00Z

modules/manage/pages/cluster-maintenance/compaction-settings.adoc

+
+You cannot enable `tombstone_retention_ms` if you have enabled any of the Tiered Storage cluster properties `cloud_storage_enabled`, `cloud_storage_enable_remote_read`, and `cloud_storage_enable_remote_write`.
+
+On the topic level, you cannot enable `delete.retention.ms` at the same time as the Tiered Storage topic configuration properties `redpanda.remote.read` and `redpanda.remote.write`.


can I enable delete.retention.ms topic property if I have the Tiered Storage cluster properties?

Yes. Cluster properties are coupled and topic properties are coupled but the delete.retention.ms topic property is not coupled to the cluster properties.

Deflaimun

lgtm. Check comments before merging. If you need to address them later, consider creating a new ticket

modules/manage/pages/cluster-maintenance/compaction-settings.adoc

Co-authored-by: Joyce Fee <[email protected]>

Deflaimun force-pushed the DOC-386-Document-feature-Tombstone-retention-policies-Support-delete_retention_ms-in-the-way-it-works-in-Kafka branch 2 times, most recently from bf75351 to 60dfd47 Compare October 28, 2024 14:24

kbatuigas commented Nov 7, 2024

View reviewed changes

modules/manage/pages/cluster-maintenance/compaction-settings.adoc Outdated Show resolved Hide resolved

kbatuigas marked this pull request as ready for review November 7, 2024 21:17

kbatuigas requested a review from a team as a code owner November 7, 2024 21:17

kbatuigas changed the base branch from main to v-WIP/24.3 November 7, 2024 21:19

kbatuigas requested a review from WillemKauf November 7, 2024 21:20

WillemKauf reviewed Nov 7, 2024

View reviewed changes

kbatuigas commented Nov 8, 2024

View reviewed changes

modules/manage/pages/cluster-maintenance/compaction-settings.adoc Outdated Show resolved Hide resolved

kbatuigas commented Nov 8, 2024

View reviewed changes

modules/reference/pages/properties/topic-properties.adoc Show resolved Hide resolved

kbatuigas requested a review from WillemKauf November 8, 2024 21:12

kbatuigas commented Nov 11, 2024

View reviewed changes

modules/manage/pages/cluster-maintenance/compaction-settings.adoc Outdated Show resolved Hide resolved

kbatuigas added 7 commits November 11, 2024 09:15

Start section for tombstone retention

9231263

rpk steps

e07cabf

Add topic config reference

df86ab5

Edits per SME review

f6968c3

Try to clarify topic tombstone removal

6c56f2f

Edits per SME feedback

6b73c14

Add tombstone retention to What's New

75841d3

kbatuigas force-pushed the DOC-386-Document-feature-Tombstone-retention-policies-Support-delete_retention_ms-in-the-way-it-works-in-Kafka branch from eb182e0 to 75841d3 Compare November 11, 2024 14:19

WillemKauf reviewed Nov 11, 2024

View reviewed changes

modules/get-started/pages/whats-new.adoc Outdated Show resolved Hide resolved

modules/manage/pages/cluster-maintenance/compaction-settings.adoc Outdated Show resolved Hide resolved

modules/manage/pages/cluster-maintenance/compaction-settings.adoc Outdated Show resolved Hide resolved

kbatuigas requested a review from mattschumpert November 11, 2024 21:50

kbatuigas added 2 commits November 11, 2024 17:57

Edits per SME review

023525e

Add xrefs

90202d4

kbatuigas requested a review from WillemKauf November 11, 2024 23:01

Replace 'message' with 'record' per SME guidance

62f3fb3

mattschumpert approved these changes Nov 12, 2024

View reviewed changes

modules/manage/pages/cluster-maintenance/compaction-settings.adoc Outdated Show resolved Hide resolved

WillemKauf reviewed Nov 12, 2024

View reviewed changes

modules/manage/pages/cluster-maintenance/compaction-settings.adoc Outdated Show resolved Hide resolved

kbatuigas added 2 commits November 12, 2024 13:11

Update What's new and Unsupported features per PM feedback

e2ecd8e

Apply suggestion from review

be2146b

Deflaimun reviewed Nov 12, 2024

View reviewed changes

Deflaimun approved these changes Nov 12, 2024

View reviewed changes

WillemKauf approved these changes Nov 12, 2024

View reviewed changes

Match layout of cluster property entries

67ff4a9

Feediver1 reviewed Nov 12, 2024

View reviewed changes

modules/manage/pages/cluster-maintenance/compaction-settings.adoc Outdated Show resolved Hide resolved

Feediver1 approved these changes Nov 12, 2024

View reviewed changes

kbatuigas and others added 2 commits November 12, 2024 14:04

Minor edit

f23861e

Update modules/manage/pages/cluster-maintenance/compaction-settings.adoc

12ee583

Co-authored-by: Joyce Fee <[email protected]>

kbatuigas merged commit 58ded83 into v-WIP/24.3 Nov 12, 2024
3 checks passed

kbatuigas deleted the DOC-386-Document-feature-Tombstone-retention-policies-Support-delete_retention_ms-in-the-way-it-works-in-Kafka branch November 12, 2024 19:06

kbatuigas added a commit that referenced this pull request Nov 13, 2024

Tombstone retention (#829)

866688b

Co-authored-by: Joyce Fee <[email protected]>

Deflaimun pushed a commit that referenced this pull request Nov 19, 2024

Tombstone retention (#829)

440bdd4

Co-authored-by: Joyce Fee <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tombstone retention #829

Tombstone retention #829

kbatuigas commented Oct 28, 2024 •

edited

Loading

netlify bot commented Oct 28, 2024 •

edited

Loading

WillemKauf left a comment

Deflaimun commented Nov 12, 2024

Deflaimun Nov 12, 2024

WillemKauf Nov 12, 2024

Deflaimun Nov 12, 2024

Deflaimun Nov 12, 2024 •

edited

Loading

WillemKauf Nov 12, 2024

Deflaimun left a comment


		== Tombstone record removal

		Compaction also enables deletion of existing records through tombstones. For example, as data is deleted from a source system, clients produce a tombstone record to the log. A tombstone record contains the key and the value `null`. Tombstones signal to brokers and consumers that records with the same key prior to it in the log should be deleted.


		You cannot enable `tombstone_retention_ms` if you have enabled any of the Tiered Storage cluster properties `cloud_storage_enabled`, `cloud_storage_enable_remote_read`, and `cloud_storage_enable_remote_write`.

		On the topic level, you cannot enable `delete.retention.ms` at the same time as the Tiered Storage topic configuration properties `redpanda.remote.read` and `redpanda.remote.write`.

Tombstone retention #829

Tombstone retention #829

Conversation

kbatuigas commented Oct 28, 2024 • edited Loading

Description

Page previews

Checks

netlify bot commented Oct 28, 2024 • edited Loading

✅ Deploy Preview for redpanda-docs-preview ready!

WillemKauf left a comment

Choose a reason for hiding this comment

Deflaimun commented Nov 12, 2024

Deflaimun Nov 12, 2024

Choose a reason for hiding this comment

WillemKauf Nov 12, 2024

Choose a reason for hiding this comment

Deflaimun Nov 12, 2024

Choose a reason for hiding this comment

Deflaimun Nov 12, 2024 • edited Loading

Choose a reason for hiding this comment

WillemKauf Nov 12, 2024

Choose a reason for hiding this comment

Deflaimun left a comment

Choose a reason for hiding this comment

kbatuigas commented Oct 28, 2024 •

edited

Loading

netlify bot commented Oct 28, 2024 •

edited

Loading

Deflaimun Nov 12, 2024 •

edited

Loading