Skip to content
Open
Show file tree
Hide file tree
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 10 additions & 10 deletions modules/manage/pages/schema-reg/schema-id-validation.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -87,8 +87,8 @@ When <<enable-schema-id-validation,schema ID validation is enabled>>, Redpanda u

To customize the subject name strategy per topic, set the following client topic properties:

* Set `redpanda.key.schema.id.validation` to `true` to enable key schema ID validation for the topic, and set `redpanda.key.subject.name.strategy` to the desired subject name strategy for keys of the topic (default: `TopicNameStrategy`).
* Set `redpanda.value.schema.id.validation` to `true` to enable value schema ID validation for the topic, and set `redpanda.value.subject.name.strategy` to the desired subject name strategy for values of the topic (default: `TopicNameStrategy`).
* Set xref:reference:properties/topic-properties.adoc#redpandakeyschemavalidation[`redpanda.key.schema.id.validation`] to `true` to enable key schema ID validation for the topic, and set xref:reference:properties/topic-properties.adoc#redpandakeysubjectnamestrategy[`redpanda.key.subject.name.strategy`] to the desired subject name strategy for keys of the topic (default: `TopicNameStrategy`).
* Set xref:reference:properties/topic-properties.adoc#redpandavalueschemavalidation[`redpanda.value.schema.id.validation`] to `true` to enable value schema ID validation for the topic, and set xref:reference:properties/topic-properties.adoc#redpandavaluesubjectnamestrategy[`redpanda.value.subject.name.strategy`] to the desired subject name strategy for values of the topic (default: `TopicNameStrategy`).

[NOTE]
====
Expand All @@ -97,17 +97,17 @@ The `redpanda.` properties have corresponding `confluent.` properties.
|===
| Redpanda property | Confluent property

| `redpanda.key.schema.id.validation`
| `confluent.key.schema.validation`
| xref:reference:properties/topic-properties.adoc#redpandakeyschemavalidation[`redpanda.key.schema.id.validation`]
| xref:reference:properties/topic-properties.adoc#confluentkeyschemavalidation[`confluent.key.schema.validation`]

| `redpanda.key.subject.name.strategy`
| `confluent.key.subject.name.strategy`
| xref:reference:properties/topic-properties.adoc#redpandakeysubjectnamestrategy[`redpanda.key.subject.name.strategy`]
| xref:reference:properties/topic-properties.adoc#confluentkeysubjectnamestrategy[`confluent.key.subject.name.strategy`]

| `redpanda.value.schema.id.validation`
| `confluent.value.schema.validation`
| xref:reference:properties/topic-properties.adoc#redpandavalueschemavalidation[`redpanda.value.schema.id.validation`]
| xref:reference:properties/topic-properties.adoc#confluentvalueschemavalidation[`confluent.value.schema.validation`]

| `redpanda.value.subject.name.strategy`
| `confluent.value.subject.name.strategy`
| xref:reference:properties/topic-properties.adoc#redpandavaluesubjectnamestrategy[`redpanda.value.subject.name.strategy`]
| xref:reference:properties/topic-properties.adoc#confluentvaluesubjectnamestrategy[`confluent.value.subject.name.strategy`]
|===

The `redpanda.*` and `confluent.*` properties are compatible. Either or both can be set simultaneously.
Expand Down
2 changes: 1 addition & 1 deletion modules/manage/partials/tiered-storage.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -1186,7 +1186,7 @@ rpk cluster config set cloud_storage_enable_segment_uploads true
For some applications, where the newest data is more valuable than historical data, data accumulation can be worse than data loss. In such cases, where you cannot afford to lose the most recently-produced data by rejecting produce requests after producers have filled the local disks during the period of paused uploads, there is a less safe pause and resume mechanism. This mechanism prioritizes the ability to receive new data over retaining data that cannot be uploaded when disks are full:

- Set the xref:reference:properties/object-storage-properties.adoc#cloud_storage_enable_remote_allow_gaps[`cloud_storage_enable_remote_allow_gaps`] cluster configuration property to `true`. This allows for gaps in the logs of all Tiered Storage topics in the cluster.
- Set the `redpanda.remote.allow_gaps` configuration property to `true`. This allows gaps for one specific topic. This topic-level configuration option overrides the cluster-level default.
- Set the xref:reference:properties/topic-properties.adoc#redpandaremoteallowgaps[`redpanda.remote.allowgaps`] topic property to `true`. This allows gaps for one specific topic. This topic-level configuration option overrides the cluster-level default.

When you pause uploads and set one of these properties to `true`, there may be gaps in the range of offsets stored in object storage. You can seamlessly resume uploads by setting `*allow_gaps` to `true` at either the cluster or topic level. If set to `false`, disk space could be depleted and produce requests would be throttled.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -712,6 +712,7 @@ When set to `true`, Redpanda can re-upload data for compacted topics to object s

---

[[cloud_storage_enable_remote_allow_gaps]]
=== cloud_storage_enable_remote_allow_gaps

Controls the eviction of locally stored log segments when Tiered Storage uploads are paused. Set to `false` to only evict data that has already been uploaded to object storage. If the retained data fills the local volume, Redpanda throttles producers. Set to `true` to allow the eviction of locally stored log segments, which may create gaps in offsets.
Expand Down Expand Up @@ -790,7 +791,7 @@ Enables adjacent segment merging. The segments are reuploaded if there is an opp

=== cloud_storage_enable_segment_uploads

Controls the upload of log segments to Tiered Storage. If set to `false`, this property temporarily pauses all log segment uploads from the Redpanda cluster. When the uploads are paused, the <<cloud_storage_enable_remote_allow_gaps, `cloud_storage_enable_remote_allow_gaps`>> cluster configuration and `redpanda.remote.allowgaps` topic properties control local retention behavior.
Controls the upload of log segments to Tiered Storage. If set to `false`, this property temporarily pauses all log segment uploads from the Redpanda cluster. When the uploads are paused, the <<cloud_storage_enable_remote_allow_gaps, `cloud_storage_enable_remote_allow_gaps`>> cluster configuration and xref:properties/topic-properties.adoc#redpandaremoteallowgaps[`redpanda.remote.allowgaps`] topic properties control local retention behavior.

*Requires restart:* No

Expand Down
149 changes: 148 additions & 1 deletion modules/reference/pages/properties/topic-properties.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,9 @@ include::develop:partial$topic-properties-warning.adoc[]
| <<replicationfactor,`replication.factor`>>
| xref:./cluster-properties.adoc#default_topic_replications[`default_topic_replications`]

| <<redpandaremoteallowgaps,`redpanda.remote.allowgaps`>>
| xref:./object-storage-properties.adoc#cloud_storage_enable_remote_allow_gaps[`cloud_storage_enable_remote_allow_gaps`]

| <<retentionbytes,`retention.bytes`>>
| xref:./cluster-properties.adoc#retention_bytes[`retention_bytes`]

Expand Down Expand Up @@ -594,6 +597,24 @@ A flag for enabling Redpanda to upload data for a topic from local storage to ob

- xref:manage:tiered-storage.adoc[Tiered Storage]

---
[[redpandaremoteallowgaps]]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Lazin can you review this? Thanks

Copy link
Collaborator Author

@paulohtb6 paulohtb6 Oct 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mattschumpert requested that we do not include this prop in docs. I'll remove it

=== redpanda.remote.allowgaps

Controls the eviction of locally stored log segments when Tiered Storage uploads are paused for this topic. Set to `false` (default) to only evict data that has already been uploaded to object storage. If the retained data fills the local volume, Redpanda throttles producers. Set to `true` to allow the eviction of locally stored log segments, which may create gaps in offsets.

This topic-level property overrides the cluster-level xref:./object-storage-properties.adoc#cloud_storage_enable_remote_allow_gaps[`cloud_storage_enable_remote_allow_gaps`] property for the topic.

*Type:* boolean

*Default:* `false`

*Related cluster property:* xref:./object-storage-properties.adoc#cloud_storage_enable_remote_allow_gaps[`cloud_storage_enable_remote_allow_gaps`]

**Related topics**:

- xref:manage:tiered-storage.adoc[Tiered Storage]

---
[[retentionlocaltargetbytes]]
=== retention.local.target.bytes
Expand Down Expand Up @@ -739,4 +760,130 @@ Controls how often the data in the Iceberg table is refreshed with new data from

**Related topics**:

- xref:manage:iceberg/about-iceberg-topics.adoc[]
- xref:manage:iceberg/about-iceberg-topics.adoc[]

---
== Schema Registry and Validation Properties

These properties control server-side schema ID validation for topics when using Schema Registry.

[[redpandakeyschemavalidation]]
=== redpanda.key.schema.id.validation

Enable validation of the schema ID for keys on a record. When enabled, Redpanda validates that the schema ID encoded in the record's key is registered in the Schema Registry according to the configured subject name strategy.

*Type:* boolean

*Default:* `false`

**Related topics**:

- xref:manage:schema-reg/schema-id-validation.adoc[Server-Side Schema ID Validation]

---
[[redpandakeysubjectnamestrategy]]
=== redpanda.key.subject.name.strategy

The subject name strategy for keys when `redpanda.key.schema.id.validation` is enabled. This determines how the topic and schema are mapped to a subject name in the Schema Registry.

*Type:* string

*Default:* `TopicNameStrategy`

*Accepted values:* [`TopicNameStrategy`, `RecordNameStrategy`, `TopicRecordNameStrategy`]

**Related topics**:

- xref:manage:schema-reg/schema-id-validation.adoc[Server-Side Schema ID Validation]

---
[[redpandavalueschemavalidation]]
=== redpanda.value.schema.id.validation

Enable validation of the schema ID for values on a record. When enabled, Redpanda validates that the schema ID encoded in the record's value is registered in the Schema Registry according to the configured subject name strategy.

*Type:* boolean

*Default:* `false`

**Related topics**:

- xref:manage:schema-reg/schema-id-validation.adoc[Server-Side Schema ID Validation]

---
[[redpandavaluesubjectnamestrategy]]
=== redpanda.value.subject.name.strategy

The subject name strategy for values when `redpanda.value.schema.id.validation` is enabled. This determines how the topic and schema are mapped to a subject name in the Schema Registry.

*Type:* string

*Default:* `TopicNameStrategy`

*Accepted values:* [`TopicNameStrategy`, `RecordNameStrategy`, `TopicRecordNameStrategy`]

**Related topics**:

- xref:manage:schema-reg/schema-id-validation.adoc[Server-Side Schema ID Validation]

---
[[confluentkeyschemavalidation]]
=== confluent.key.schema.validation

Enable validation of the schema ID for keys on a record. This is a compatibility alias for `redpanda.key.schema.id.validation`. When enabled, Redpanda validates that the schema ID encoded in the record's key is registered in the Schema Registry according to the configured subject name strategy.

*Type:* boolean

*Default:* `false`

**Related topics**:

- xref:manage:schema-reg/schema-id-validation.adoc[Server-Side Schema ID Validation]

---
[[confluentkeysubjectnamestrategy]]
=== confluent.key.subject.name.strategy

The subject name strategy for keys when `confluent.key.schema.validation` is enabled. This is a compatibility alias for `redpanda.key.subject.name.strategy`. This determines how the topic and schema are mapped to a subject name in the Schema Registry.

*Type:* string

*Default:* `io.confluent.kafka.serializers.subject.TopicNameStrategy`

*Accepted values:* [`io.confluent.kafka.serializers.subject.TopicNameStrategy`, `io.confluent.kafka.serializers.subject.RecordNameStrategy`, `io.confluent.kafka.serializers.subject.TopicRecordNameStrategy`]

**Related topics**:

- xref:manage:schema-reg/schema-id-validation.adoc[Server-Side Schema ID Validation]

---
[[confluentvalueschemavalidation]]
=== confluent.value.schema.validation

Enable validation of the schema ID for values on a record. This is a compatibility alias for `redpanda.value.schema.id.validation`. When enabled, Redpanda validates that the schema ID encoded in the record's value is registered in the Schema Registry according to the configured subject name strategy.

*Type:* boolean

*Default:* `false`

**Related topics**:

- xref:manage:schema-reg/schema-id-validation.adoc[Server-Side Schema ID Validation]

---
[[confluentvaluesubjectnamestrategy]]
=== confluent.value.subject.name.strategy

The subject name strategy for values when `confluent.value.schema.validation` is enabled. This is a compatibility alias for `redpanda.value.subject.name.strategy`. This determines how the topic and schema are mapped to a subject name in the Schema Registry.

*Type:* string

*Default:* `io.confluent.kafka.serializers.subject.TopicNameStrategy`

*Accepted values:* [`io.confluent.kafka.serializers.subject.TopicNameStrategy`, `io.confluent.kafka.serializers.subject.RecordNameStrategy`, `io.confluent.kafka.serializers.subject.TopicRecordNameStrategy`]

**Related topics**:

- xref:manage:schema-reg/schema-id-validation.adoc[Server-Side Schema ID Validation]

---