Skip to content

(2.12) Atomic batch: support deduplication#7391

Merged
neilalexander merged 1 commit intomainfrom
maurice/batch-duplicates
Oct 2, 2025
Merged

(2.12) Atomic batch: support deduplication#7391
neilalexander merged 1 commit intomainfrom
maurice/batch-duplicates

Conversation

@MauriceVanVeen
Copy link
Copy Markdown
Member

2.12.0 released atomic batch publish, but didn't contain deduplication support yet. Now that we've decided to have both atomic batch publishing (atomic & consistent) and non-atomic high-speed publishing (performance), we can make a decision around how deduplication works as well.

When the Nats-Msg-Id deduplication header is used and a duplicate message is detected, the whole batch gets rejected with an error: atomic publish batch contains duplicate message id. This ensures the all-or-nothing-gets-persisted atomic nature is preserved, also when deduplication is used. There are no exceptions.

The high-speed non-atomic batching could then support deduplication differently by allowing them to be omitted from the stored messages, similar to how js.PublishAsync would work.

ADR-50 update: nats-io/nats-architecture-and-design#385

Resolves #6980

Signed-off-by: Maurice van Veen github@mauricevanveen.com

Signed-off-by: Maurice van Veen <github@mauricevanveen.com>
@MauriceVanVeen MauriceVanVeen requested review from a team as code owners October 2, 2025 14:00
Copy link
Copy Markdown
Member

@neilalexander neilalexander left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@neilalexander neilalexander merged commit 7726cba into main Oct 2, 2025
90 of 92 checks passed
@neilalexander neilalexander deleted the maurice/batch-duplicates branch October 2, 2025 15:43
neilalexander added a commit that referenced this pull request Oct 3, 2025
Includes the following:

- #7374
- #7373
- #7377
- #7380
- #7382
- #7381
- #7364
- #7384
- #7385
- #7388
- #7386
- #7391
- #7242

Signed-off-by: Neil Twigg <neil@nats.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Batch publish - optimize deduplication

2 participants