Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix][broker] Fix compaction/replication data loss when expire messages #21865

Merged
merged 2 commits into from
Jan 11, 2024

Conversation

coderzc
Copy link
Member

@coderzc coderzc commented Jan 8, 2024

Motivation

When expiring messages during compaction, we may lose the middle message due to __compaction cursor mark-delete position being advanced. So I think we should skip TTL check for __compaction.

replicators also have a similar problem, so also skip the expiration check of replicator

Modifications

  • Skip TTL check for __compaction and skip expire __compaction subscriptions.
  • Passing lost cursor.properties when the mark deletes for the cursor.
  • Skip TTL check for replicator, and remove the expired message replicator.

Verifying this change

  • Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

  • Added integration tests for end-to-end deployment with large payloads (10MB)
  • Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

  • Dependencies (add or upgrade a dependency)
  • The public API
  • The schema
  • The default values of configurations
  • The threading model
  • The binary protocol
  • The REST endpoints
  • The admin CLI options
  • The metrics
  • Anything that affects deployment

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

Matching PR in forked repository

PR in forked repository:

@github-actions github-actions bot added the doc-not-needed Your PR changes do not impact docs label Jan 8, 2024
@coderzc coderzc changed the title [fix][broker] Fix compaction data loss when expire messages [fix][broker] Fix compaction/replication data loss when expire messages Jan 9, 2024
@coderzc coderzc force-pushed the fix_ttl_compaction branch 3 times, most recently from af3ffcd to 9b9fbd7 Compare January 9, 2024 03:13
@coderzc coderzc force-pushed the fix_ttl_compaction branch from 9b9fbd7 to 42790cb Compare January 9, 2024 03:14
@coderzc coderzc added this to the 3.3.0 milestone Jan 9, 2024
@coderzc coderzc added type/bug The PR fixed a bug or issue reported a bug area/broker ready-to-test labels Jan 9, 2024
@coderzc coderzc closed this Jan 9, 2024
@coderzc coderzc reopened this Jan 9, 2024
@merlimat merlimat merged commit 0e722d0 into apache:master Jan 11, 2024
67 of 70 checks passed
@coderzc coderzc self-assigned this Jan 11, 2024
mukesh-ctds pushed a commit to datastax/pulsar that referenced this pull request Mar 1, 2024
mukesh-ctds pushed a commit to datastax/pulsar that referenced this pull request Mar 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants