Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

storage: fix shutdown token for copy to s3 sink #30887

Closed
wants to merge 3 commits into from

Conversation

benesch
Copy link
Member

@benesch benesch commented Dec 22, 2024

The COPY TO S3 sink must not flush files to S3 during shutdown, as those
files may only be partially written.

This can cause correctness issues when running copy operations on
multiprocess replicas, as the lagging replica's copy sink will get
dropped partway through.

Motivation

  • This PR fixes a recognized bug.

Tips for reviewer

This is essentially just a copy of #30844, but I didn't want to lose my in progress CI run over there.

Checklist

  • This PR has adequate test coverage / QA involvement has been duly considered. (trigger-ci for additional test/nightly runs)
  • This PR has an associated up-to-date design doc, is a design doc (template), or is sufficiently small to not require a design.
  • If this PR evolves an existing $T ⇔ Proto$T mapping (possibly in a backwards-incompatible way), then it is tagged with a T-proto label.
  • If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label (example).
  • If this PR includes major user-facing behavior changes, I have pinged the relevant PM to schedule a changelog post.

The COPY TO S3 sink must not flush files to S3 during shutdown, as those
files may only be partially written.

This can cause correctness issues when running copy operations on
multiprocess replicas, as the lagging replica's copy sink will get
dropped partway through.
@benesch benesch requested review from petrosagg and teskje December 22, 2024 08:37
@benesch
Copy link
Member Author

benesch commented Dec 22, 2024

Closing in favor of #30844.

@benesch benesch closed this Dec 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant