-
Notifications
You must be signed in to change notification settings - Fork 468
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Correctly shutdown sources on Drop
#12082
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
guswynn
requested review from
antiguru,
frankmcsherry and
petrosagg
and removed request for
antiguru
April 26, 2022 21:44
frankmcsherry
approved these changes
Apr 26, 2022
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This generally looks good to me, except that I am confused why MapInPlaceWithSyncToken
is now a thing when it was previously just a map_in_place
one-liner.
1 task
benesch
added a commit
to benesch/materialize
that referenced
this pull request
Jun 1, 2022
Resolve a TODO to read the Debezium transactional metadata source out of persist, rather than re-rendering the source. This PR will unblock creating a pod per source (MaterializeInc#12770), but it is blocked on reverting (MaterializeInc#12082), which is no longer necessary now that TCP boundary has been removed.
benesch
added a commit
to benesch/materialize
that referenced
this pull request
Jun 1, 2022
Resolve a TODO to read the Debezium transactional metadata source out of persist, rather than re-rendering the source. This PR will unblock creating a pod per source (MaterializeInc#12770), but it is blocked on reverting (MaterializeInc#12082), which is no longer necessary now that TCP boundary has been removed.
1 task
benesch
added a commit
to benesch/materialize
that referenced
this pull request
Jun 1, 2022
PR MaterializeInc#12082 converted source tokens to thread-safe `Arc`s to be compatible with the TCP storage/compute boundary, but since MaterializeInc#12216 replaced the TCP boundary with persist we can go back to Rcs.
benesch
added a commit
to benesch/materialize
that referenced
this pull request
Jun 1, 2022
Resolve a TODO to read the Debezium transactional metadata source out of persist, rather than re-rendering the source. This PR will unblock creating a pod per source (MaterializeInc#12770), but it is blocked on reverting (MaterializeInc#12082), which is no longer necessary now that TCP boundary has been removed.
benesch
added a commit
to benesch/materialize
that referenced
this pull request
Jun 1, 2022
PR MaterializeInc#12082 converted source tokens to thread-safe `Arc`s to be compatible with the TCP storage/compute boundary, but since MaterializeInc#12216 replaced the TCP boundary with persist we can go back to Rcs.
1 task
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Before this pr, sources are timely-worker-local objects, and they return what amounts
Rc
's orRc<RefCell>
's of thread-local capabilities. The intent is that if these capabilities are dropped, then the source operators are correctly shutdown. However, now that all sources end withEventPusher
's that consolidate all messages into a single tcp boundary. This boundary holds onto a thread-safe token, which theEventPusher
's notice is dropped and drop their worker-local tokens. However, there is not guarantee that new data or progress, in either theok
orerr
streams of a source will happen, which means the trigger for dropping those tokens may never run.Instead, @frankmcsherry and I decided that we should just upgrade the original source-tokens to be thread-safe. The way we do this is very consistent across multiple locations:
SourceToken
, which is a share-ableSyncActivator
that activates on Drop.Arc
'sWeak
around in the operator. Whenupgrading
is impossible, that means the token is dropped, and respond by dropping the local capabilities.The places we do this are:
source
operator used by sources inmz_storage::source::util
import_table
inmz_storage::render::sources
map_in_place
to its implementation with some complexity addedmz_storage::render
. I have no idea how to test this one (cc @antiguru @frankmcsherry )Perf implications
This pr means each source timely step will have at least 2 added atomic
CAS
's. If this becomes a problem, then we can switch to an atomic flag as opposed to a fullWeak::upgrade
, which may save us some perf.Additionally: we may be able to switch back to
Rc
's someday, if we move off oftcp_boundary
andpersist
-per-worker instead.Motivation
Sources are not correctly dropped
Tips for reviewer
TimelyDataflow/differential-dataflow#364 is a smaller, more readable change that gives an example of the exact same kind of change we are doing in multiple places here (and in fact, that change is required for the
decode_cdcv2
token to work in this pr.Also, this pr has some code that is very similar copied to many places. While we may avoid bugs by consolidating it into a function, its unclear if there is a clean way to do so, and it may make the code more unreadable.
Testing
Release notes
None