-
Notifications
You must be signed in to change notification settings - Fork 468
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Control arrangement merging with default_exert_logic
#23029
Conversation
819841b
to
df45314
Compare
754b453
to
61c1df8
Compare
This PR has higher risk. Make sure to carefully review the file hotspots. In addition to having this change reviewed, adequate tests should be considered and it may be useful to add observability and/or a feature flag. What's This?
Buggy File Hotspots:
|
61c1df8
to
1189134
Compare
default_extert_logic
default_exert_logic
Signed-off-by: Moritz Hoffmann <[email protected]>
1189134
to
78d8b34
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The idea sounds fine, but I can't actually evaluate what this does without reading the internals of Differential
Signed-off-by: Moritz Hoffmann <[email protected]>
Add an alternative merging strategy for arrangements.
Up to this point, we could either not merge arrangements outside of ingesting updates, or always keep them in the most compacted form by specifying
idle_merge_effort
. Since TimelyDataflow/differential-dataflow#411 landed, Differential offers to replace the whole exertion logic, which we conditionally do with this PR. When the idle merge effort is 0, we honor an alternative setting of default_arrangement_exert_proprotionality that specifies which batches in an arrangement should be merged. Specifically, the proportionality determines when to merge the largest with the next largest batch.Motivation
Advanced MaterializeInc/database-issues#4918
Checklist
$T ⇔ Proto$T
mapping (possibly in a backwards-incompatible way), then it is tagged with aT-proto
label.