[native] Derive TableScan stream type as FIXED#24468
[native] Derive TableScan stream type as FIXED#24468arhimondr merged 1 commit intoprestodb:masterfrom
Conversation
|
@arhimondr has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
Is there a reason why only native execution creates fixed number of drivers? |
|
In Presto (Java) the number of TableScan drivers created was chosen adaptively. That added a lot of complexity to TaskExecutor and the benefit was not clear. In Velox the number of TableScan drivers is fixed. The adaptivity can still be achieved via delivering splits to only a subset of TableScan drivers without going into complexity of managing the number of drivers dynamically. Here's an example where @xiaoxmeng adds TableScan scaling: facebookincubator/velox#11879 |
320464c to
f45a1eb
Compare
|
Ready for review |
aditi-pandit
left a comment
There was a problem hiding this comment.
Thanks @arhimondr for this PR and the context.
In native a fixed number of drivers is created for a pipeline containing TableScan
f45a1eb to
0ba7548
Compare
|
@arhimondr has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
Description
In native a fixed number of drivers is created for a pipeline containing TableScan
Motivation and Context
To avoid unnecessary local exchange when fixed number of streams is necessary (for example for TableWriteOperator)
Impact
Improved performance for certain type of INSERT INTO queries
Test Plan
CI, Unit Tests
Contributor checklist
Release Notes
Please follow release notes guidelines and fill in the release notes below.