[Rust] [DataFusion] Implement projection push down query optimizer rule

[asfimport]

If I run a query like the following:

SELECT MIN(fare_amount), MAX(fare_amount) FROM tripdata

I see this logical plan:

Logical plan: Aggregate: groupBy=[[]], aggr=[[MIN(#10), MAX(#10)]]
  TableScan: tripdata projection=None

 

This means that every column is being loaded into arrays rather than just the two columns that I care about, resulting in terrible performance.

Reporter: Andy Grove / @andygrove
Assignee: Andy Grove / @andygrove

PRs and other links:

• GitHub Pull Request #3664

_{Note: This issue was originally created as ARROW-4589. Please see the migration documentation for further details.}

[asfimport]

Andy Grove / @andygrove:
Merged. The merge script failed with:

Pull request #3664 merged!
Merge hash: 3606aa21

Would you like to update the associated JIRA? (y/n): y
Enter comma-separated fix version(s) [0.13.0]: y
Traceback (most recent call last):
  File "./dev/merge_arrow_pr.py", line 455, in <module>
    cli()
  File "./dev/merge_arrow_pr.py", line 449, in cli
    fix_versions_json = [get_version_json(v) for v in issue_fix_versions]
  File "./dev/merge_arrow_pr.py", line 447, in get_version_json
    return [x for x in versions if x.name == version_str][0].raw
IndexError: list index out of range