Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support Substrait's VirtualTables #10531

Merged
merged 10 commits into from
May 28, 2024
Merged

Support Substrait's VirtualTables #10531

merged 10 commits into from
May 28, 2024

Commits on May 24, 2024

  1. Add support for Substrait VirtualTables 

    Adds support for Substrait's VirtualTables, ie. tables with data baked-in into the Substrait plan instead of being read from a source.

Adds conversion in both ways (Substrait -> DataFusion and DataFusion -> Substrait)
and a roundtrip test.
    @Blizzara
    Blizzara committed May 24, 2024
    Configuration menu
    Copy the full SHA
    02e7bc0 View commit details
    Browse the repository at this point in the history

  2. fix clippy

    @Blizzara
    Blizzara committed May 24, 2024
    Configuration menu
    Copy the full SHA
    b5b4430 View commit details
    Browse the repository at this point in the history

  3. Add support for empty relations

    @Blizzara
    Blizzara committed May 24, 2024
    Configuration menu
    Copy the full SHA
    3f40ecf View commit details
    Browse the repository at this point in the history

Commits on May 27, 2024

  1. Fix consuming Structs inside Lists and Structs 

    Also adds roundtrip schema assertions for cases where possible
    @Blizzara
    Blizzara committed May 27, 2024
    Configuration menu
    Copy the full SHA
    24d07c9 View commit details
    Browse the repository at this point in the history

  2. Rename from_substrait_struct -> from_substrait_struct_type for clarity

    @Blizzara
    Blizzara committed May 27, 2024
    Configuration menu
    Copy the full SHA
    e40f721 View commit details
    Browse the repository at this point in the history

  3. Add DataType::LargeList to to_substrait_named_struct

    @Blizzara
    Blizzara committed May 27, 2024
    Configuration menu
    Copy the full SHA
    9c83ba4 View commit details
    Browse the repository at this point in the history

  4. Merge remote-tracking branch 'upstream/main' into avo/substrait-virtu… 

    …al-tables
    @Blizzara
    Blizzara committed May 27, 2024
    Configuration menu
    Copy the full SHA
    6b580ce View commit details
    Browse the repository at this point in the history

  5. cargo fmt --all

    @Blizzara
    Blizzara committed May 27, 2024
    Configuration menu
    Copy the full SHA
    0680b4f View commit details
    Browse the repository at this point in the history

  6. Add validation that names list matches schema exactly

    @Blizzara
    Blizzara committed May 27, 2024
    Configuration menu
    Copy the full SHA
    d7821bc View commit details
    Browse the repository at this point in the history

  7. Add a LargeList into VALUES test

    @Blizzara
    Blizzara committed May 27, 2024
    Configuration menu
    Copy the full SHA
    4b8babe View commit details
    Browse the repository at this point in the history