Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

validate and adjust Substrait NamedTable schemas (#12223) #12245

Merged
merged 21 commits into from
Sep 10, 2024
Merged

validate and adjust Substrait NamedTable schemas (#12223) #12245

merged 21 commits into from
Sep 10, 2024

Commits on Aug 29, 2024

  1. fix: producer did not emit base_schema struct field for ReadRel 

    Substrait plans are not valid without this, and it is generally useful
for round trip testing
    @vbarua
    vbarua committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    429cd2b View commit details
    Browse the repository at this point in the history

  2. feat: include field_qualifier param for from_substrait_named_struct

    @vbarua
    vbarua committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    a44a689 View commit details
    Browse the repository at this point in the history

  3. feat: verify that Substrait and DataFusion agree on NamedScan schemas 

    If the schema registered with DataFusion and the schema as given by the
Substrait NamedScan do not have the same names and types, DataFusion
should reject it
    @vbarua
    vbarua committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    86a7339 View commit details
    Browse the repository at this point in the history

  4. test: update existing substrait test + substrait validation test 

    * added substrait_validation test
* extracted useful test utilities

The utils::test::TestSchemaCollector::generate_context_from_plan
function can be used to dynamically generate a SessionContext from a
Substrait plan, which will include the schemas for NamedTables as given
in the Substrait plan.

This helps us avoid the issue of DataFusion test schemas and Substrait
plan schemas not being in sync.
    @vbarua
    vbarua committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    1bfa887 View commit details
    Browse the repository at this point in the history

  5. feat: expose from_substrait_named_struct

    @vbarua
    vbarua committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    2f71997 View commit details
    Browse the repository at this point in the history

  6. refactor: remove unused imports

    @vbarua
    vbarua committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    8634d23 View commit details
    Browse the repository at this point in the history

Commits on Aug 30, 2024

  1. docs: add missing licenses

    @vbarua
    vbarua committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    5051991 View commit details
    Browse the repository at this point in the history

  2. refactor: deal with unused code warnings

    @vbarua
    vbarua committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    f52f01c View commit details
    Browse the repository at this point in the history

Commits on Sep 4, 2024

  1. remove optional qualifier from from_substrait_named_struct

    @vbarua
    vbarua committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    aa063c7 View commit details
    Browse the repository at this point in the history

  2. return DFSchema from from_substrait_named_struct

    @vbarua
    vbarua committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    f9319ba View commit details
    Browse the repository at this point in the history

  3. one must imagine clippy happy

    @vbarua
    vbarua committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    444aad6 View commit details
    Browse the repository at this point in the history

  4. accidental blah

    @vbarua
    vbarua committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    c2b0cb0 View commit details
    Browse the repository at this point in the history

  5. loosen the validation for schemas 

    allow cases where the Substrait schema is a subset of the DataFusion
schema
    @vbarua
    vbarua committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    0afb845 View commit details
    Browse the repository at this point in the history

  6. minor doc tweaks

    @vbarua
    vbarua committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    9ba5c4d View commit details
    Browse the repository at this point in the history

  7. update test data to deal with case issues in tests

    @vbarua
    vbarua committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    83529f5 View commit details
    Browse the repository at this point in the history

Commits on Sep 5, 2024

  1. fix error message

    @vbarua
    vbarua committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    acb2b47 View commit details
    Browse the repository at this point in the history

  2. improve readability of field compatability check

    @vbarua
    vbarua committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    386264e View commit details
    Browse the repository at this point in the history

  3. make TestSchemaCollector more flexible

    @vbarua
    vbarua committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    f6017eb View commit details
    Browse the repository at this point in the history

  4. fix doc typo 

    Co-authored-by: Arttu <[email protected]>
    @vbarua @Blizzara
    vbarua and Blizzara committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    185bff5 View commit details
    Browse the repository at this point in the history

  5. remove unecessary TODO

    @vbarua
    vbarua committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    aeb6c3d View commit details
    Browse the repository at this point in the history

Commits on Sep 6, 2024

  1. handle ReadRel projection on top of mismatched schema

    @vbarua
    vbarua committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    d571eb2 View commit details
    Browse the repository at this point in the history