Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bin-pack Writes Operation into multiple parquet files, and parallelize writing WriteTasks #444

Merged
merged 15 commits into from
Mar 28, 2024
Merged

Bin-pack Writes Operation into multiple parquet files, and parallelize writing WriteTasks #444

merged 15 commits into from
Mar 28, 2024

Commits on Mar 9, 2024

  1. bin pack write

    @kevinjqliu
    kevinjqliu committed Mar 9, 2024
    Configuration menu
    Copy the full SHA
    5a7c8f9 View commit details
    Browse the repository at this point in the history

  2. add write target file size config

    @kevinjqliu
    kevinjqliu committed Mar 9, 2024
    Configuration menu
    Copy the full SHA
    bae32e2 View commit details
    Browse the repository at this point in the history

  3. test

    @kevinjqliu
    kevinjqliu committed Mar 9, 2024
    Configuration menu
    Copy the full SHA
    2730a8f View commit details
    Browse the repository at this point in the history

  4. add test for multiple data files

    @kevinjqliu
    kevinjqliu committed Mar 9, 2024
    Configuration menu
    Copy the full SHA
    ef64c92 View commit details
    Browse the repository at this point in the history

  5. parquet writer write once

    @kevinjqliu
    kevinjqliu committed Mar 9, 2024
    Configuration menu
    Copy the full SHA
    9cb9649 View commit details
    Browse the repository at this point in the history

  6. parallelize write tasks

    @kevinjqliu
    kevinjqliu committed Mar 9, 2024
    Configuration menu
    Copy the full SHA
    3f284b2 View commit details
    Browse the repository at this point in the history

  7. refactor

    @kevinjqliu
    kevinjqliu committed Mar 9, 2024
    Configuration menu
    Copy the full SHA
    6462d06 View commit details
    Browse the repository at this point in the history

  8. chunk correctly using to_batches

    @kevinjqliu
    kevinjqliu committed Mar 9, 2024
    Configuration menu
    Copy the full SHA
    fd1efe0 View commit details
    Browse the repository at this point in the history

  9. change variable names

    @kevinjqliu
    kevinjqliu committed Mar 9, 2024
    Configuration menu
    Copy the full SHA
    7ccfdb2 View commit details
    Browse the repository at this point in the history

  10. get rid of assert

    @kevinjqliu
    kevinjqliu committed Mar 9, 2024
    Configuration menu
    Copy the full SHA
    1ee3a55 View commit details
    Browse the repository at this point in the history

  11. configure PackingIterator

    @kevinjqliu
    kevinjqliu committed Mar 9, 2024
    Configuration menu
    Copy the full SHA
    f92de1a View commit details
    Browse the repository at this point in the history

  12. add more tests

    @kevinjqliu
    kevinjqliu committed Mar 9, 2024
    Configuration menu
    Copy the full SHA
    0047fd8 View commit details
    Browse the repository at this point in the history

  13. rewrite set_properties

    @kevinjqliu
    kevinjqliu committed Mar 9, 2024
    Configuration menu
    Copy the full SHA
    c6cb8de View commit details
    Browse the repository at this point in the history

  14. set int property

    @kevinjqliu
    kevinjqliu committed Mar 9, 2024
    Configuration menu
    Copy the full SHA
    d80054d View commit details
    Browse the repository at this point in the history

Commits on Mar 28, 2024

  1. Merge branch 'main' into kevinjqliu/bin-pack-write

    @kevinjqliu
    kevinjqliu committed Mar 28, 2024
    Configuration menu
    Copy the full SHA
    8cd7160 View commit details
    Browse the repository at this point in the history