Skip to content

[Converter] Add assertion for correctness; Add aggregate stats and additional logging#545

Merged
Zyiqin-Miranda merged 2 commits intoray-project:2.0from
Zyiqin-Miranda:adding-assertion
May 10, 2025
Merged

[Converter] Add assertion for correctness; Add aggregate stats and additional logging#545
Zyiqin-Miranda merged 2 commits intoray-project:2.0from
Zyiqin-Miranda:adding-assertion

Conversation

@Zyiqin-Miranda
Copy link
Member

@Zyiqin-Miranda Zyiqin-Miranda commented May 6, 2025

Summary

This PR mainly adds:

  1. Assertion around lower-level Pyarrow function like concatenate arrays; Daft string casting to allow us to detect potential issues earlier on.
  2. Aggregate stats mainly for input/output record counting, in-memory/on-disk sizes.

Note file changes contain previous stacked PR #542, will rebase once #542 is merged. (Rebased)

Rationale

Explain the reasoning behind the changes and their benefits to the project.

Changes

List the major changes made in this pull request.

Impact

Discuss any potential impacts the changes may have on existing functionalities.

Testing

Describe how the changes have been tested, including both automated and manual testing strategies.
If this is a bugfix, explain how the fix has been tested to ensure the bug is resolved without introducing new issues.

Regression Risk

If this is a bugfix, assess the risk of regression caused by this fix and steps taken to mitigate it.

Checklist

  • Unit tests covering the changes have been added

    • If this is a bugfix, regression tests have been added
  • E2E testing has been performed

Additional Notes

Any additional information or context relevant to this PR.

Copy link
Member

@pdames pdames left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@Zyiqin-Miranda Zyiqin-Miranda merged commit 218a83f into ray-project:2.0 May 10, 2025
6 checks passed
@Zyiqin-Miranda Zyiqin-Miranda deleted the adding-assertion branch May 10, 2025 01:12
025rhu pushed a commit to nsaadhvi/codebase-deltacat that referenced this pull request Jul 16, 2025
…ditional logging (ray-project#545)

* Rebase changes

* [Converter] Add assertion for correctness; Add aggregate stats and additional logging; Code clean-up

---------

Co-authored-by: Miranda <yiqin121@gmail.com>
025rhu pushed a commit to nsaadhvi/codebase-deltacat that referenced this pull request Jul 16, 2025
…ditional logging (ray-project#545)

* Rebase changes

* [Converter] Add assertion for correctness; Add aggregate stats and additional logging; Code clean-up

---------

Co-authored-by: Miranda <yiqin121@gmail.com>
rnapark pushed a commit to rnapark/deltacat that referenced this pull request Aug 17, 2025
…ditional logging (ray-project#545)

* Rebase changes

* [Converter] Add assertion for correctness; Add aggregate stats and additional logging; Code clean-up

---------

Co-authored-by: Miranda <yiqin121@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants