Experimental Parquet reader supports retention vectors by mhaseeb123 · Pull Request #23402 · NVIDIA/cudf

mhaseeb123 · 2026-07-22T18:26:35Z

Description

This PR enables the experimental parquet reader to now also support retention vectors in addition to deletion vectors

Checklist

I am familiar with the Contributing Guidelines.
New or existing tests cover these changes.
The documentation is up to date with these changes.

copy-pr-bot · 2026-07-22T18:26:38Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

copy-pr-bot · 2026-07-22T18:34:59Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

coderabbitai · 2026-07-22T21:47:32Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 405fb8ef-a94e-4fdb-967b-2a3016c2ecb3

📥 Commits

Reviewing files that changed from the base of the PR and between e305f18 and 00c95c2.

📒 Files selected for processing (2)

cpp/src/io/parquet/experimental/deletion_vectors.cu
cpp/tests/io/parquet_deletion_vectors_test.cpp

🚧 Files skipped from review as they are similar to previous changes (2)

cpp/src/io/parquet/experimental/deletion_vectors.cu
cpp/tests/io/parquet_deletion_vectors_test.cpp

📝 Walkthrough

Summary by CodeRabbit

New Features
- Added support for retention vectors when reading Parquet data.
- Retention vectors are now handled consistently for both standard and chunked reads.
- Added retention-vector support for custom row-index columns and concatenated tables.
Bug Fixes
- Corrected row mask semantics so retention and deletion vectors are applied with the proper retained/deleted interpretation.
Tests
- Expanded Parquet deletion/retention vector coverage, including multi-concat, custom-index, and row-count verification.

Walkthrough

Changes

Retention Vector Support

Layer / File(s)	Summary
Vector type contract `cpp/include/cudf/io/experimental/deletion_vectors.hpp`	Adds retention-vector metadata to `deletion_vector_info` and stores the interpretation for chunked readers.
Mask application and row counting `cpp/src/io/parquet/experimental/deletion_vectors.cu`	Selects `RETENTION` or `DELETION` masking for regular and chunked parquet reads and adjusts deleted-row counts for retention vectors.
Retention-aware validation `cpp/tests/io/parquet_deletion_vectors_test.cpp`	Updates bitmap helpers and tests to validate deletion and retention scenarios across row-index, concatenation, and counting paths.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Suggested labels: cuIO

Suggested reviewers: simoneves, jihoonson, bdice, vuule

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the main change: adding retention-vector support to the experimental Parquet reader.
Description check	✅ Passed	The description matches the changeset and states that the experimental Parquet reader now supports retention vectors with tests updated.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@cpp/include/cudf/io/experimental/deletion_vectors.hpp`:
- Around line 51-53: Update compute_num_deleted_rows to honor
are_retention_vectors: when retention mode is enabled, invert the bitmap count
so the API returns deleted-row count rather than retained-row count. Preserve
the existing count behavior for non-retention vectors.

In `@cpp/tests/io/parquet_deletion_vectors_test.cpp`:
- Around line 303-305: Update the concatenated bitmap generation in
build_roaring_bitmap_and_expected_row_mask to forward
final_deletion_vector_info.are_retention_vectors to each generated bitmap,
preserving retention semantics when num_concat is greater than one.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: a99b0488-dd73-44eb-90cf-2834e00db6a2

📥 Commits

Reviewing files that changed from the base of the PR and between b6b58b3 and e305f18.

📒 Files selected for processing (3)

cpp/include/cudf/io/experimental/deletion_vectors.hpp
cpp/src/io/parquet/experimental/deletion_vectors.cu
cpp/tests/io/parquet_deletion_vectors_test.cpp

jihoonson

LGTM. Thanks @mhaseeb123

vuule · 2026-07-23T00:26:53Z

/merge

) First of all, I apologize for making a PR targeting the 26.08 branch. I understand the code freeze has taken effect since yesterday, but wish this change can still be shipped in the 26.08 release. ## Description This PR adds the JNI bridge for the recent change in #23402 to support the retention filter for the deletion vector. Previously, it was assumed that deletion vectors represent only the rows deleted. With #23402 and this change, now Java applications can pass the correct filter type for deletion vectors. ## Checklist - [x] I am familiar with the [Contributing Guidelines](https://github.com/rapidsai/cudf/blob/HEAD/CONTRIBUTING.md). - [x] New or existing tests cover these changes. - [x] The documentation is up to date with these changes.

…ors for OSS delta [databricks] (#15368) Fixes #15326. ### Description The CDF read with deletion vectors currently fails. Two things were missing to support this case: - The `IF_NOT_CONTAINED` row index filter type support. The Delta CDC reader can use this type of row index filter. - Inline deletion vector support. The CDC reader creates inline deletion vectors. This PR adds those supports based on NVIDIA/cudf#23402 for OSS Delta. The plugin now can load inline deletion vectors and process the `IF_NOT_CONTAINED` filter properly with all 3 Delta readers. Note that the issue exists only with the native readers (`GpuDeltaParquetFileFormatBase2`). The legacy reader (`GpuDeltaParquetFileFormatBase`) does not have this issue. Databricks readers have the same issue, and will be fixed in #15365. ### Checklists Documentation - [ ] Updated for new or modified user-facing features or behaviors - [x] No user-facing change Testing - [x] Added or modified tests to cover new code paths - [ ] Covered by existing tests (Please provide the names of the existing tests in the PR description.) - [ ] Not required Performance - [ ] Tests ran and results are added in the PR description - [ ] Issue filed with a link in the PR description - [x] Not required --------- Signed-off-by: Jihoon Son <ghoonson@gmail.com>

Support retention vectors in parquet experimental reader

7c2e007

github-actions Bot assigned mhaseeb123 Jul 22, 2026

github-actions Bot added libcudf Affects libcudf (C++/CUDA) code. CMake CMake build issue Java Affects Java cuDF API. labels Jul 22, 2026

mhaseeb123 added this to libcudf Jul 22, 2026

mhaseeb123 moved this to Burndown in libcudf Jul 22, 2026

mhaseeb123 changed the base branch from main to release/26.08 July 22, 2026 18:27

mhaseeb123 added Spark Functionality that helps Spark RAPIDS 4 - Needs Review Waiting for reviewer to review or respond and removed Java Affects Java cuDF API. labels Jul 22, 2026

Minor bug fix

f6231f6

Add gtests

99d1436

mhaseeb123 added feature request New feature or request non-breaking Non-breaking change labels Jul 22, 2026

Copyright header fix

e305f18

mhaseeb123 marked this pull request as ready for review July 22, 2026 21:41

mhaseeb123 requested a review from a team as a code owner July 22, 2026 21:41

mhaseeb123 requested review from bdice, jihoonson, simoneves and vuule July 22, 2026 21:41

coderabbitai Bot reviewed Jul 22, 2026

View reviewed changes

Comment thread cpp/include/cudf/io/experimental/deletion_vectors.hpp

Comment thread cpp/tests/io/parquet_deletion_vectors_test.cpp Outdated

Bug fixing

00c95c2

vuule approved these changes Jul 22, 2026

View reviewed changes

jihoonson approved these changes Jul 22, 2026

View reviewed changes

rapids-bot Bot merged commit c16794f into NVIDIA:release/26.08 Jul 23, 2026
260 of 261 checks passed

mhaseeb123 deleted the fea/support-retention-vectors branch July 23, 2026 01:31

GregoryKimball moved this from Burndown to Landed in libcudf Jul 23, 2026

This was referenced Jul 23, 2026

Support IF_NOT_CONTAINED filter type and loading inline deletion vectors for OSS delta [databricks] NVIDIA/cudf-spark#15368

Merged

JNI bridge for the retention filter support for deletion vectors #23423

Merged

coderabbitai Bot mentioned this pull request Jul 27, 2026

Forward-merge release/26.08 into main #23401

Merged

3 tasks

josephine-wolf-oberholtzer moved this to Landed in libcudf Aug 12, 2026

josephine-wolf-oberholtzer added this to libcudf Aug 12, 2026

GregoryKimball removed this from libcudf Aug 21, 2026

Navigation Menu

Uh oh!

Experimental Parquet reader supports retention vectors - #23402

Experimental Parquet reader supports retention vectors#23402
rapids-bot[bot] merged 5 commits into
NVIDIA:release/26.08from
mhaseeb123:fea/support-retention-vectors

mhaseeb123 commented Jul 22, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Jul 22, 2026

Uh oh!

copy-pr-bot Bot commented Jul 22, 2026

Uh oh!

coderabbitai Bot commented Jul 22, 2026 •

edited

Loading

Summary by CodeRabbit

Walkthrough

Changes

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

Uh oh!

jihoonson left a comment

Uh oh!

vuule commented Jul 23, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

mhaseeb123 commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Uh oh!

copy-pr-bot Bot commented Jul 22, 2026

Uh oh!

copy-pr-bot Bot commented Jul 22, 2026

Uh oh!

coderabbitai Bot commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Walkthrough

Changes

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jihoonson left a comment

Choose a reason for hiding this comment

Uh oh!

vuule commented Jul 23, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

mhaseeb123 commented Jul 22, 2026 •

edited

Loading

coderabbitai Bot commented Jul 22, 2026 •

edited

Loading