Skip to content

Add hybrid scan multifile single-step materializers - #22793

Merged
rapids-bot[bot] merged 34 commits into
NVIDIA:mainfrom
mhaseeb123:fea/column-chunk-byte-ranges
Jun 13, 2026
Merged

Add hybrid scan multifile single-step materializers#22793
rapids-bot[bot] merged 34 commits into
NVIDIA:mainfrom
mhaseeb123:fea/column-chunk-byte-ranges

Conversation

@mhaseeb123

@mhaseeb123 mhaseeb123 commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

Description

Contributes to #22583

This PR adds single-step column materialization APIs for the multifile hybrid scan reader

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

mhaseeb123 and others added 29 commits May 21, 2026 01:15
Co-authored-by: Yunsong Wang <12716979+PointKernel@users.noreply.github.com>
Co-authored-by: Yunsong Wang <12716979+PointKernel@users.noreply.github.com>
@mhaseeb123
mhaseeb123 requested a review from a team as a code owner June 5, 2026 00:48
@coderabbitai

coderabbitai Bot commented Jun 5, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

This PR extends the experimental hybrid_scan_multifile API by adding four public methods for row mask construction and column materialization from multi-file Parquet sources. Implementations delegate to the underlying implementation, while test utilities provide shared helpers for multifile setup, and new test cases validate row masking and column materialization behavior.

Changes

Hybrid Scan Multifile API Extension

Layer / File(s) Summary
Public API declarations and documentation
cpp/include/cudf/io/experimental/hybrid_scan_multifile.hpp, cpp/src/io/parquet/experimental/hybrid_scan_impl.hpp
Four new public methods declared: build_all_true_row_mask, build_row_mask_with_page_index_stats, all_column_chunks_byte_ranges, and materialize_all_columns. Doxygen references retargeted to multifile API variants.
Public API method implementations
cpp/src/io/parquet/experimental/hybrid_scan_multifile.cpp
Four methods implemented as forwarding wrappers that delegate row-group indices, options, and stream/memory parameters to underlying _impl and return results.
Test utilities and shared helpers
cpp/tests/io/experimental/hybrid_scan_common.hpp, cpp/tests/io/experimental/hybrid_scan_multifile_common.hpp, cpp/tests/io/experimental/hybrid_scan_test.cpp, cpp/tests/CMakeLists.txt
make_list_str_column helper added/moved to common header; new multifile_inputs struct and build_source_info/setup_page_indexes helpers introduced for test data management; duplicate helper removed from a test file; new test source registered in CMake.
Filter tests refactoring and row mask test suite
cpp/tests/io/experimental/hybrid_scan_multifile_filters_test.cpp
Existing tests refactored to use shared utilities; new host_row_mask_data and make_scalar helpers added; BuildAllTrueRowMask test validates mask properties; typed test suite HybridScanMultifilePageIndexRowMaskTest validates row mask filtering with page-index statistics across multiple filter expressions and error handling.
Column materialization test suite
cpp/tests/io/experimental/hybrid_scan_multifile_test.cpp
Adds tests for materialize_all_columns covering lists, list-of-strings, and struct column types; helper groups and flattens per-row-group byte ranges for device input.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

  • rapidsai/cudf#22613: Related multifile page-index fetching helpers used by page-index row-mask tests.
  • rapidsai/cudf#22616: Introduces base hybrid_scan_multifile infrastructure that this PR extends.

Suggested labels

feature request, 4 - Needs Review, non-breaking

Suggested reviewers

  • lamarrr
  • shrshi
  • qbacpey
  • PointKernel
  • bdice
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 7.14% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly and specifically describes the main change: adding single-step materializers for the multifile hybrid scan reader.
Description check ✅ Passed The pull request description is clearly related to the changeset, describing the addition of single-step column materialization APIs for the multifile hybrid scan reader.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands and usage tips.

@vuule
vuule self-requested a review June 6, 2026 01:43
@mhaseeb123 mhaseeb123 added feature request New feature or request 4 - Needs Review Waiting for reviewer to review or respond cuIO cuIO issue non-breaking Non-breaking change labels Jun 8, 2026
@Matt711 Matt711 removed their assignment Jun 9, 2026
@mhaseeb123 mhaseeb123 changed the title Add all column materialize API for multifile hybrid scan Add hybrid scan multifile single-step column materializers Jun 10, 2026
@mhaseeb123 mhaseeb123 changed the title Add hybrid scan multifile single-step column materializers Add hybrid scan multifile single-step materializers Jun 10, 2026
@mhaseeb123
mhaseeb123 requested a review from Matt711 June 10, 2026 22:09
@mhaseeb123 mhaseeb123 added 5 - Ready to Merge Testing and reviews complete, ready to merge and removed 4 - Needs Review Waiting for reviewer to review or respond labels Jun 12, 2026

@vyasr vyasr left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CMake approval (didn't look at the C++)

@mhaseeb123

Copy link
Copy Markdown
Contributor Author

/merge

@rapids-bot
rapids-bot Bot merged commit 3bcf033 into NVIDIA:main Jun 13, 2026
134 checks passed
@mhaseeb123
mhaseeb123 deleted the fea/column-chunk-byte-ranges branch June 13, 2026 00:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

5 - Ready to Merge Testing and reviews complete, ready to merge CMake CMake build issue cuIO cuIO issue feature request New feature or request libcudf Affects libcudf (C++/CUDA) code. non-breaking Non-breaking change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants