Prepend source index column in Parquet reader by mhaseeb123 · Pull Request #22879 · NVIDIA/cudf

mhaseeb123 · 2026-06-12T23:14:40Z

Description

Contributes to #22849

This PR enables optionally prepending a source index column to the read table.

Checklist

I am familiar with the Contributing Guidelines.
New or existing tests cover these changes.
The documentation is up to date with these changes.

copy-pr-bot · 2026-06-12T23:14:43Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

mhaseeb123 · 2026-06-12T23:22:19Z

  std::optional<int32_t> type_length;      ///< Byte width of data (for fixed length data)
  std::vector<column_name_info> children;  ///< Child column names

-  /**


No need for explicit constructor or a == operator. Use designated initializers and default == operator works.

mhaseeb123 · 2026-06-12T23:23:13Z

  return {filtered_row_group_indices};
 }

+offset_column_references::offset_column_references(


Very simple transformer with lots of boilerplate. Simply offsets the column_references in the input tree by a specified number. That's it, it changes nothing else.

mhaseeb123 · 2026-06-12T23:23:44Z

-      out_metadata.num_rows_per_source =
-        std::vector<size_t>(_file_itm_data.num_rows_per_source.size(), 0);
-    }
+    // Empty dataframe case: Simply initialize to a list of zeros


Always write number of rows read from each source. Even if it's zero.

mhaseeb123 · 2026-06-12T23:23:59Z

-      // Move is okay here as we are reading in one go.
-      out_metadata.num_rows_per_source = std::move(_file_itm_data.num_rows_per_source);
-    }
+  // Compute the output number of rows per source


Same. Always write number of rows per source

mhaseeb123 · 2026-06-12T23:24:26Z

  _file_itm_data._output_chunk_count++;

+  // Prepend the source index column if requested
+  if (_options.prepend_source_index_column) {


Build and prepend source index column using the rows read per source information which we now unconditionally write.

mhaseeb123 · 2026-06-12T23:25:19Z

    if (_num_filter_only_columns > 0) { out_metadata.schema_info.resize(output_count); }

+    // Clear the number of rows per source as it is not valid after filtering
+    out_metadata.num_rows_per_source.clear();


Clear rows per source information (which was previously guarded under the same condition) as applying a filter modifies it. Note that the prepended column also gets filter so it still remains valid.

mhaseeb123 · 2026-06-12T23:26:27Z

+  }
+
+  // Offset column references in `_expr_conv` by the number of prepended columns
+  auto const num_prepended_cols = static_cast<size_type>(_options.prepend_source_index_column);


We must now update the column references in our filter expression by the number of prepended columns so the filter expression works on correct data columns.

mhaseeb123 · 2026-06-12T23:26:35Z

    return _file_itm_data._output_chunk_count == 0;
  }

-  /**


No need for this anymore

mhaseeb123 · 2026-06-12T23:26:51Z

                     index_names.cend(),
                     std::back_inserter(selected_columns),
-                     [](std::string const& name) { return column_name_info(name); });
+                     [](std::string const& name) { return column_name_info{.name = name}; });


Use designated initializer

mhaseeb123 · 2026-06-12T23:28:24Z

+  auto col_data =
+    cudf::detail::make_zeroed_device_uvector_async<column_type>(num_rows, _stream, _mr);
+
+  // Multiple sources


Reused logic from multiple places. Start with a zeroed column, scatter (increasing) source indices to the column locations where they start contributing. Do an inclusive_scan with cuda::maximum to update those zeros to the last source index.

coderabbitai · 2026-06-13T00:19:03Z

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

@coderabbitai resume to resume automatic reviews.
@coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

▶️ Resume reviews
🔍 Trigger review

📝 Walkthrough

Walkthrough

This PR adds a source index column prepending feature to the Parquet reader, allowing users to track which input file each output row originated from. It introduces a new configuration option, implements GPU-accelerated column generation for single and multi-source cases, and integrates filter expression offset handling to maintain correctness during filtering.

Changes

Prepend Source Index Column Feature

Layer / File(s)	Summary
Public API and Configuration `cpp/include/cudf/io/parquet.hpp`	New `parquet_reader_options` option flag `_prepend_source_index_column` with public getter `is_enabled_prepend_source_index_column()`, setter `enable_prepend_source_index_column(bool)`, and builder-chain method `prepend_source_index_column(bool)` to control source index column prepending.
Expression Transformer for Adjusted Column References `cpp/src/io/parquet/expression_transform_helpers.hpp`, `cpp/src/io/parquet/expression_transform_helpers.cpp`	New `offset_column_references` AST visitor class that rewrites filter expressions by offsetting column reference indices to account for the prepended source index column; handles literals unchanged, shifts column references, rejects column name references, and rebuilds operations recursively.
Reader Implementation Configuration `cpp/src/io/parquet/reader_impl.hpp`	Adds reader config field `prepend_source_index_column` (default `false`) and declares new `prepend_source_index_column(...)` method to construct and insert the source index column.
GPU Source Index Column Construction `cpp/src/io/parquet/reader_impl_preprocess.cu`	Implements `prepend_source_index_column(...)` helper with specialized GPU paths: empty output (empty column), single source (constant zeros), and multiple sources (scatter source indices into first row per source, then inclusive-scan fill remaining rows). Updates `preprocess_file` to run exclusive-sum scan for all `CHUNKED_READ` cases instead of only when filter expressions are present.
Reader Flow and Finalization `cpp/src/io/parquet/reader_impl.cpp`	Initializes reader option, updates empty-table metadata initialization, and extends `finalize_output` to prepend source index column when enabled, rewrite filter expressions via `offset_column_references` to adjust column indices, and clear source metadata after filtering.
Supporting Changes `cpp/include/cudf/io/types.hpp`, `cpp/src/io/parquet/reader_impl_helpers.cpp`	Defaults `column_name_info::operator==` to compiler-generated equality, updates copyright year to 2026, and converts `column_name_info` construction to designated initialization.
Test Coverage and Validation `cpp/tests/io/parquet_chunked_reader_test.cu`, `cpp/tests/io/parquet_reader_test.cpp`	Updated chunked-reader tests to validate source index column across single and multiple sources with filtering; new comprehensive test cases `SourceIndexColumn` and `SourceIndexSelectedRead` verify correct source indices with full reads, filtered reads, row bounds, and explicit row group selection.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Suggested labels

tests

Suggested reviewers

vuule
bdice
davidwendt
Matt711

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 25.81% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The pull request title 'Prepend source index column in Parquet reader' clearly and concisely summarizes the main change: adding functionality to prepend a source index column to the Parquet reader output.
Description check	✅ Passed	The description is directly related to the changeset, explaining that the PR enables optionally prepending a source index column to the read table and referencing the associated issue `#22849`.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

cpp/src/io/parquet/reader_impl_preprocess.cu (1)
1130-1130: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Integer overflow in std::accumulate with mismatched types.

Using 0 (int literal) as the initial value causes std::accumulate to use int as the accumulator type. Since num_rows_per_source contains std::size_t values, summing them into an int accumulator causes signed integer overflow (undefined behavior) if the total exceeds INT_MAX.
Suggested fix
-  auto const num_rows = std::accumulate(num_rows_per_source.begin(), num_rows_per_source.end(), 0);
+  auto const num_rows = std::accumulate(num_rows_per_source.begin(), num_rows_per_source.end(), std::size_t{0});
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@cpp/src/io/parquet/reader_impl_preprocess.cu` at line 1130, The
std::accumulate call on num_rows_per_source uses an int literal 0 as the initial
value, which causes the accumulator to be typed as int rather than matching the
std::size_t type of the elements in num_rows_per_source. This creates a signed
integer overflow vulnerability when summing values. Replace the initial value 0
with a std::size_t typed literal (such as 0UL or std::size_t(0)) to ensure the
accumulator uses the same type as the elements being summed, preventing the
integer overflow.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@cpp/src/io/parquet/reader_impl_preprocess.cu`:
- Line 1130: The std::accumulate call on num_rows_per_source uses an int literal
0 as the initial value, which causes the accumulator to be typed as int rather
than matching the std::size_t type of the elements in num_rows_per_source. This
creates a signed integer overflow vulnerability when summing values. Replace the
initial value 0 with a std::size_t typed literal (such as 0UL or std::size_t(0))
to ensure the accumulator uses the same type as the elements being summed,
preventing the integer overflow.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: f225f573-1bf6-4060-980b-ba3dd8973761

📥 Commits

Reviewing files that changed from the base of the PR and between 51c2b74 and f366a04.

📒 Files selected for processing (2)

cpp/src/io/parquet/reader_impl_preprocess.cu
cpp/tests/io/parquet_chunked_reader_test.cu

🚧 Files skipped from review as they are similar to previous changes (1)

cpp/tests/io/parquet_chunked_reader_test.cu

mhaseeb123 · 2026-06-24T00:58:34Z

-    // Compute column chunk level page count offsets, and page level row counts and row offsets.
-    auto const [page_row_counts, page_row_offsets, col_chunk_page_offsets] =
-      compute_page_row_counts_and_offsets(per_file_metadata, row_group_indices, schema_idx, stream);
+    // Compute column chunk level page count offsets and page level row offsets.


This function now doesn't return page_row_counts (not needed anymore)

mhaseeb123 · 2026-06-24T00:59:09Z

-      auto [page_row_counts, page_row_offsets, col_chunk_page_offsets, min, max, is_null] =
+      // Compute page row offsets, column chunk page offsets, min, max and optional is_null stats
+      // host columns.
+      auto [page_row_offsets, col_chunk_page_offsets, min, max, is_null] =


Return type doesn't include page_row_counts anymore

mhaseeb123 · 2026-06-24T00:59:13Z

-      auto [page_row_counts, page_row_offsets, col_chunk_page_offsets, min, max, is_null] =
+      // Compute page row offsets, column chunk page offsets, min, max and optional is_null stats
+      // host columns.
+      auto [page_row_offsets, col_chunk_page_offsets, min, max, is_null] =


Return type doesn't include page_row_counts anymore

mhaseeb123 · 2026-06-24T00:59:28Z

-                                                           stream,
-                                                           cudf::get_current_device_resource_ref());
+      // Construct a row indices mapping based on page row offsets.
+      auto const page_indices = compute_page_indices_async(


compute_page_indices_async doesn't need page_row_counts anymore

mhaseeb123 · 2026-06-24T00:59:58Z

-                         page_indices.begin(),
-                         cuda::maximum<cudf::size_type>());
+  auto page_indices = rmm::device_uvector<cudf::size_type>(total_rows, stream, mr);
+  cudf::detail::label_segments(


Use detail::label_segments instead of manual computation

vyasr

Looks fine to me.

mhaseeb123 · 2026-07-01T19:31:04Z

/merge

Review feedback: revert the conflict-resolution judgment call from the NVIDIA#22879 backport that kept cudf::host_span on compute_has_page_index() and compute_page_row_offsets_and_colchunk_page_offsets(). Use std::span on these two signatures instead, matching upstream PR NVIDIA#22879/main exactly. Callers in page_index_filter.cu continue to pass cudf::host_span-typed members, which convert implicitly via host_span::operator std::span<T>(). compute_page_row_offsets() and compute_page_indices_async() are unrelated to this PR's diff and keep cudf::host_span as in the 26.06 base.

Contributes to NVIDIA#22849 This PR enables optionally prepending a source index column to the read table. Authors: - Muhammad Haseeb (https://github.com/mhaseeb123) - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Yunsong Wang (https://github.com/PointKernel) - Vyas Ramasubramani (https://github.com/vyasr) URL: NVIDIA#22879

…snapshot] Backport of the effective diff of rapidsai/cudf PR NVIDIA#23077 ("Prepend row index column in Parquet reader"), which was still OPEN at the time this branch was created. This captures the PR's net change as a single squashed commit computed as the diff between its merge-base with main and its head: merge-base: 52d322c head: 2574b9d Applied on top of the 26.06.01 pin + NVIDIA#22773 + NVIDIA#22879 backport. NOTE: This is a PRE-MERGE snapshot; re-sync to the squashed merge commit once NVIDIA#23077 lands upstream. Unrelated main-only tests present as diff context (MismatchedSchema*) were intentionally excluded.

Review feedback: revert the conflict-resolution judgment call from the NVIDIA#22879 backport that kept cudf::host_span on compute_has_page_index() and compute_page_row_offsets_and_colchunk_page_offsets(). Use std::span on these two signatures instead, matching upstream PR NVIDIA#22879/main exactly. Callers in page_index_filter.cu continue to pass cudf::host_span-typed members, which convert implicitly via host_span::operator std::span<T>(). compute_page_row_offsets() and compute_page_indices_async() are unrelated to this PR's diff and keep cudf::host_span as in the 26.06 base.

Closes #22849. Follow up of #22879 + #23077 This PR enables prepending source and row index columns in hybrid scan. For two-step materialization, the extra index columns are prepended to filter columns only. Single-step materialization is identical to the main parquet reader Authors: - Muhammad Haseeb (https://github.com/mhaseeb123) Approvers: - Yunsong Wang (https://github.com/PointKernel) - Vyas Ramasubramani (https://github.com/vyasr) URL: #22878

Prepend source idx column to parquet

6165df4

github-actions Bot added the libcudf Affects libcudf (C++/CUDA) code. label Jun 12, 2026

github-actions Bot assigned mhaseeb123 Jun 12, 2026

mhaseeb123 mentioned this pull request Jun 12, 2026

Prepend source and row index columns in Hybrid scan reader #22878

Merged

3 tasks

mhaseeb123 changed the title ~~Prepend source index column to Parquet reader~~ Prepend source index column in Parquet reader Jun 12, 2026

mhaseeb123 added feature request New feature or request 3 - Ready for Review Ready for review by team cuIO cuIO issue Spark Functionality that helps Spark RAPIDS Velox Functionality that helps Velox-cudf non-breaking Non-breaking change labels Jun 12, 2026

mhaseeb123 mentioned this pull request Jun 12, 2026

[STORY] Parquet reader features for Apache Iceberg #22849

Closed

mhaseeb123 commented Jun 12, 2026

View reviewed changes

mhaseeb123 added 2 commits June 12, 2026 17:10

Merge branch 'main' into codex/issue-22849-source-index-visible-review

a6a05f6

Style

cdb627a

mhaseeb123 marked this pull request as ready for review June 13, 2026 00:10

mhaseeb123 requested a review from a team as a code owner June 13, 2026 00:10

mhaseeb123 requested review from PointKernel and qbacpey June 13, 2026 00:10

Revert include_output_num_rows_per_source until PR NVIDIA#22878

51c2b74

mhaseeb123 requested a review from vuule June 15, 2026 21:18

Minor bugfix

f366a04

coderabbitai Bot reviewed Jun 16, 2026

View reviewed changes

mhaseeb123 added 2 commits June 22, 2026 09:10

Merge branch 'main' into codex/issue-22849-source-index-visible-review

8e53fac

Merge branch 'main' into codex/issue-22849-source-index-visible-review

e8bef6e

PointKernel reviewed Jun 22, 2026

View reviewed changes

Comment thread cpp/src/io/parquet/reader_impl.cpp

Comment thread cpp/src/io/parquet/reader_impl_preprocess.cu Outdated

mhaseeb123 and others added 2 commits June 23, 2026 17:30

Merge branch 'main' into codex/issue-22849-source-index-visible-review

1ddc3dd

Style fix

942a59c

mhaseeb123 commented Jun 24, 2026

View reviewed changes

PointKernel approved these changes Jun 24, 2026

View reviewed changes

mhaseeb123 added 2 commits June 24, 2026 11:56

Merge branch 'main' into codex/issue-22849-source-index-visible-review

d6a9fe4

Merge branch 'main' into codex/issue-22849-source-index-visible-review

e8fa3b7

mhaseeb123 added 4 - Needs Review Waiting for reviewer to review or respond and removed 3 - Ready for Review Ready for review by team labels Jun 26, 2026

mhaseeb123 and others added 5 commits June 25, 2026 18:02

Merge branch 'main' into codex/issue-22849-source-index-visible-review

72a4d2d

Merge branch 'main' into codex/issue-22849-source-index-visible-review

ad088cd

Merge branch 'main' into codex/issue-22849-source-index-visible-review

f7041b3

Merge branch 'main' into codex/issue-22849-source-index-visible-review

759aa06

Merge branch 'main' into codex/issue-22849-source-index-visible-review

a7cb9db

github-actions Bot assigned vyasr Jun 30, 2026

vyasr approved these changes Jun 30, 2026

View reviewed changes

Comment thread cpp/src/io/parquet/expression_transform_helpers.cpp Outdated

Comment thread cpp/src/io/parquet/expression_transform_helpers.cpp Outdated

mhaseeb123 added 2 commits June 30, 2026 17:43

Merge branch 'main' into codex/issue-22849-source-index-visible-review

1b2e6e3

Address comments

077f0b5

mhaseeb123 added 5 - Ready to Merge Testing and reviews complete, ready to merge and removed 4 - Needs Review Waiting for reviewer to review or respond labels Jul 1, 2026

mhaseeb123 added 3 commits July 1, 2026 00:52

Use span instead of host_span

1711052

Merge branch 'main' into codex/issue-22849-source-index-visible-review

db506ff

Merge branch 'main' into codex/issue-22849-source-index-visible-review

5e67503

rapids-bot Bot merged commit e7cae5e into NVIDIA:main Jul 1, 2026
135 of 136 checks passed

mhaseeb123 deleted the codex/issue-22849-source-index-visible-review branch July 1, 2026 19:31

coderabbitai Bot mentioned this pull request Jul 7, 2026

Prepend row index column in Parquet reader #23077

Merged

3 tasks

coderabbitai Bot mentioned this pull request Jul 9, 2026

Supply stream and mr to column synthesizers #23209

Merged

3 tasks

Uh oh!

Conversation

mhaseeb123 commented Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Uh oh!

copy-pr-bot Bot commented Jun 12, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot commented Jun 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reviews paused

Walkthrough

Changes

Estimated code review effort

Suggested labels

Suggested reviewers

❌ Failed checks (1 warning)

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vyasr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mhaseeb123 commented Jul 1, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mhaseeb123 commented Jun 12, 2026 •

edited

Loading

coderabbitai Bot commented Jun 13, 2026 •

edited

Loading