Skip to content

馃毀 [DO NOT MERGE] Transcode parquet to cuDF dictionaries for STRING - #22247

Closed
mhaseeb123 wants to merge 4 commits into
NVIDIA:mainfrom
mhaseeb123:fea/pq-to-dict-encode
Closed

馃毀 [DO NOT MERGE] Transcode parquet to cuDF dictionaries for STRING#22247
mhaseeb123 wants to merge 4 commits into
NVIDIA:mainfrom
mhaseeb123:fea/pq-to-dict-encode

Conversation

@mhaseeb123

@mhaseeb123 mhaseeb123 commented Apr 21, 2026

Copy link
Copy Markdown
Contributor

Description

This PR adds a prototype implementation of transcoding dictionaries from parquet to cuDF directly when possible and specified for flat string type columns.

What's missing or needs fixing

  • Deduplication of dictionary values across row groups. Current prototype just concatenates dictionary values into a single column
  • Update the keys to use the deduplicated value indices
  • Ensure we aren't just hacking into the decode process with extra kernel launched and actually allocating required buffers and directly decoding data into them in the decode kernel launch.
  • Ensure memory and compute patterns are optimal and code modernization
  • Support for chunked reading (may need to preserve sub-colchunk dictionaries across sub-passes). Currently, we assume that we will not chunk anything.
  • Add more tests

Note: Do not add a merge commit to this PR. Let's keep the commit tree clean.

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@mhaseeb123
mhaseeb123 requested review from a team as code owners April 21, 2026 23:22
@github-actions github-actions Bot added libcudf Affects libcudf (C++/CUDA) code. CMake CMake build issue labels Apr 21, 2026
@mhaseeb123
mhaseeb123 marked this pull request as draft April 21, 2026 23:23
@copy-pr-bot

copy-pr-bot Bot commented Apr 21, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@mhaseeb123
mhaseeb123 removed the request for review from davidwendt April 21, 2026 23:23
@GregoryKimball GregoryKimball moved this to Burndown in libcudf May 4, 2026
@mhaseeb123 mhaseeb123 moved this from Burndown to Slip in libcudf May 7, 2026
@mhaseeb123 mhaseeb123 removed the status in libcudf May 14, 2026
@mhaseeb123 mhaseeb123 moved this to Slip in libcudf May 22, 2026
@GregoryKimball GregoryKimball removed this from libcudf Jul 13, 2026
@mhaseeb123

Copy link
Copy Markdown
Contributor Author

Superseded by #22532

@mhaseeb123 mhaseeb123 closed this Aug 8, 2026
@mhaseeb123
mhaseeb123 deleted the fea/pq-to-dict-encode branch August 8, 2026 04:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CMake CMake build issue libcudf Affects libcudf (C++/CUDA) code.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants