Add DWRF file format support for Iceberg data sink#16875
Add DWRF file format support for Iceberg data sink#16875apurva-meta wants to merge 7 commits intofacebookincubator:mainfrom
Conversation
Summary: - Add explicit equality deletes NYI branch in prepareSplit() - Improve VELOX_NYI error messages with descriptive content type info - Fix FILE handle leaks in IcebergReadTest by extracting getTestFileSize() helper - Minor doc comment formatting improvements in IcebergSplitReader.h Differential Revision: D97530140
✅ Deploy Preview for meta-velox canceled.
|
meta-cla
bot
added
the
CLA Signed
|
@apurva-meta has exported this pull request. If you are a Meta employee, you can view the originating Diff in D97530138. |
PingLiuPing
left a comment
PingLiuPing
left a comment
There was a problem hiding this comment.
Hi @apurva-meta, thanks for the code.
I noticed that several recently opened PRs seem to overlap. Could you please open a GitHub issue outlining your overall plan and how these changes fit together? If the work is still in progress, it would be great to mark the PRs as drafts for now.
|
Why you add dwrf? Iceberg does not support dwrf, it only supports ORC, if you write as dwrf, Apache Iceberg cannot read it. |
|
Velox stack |
Presto stack: | Diff | Title | GitHub PR-- | -- | -- | -- |
@PingLiuPing Request you to review above PRs. I will remove the duplicates (not listed here). |
|
Summary: Add deletion vector (DV) reader to the Velox Iceberg connector, enabling Iceberg V3 spec support for row-level deletes. Iceberg V3 replaces positional delete files with deletion vectors — compact roaring bitmaps stored as blobs inside Puffin files. Compared to V2 positional delete files, DVs are more compact and avoid sorted merge of multiple delete files at read time. Changes: - IcebergDeleteFile.h: Add FileContent::kDeletionVector enum value - DeletionVectorReader.h/cpp: New reader that loads a Puffin blob, deserializes the roaring bitmap portable format (array, bitset, and run containers), and sets bits in the deleteBitmap for the current batch range. No CRoaring dependency — self-contained parser. - IcebergSplitReader.h: Add deletionVectorReaders_ member, include header - IcebergSplitReader.cpp: Route kDeletionVector in prepareSplit(), apply DVs alongside positional deletes in next() - BUCK: Add DeletionVectorReader library target Differential Revision: D97530142
Summary: Implements Iceberg equality delete support for the Velox Iceberg connector. Equality delete files contain rows with values for one or more columns (identified by equalityFieldIds). A base data row is deleted if its values match ALL specified columns of ANY row in the delete file. The implementation: - Adds EqualityDeleteFileReader that eagerly reads the entire delete file and builds an in-memory hash multimap of delete key tuples during construction. - Wires EqualityDeleteFileReader into IcebergSplitReader::prepareSplit() to resolve equalityFieldIds to column names/types from the table schema, and into IcebergSplitReader::next() to apply post-read equality delete filtering with row compaction. - Handles lazy vectors from file readers via loadedVector() before accessing values for hashing and comparison. - Supports BOOLEAN, TINYINT, SMALLINT, INTEGER, BIGINT, REAL, DOUBLE, VARCHAR, VARBINARY, and TIMESTAMP column types. Differential Revision: D97530141
Summary: Implements Iceberg V2+ sequence number conflict resolution for equality delete files. Per the Iceberg spec, an equality delete file should only be applied to data files whose data sequence number is strictly less than the delete file's data sequence number. This prevents concurrent write conflicts where a delete file written concurrently with a data file could incorrectly delete rows that were not intended to be deleted. Changes: - Added `dataSequenceNumber` field to `IcebergDeleteFile` struct with default value 0 (unassigned/legacy V1). When 0, sequence number filtering is disabled for backward compatibility. - Added `dataSequenceNumber` field to `HiveIcebergSplit` to carry the base data file's sequence number. - Updated `IcebergSplitReader::prepareSplit()` to skip equality delete files when `deleteFile.dataSequenceNumber <= split.dataSequenceNumber` (unless either is 0, which disables the check). - Updated test constructor of `HiveIcebergSplit` to accept the new `dataSequenceNumber` parameter. Differential Revision: D97530136
…letion vectors Summary: Extend the sequence number conflict resolution logic to positional deletes and deletion vectors. Per the Iceberg spec: - Positional deletes and DVs skip when deleteFileSeqNum < dataFileSeqNum (strictly less than, unlike equality deletes which use <=) - Sequence number 0 (V1 legacy) never skips Changes: - IcebergSplitReader: Apply sequence number filtering for positional deletes and deletion vectors before passing to readers - IcebergReadFile: Store dataSequenceNumber for conflict resolution - Tests: Add sequence number conflict resolution tests for positional deletes and DVs Differential Revision: D97530139
Summary: - Add DeletionVectorWriter.cpp/h implementing DV file writing using RoaringBitmapArray - Support both position-based (positional deletes) and DV-based delete file writing - Add DeletionVectorWriterTest with comprehensive unit tests - Update BUCK and CMakeLists.txt build targets Differential Revision: D97530137
DWRF is also open sourced: https://github.com/prestodb/presto-hive-dwrf |
|
Now Velox does not support fully Iceberg parquet, for example, schema evolution, if user add a column, then drop column, then add a new column with same name, in Iceberg, they are the different column, so in Gluten, we will fallback this case, and read by JVM Iceberg, in another word, write native, read JVM, but for dwrf, it cannot fallback then, this means we lack for the schema evolution, forbid users from performing schema evolution or user cannot read the files if they alter the format to parquet and add column. Iceberg supports multiple formats in one table. DWRF does not have a significant advantage over Parquet, so I suggest to consider seriously to expose this feature to user at this time. |
Summary: - Add DWRF file format support in IcebergDataSink for read and write paths - Update BUCK build targets for DWRF dependencies - Add IcebergDwrfInsertTest with comprehensive insert/read tests for DWRF format - Update CMakeLists.txt for new test targets Differential Revision: D97530138
apurva-meta
force-pushed
the
export-D97530138
branch
from
fdfb9fe to
7d9e3d8
Compare
meta-codesync
bot
changed the title
3 participants
Summary:
Differential Revision: D97530138