Add Iceberg equality delete file reader by apurva-meta · Pull Request #16871 · facebookincubator/velox

apurva-meta · 2026-03-21T06:12:36Z

Summary:
Implements Iceberg equality delete support for the Velox Iceberg connector.

Equality delete files contain rows with values for one or more columns
(identified by equalityFieldIds). A base data row is deleted if its values
match ALL specified columns of ANY row in the delete file.

The implementation:

Adds EqualityDeleteFileReader that eagerly reads the entire delete file
and builds an in-memory hash multimap of delete key tuples during
construction.
Wires EqualityDeleteFileReader into IcebergSplitReader::prepareSplit()
to resolve equalityFieldIds to column names/types from the table schema,
and into IcebergSplitReader::next() to apply post-read equality delete
filtering with row compaction.
Handles lazy vectors from file readers via loadedVector() before
accessing values for hashing and comparison.
Supports BOOLEAN, TINYINT, SMALLINT, INTEGER, BIGINT, REAL, DOUBLE,
VARCHAR, VARBINARY, and TIMESTAMP column types.

Differential Revision: D97530141

Summary: - Add explicit equality deletes NYI branch in prepareSplit() - Improve VELOX_NYI error messages with descriptive content type info - Fix FILE handle leaks in IcebergReadTest by extracting getTestFileSize() helper - Minor doc comment formatting improvements in IcebergSplitReader.h Differential Revision: D97530140

netlify · 2026-03-21T06:12:42Z

✅ Deploy Preview for meta-velox canceled.

Name	Link
🔨 Latest commit	`7de3176`
🔍 Latest deploy log	https://app.netlify.com/projects/meta-velox/deploys/69c61ba17f162b0008b769c0

meta-codesync · 2026-03-21T06:12:51Z

@apurva-meta has exported this pull request. If you are a Meta employee, you can view the originating Diff in D97530141.

Summary: Add deletion vector (DV) reader to the Velox Iceberg connector, enabling Iceberg V3 spec support for row-level deletes. Iceberg V3 replaces positional delete files with deletion vectors — compact roaring bitmaps stored as blobs inside Puffin files. Compared to V2 positional delete files, DVs are more compact and avoid sorted merge of multiple delete files at read time. Changes: - IcebergDeleteFile.h: Add FileContent::kDeletionVector enum value - DeletionVectorReader.h/cpp: New reader that loads a Puffin blob, deserializes the roaring bitmap portable format (array, bitset, and run containers), and sets bits in the deleteBitmap for the current batch range. No CRoaring dependency — self-contained parser. - IcebergSplitReader.h: Add deletionVectorReaders_ member, include header - IcebergSplitReader.cpp: Route kDeletionVector in prepareSplit(), apply DVs alongside positional deletes in next() - BUCK: Add DeletionVectorReader library target Differential Revision: D97530142

jinchengchenghh · 2026-03-25T10:26:21Z

velox/connectors/hive/iceberg/EqualityDeleteFileReader.h

+  std::vector<TypePtr> equalityColumnTypes_;
+
+  /// Column indices in the delete file output vector.
+  std::vector<column_index_t> deleteColumnIndices_;


How do you think about the complex data type column delete, such as ROW(ROW(a))

jinchengchenghh · 2026-03-25T10:52:35Z

velox/connectors/hive/iceberg/EqualityDeleteFileReader.h

+  /// @param connectorId Connector identifier.
+  EqualityDeleteFileReader(
+      const IcebergDeleteFile& deleteFile,
+      const std::vector<std::string>& equalityColumnNames,


Could you use IcebergMetadataColumn as equalityColumn specifier, this will help schema evolution and interface consistence

jinchengchenghh · 2026-03-25T10:58:45Z

velox/connectors/hive/iceberg/IcebergSplitReader.cpp

+    // Use a separate bitmap for equality deletes to track which rows to
+    // remove from the output.
+    BufferPtr eqDeleteBitmap = AlignedBuffer::allocate<bool>(
+        numRows, connectorQueryCtx_->memoryPool());


bits::nbytes(numRows)

jinchengchenghh · 2026-03-25T11:01:13Z

velox/connectors/hive/iceberg/IcebergSplitReader.h

  uint64_t splitOffset_;
  std::list<std::unique_ptr<PositionalDeleteFileReader>>
      positionalDeleteFileReaders_;
  BufferPtr deleteBitmap_;
+
+  /// Readers for Iceberg V3 deletion vectors (Puffin-encoded roaring bitmaps).
+  std::list<std::unique_ptr<DeletionVectorReader>> deletionVectorReaders_;


std::list -> std::vector

jinchengchenghh · 2026-03-25T11:04:15Z

velox/connectors/hive/iceberg/EqualityDeleteFileReader.cpp

+
+/// Hashes a single value from a vector at the given index.
+/// Handles lazy vectors via loadedVector(). Returns 0 for null values.
+uint64_t hashValue(const VectorPtr& vectorPtr, vector_size_t index) {


Can we use VectorHasher?

jinchengchenghh · 2026-03-25T11:07:59Z

velox/connectors/hive/iceberg/IcebergSplitReader.cpp

+
+    // Use a separate bitmap for equality deletes to track which rows to
+    // remove from the output.
+    BufferPtr eqDeleteBitmap = AlignedBuffer::allocate<bool>(


Maybe we can optimize the eqDeleteBitmap life time for all the inputs later

jinchengchenghh · 2026-03-25T11:09:19Z

velox/connectors/hive/iceberg/IcebergSplitReader.cpp

+    auto numRows = outputRowVector->size();
+
+    // Use a separate bitmap for equality deletes to track which rows to
+    // remove from the output.


Looks like the usage is similar to SelectivityVector

Summary: Implements Iceberg equality delete support for the Velox Iceberg connector. Equality delete files contain rows with values for one or more columns (identified by equalityFieldIds). A base data row is deleted if its values match ALL specified columns of ANY row in the delete file. The implementation: - Adds EqualityDeleteFileReader that eagerly reads the entire delete file and builds an in-memory hash multimap of delete key tuples during construction. - Wires EqualityDeleteFileReader into IcebergSplitReader::prepareSplit() to resolve equalityFieldIds to column names/types from the table schema, and into IcebergSplitReader::next() to apply post-read equality delete filtering with row compaction. - Handles lazy vectors from file readers via loadedVector() before accessing values for hashing and comparison. - Supports BOOLEAN, TINYINT, SMALLINT, INTEGER, BIGINT, REAL, DOUBLE, VARCHAR, VARBINARY, and TIMESTAMP column types. Differential Revision: D97530141

nmahadevuni · 2026-03-30T07:22:05Z

@apurva-meta Thanks for this PR. Can we move the changes related to deletion vector reader into a separate PR?

apurva-meta requested a review from majetideepak as a code owner March 21, 2026 06:12

meta-cla bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Mar 21, 2026

meta-codesync bot added fb-exported meta-exported labels Mar 21, 2026

apurva-meta force-pushed the export-D97530141 branch from 0a911ca to cb6324c Compare March 21, 2026 06:24

meta-codesync bot changed the title ~~feat: [velox][iceberg] Add Iceberg equality delete file reader~~ feat: [velox][iceberg] Add Iceberg equality delete file reader (#16871) Mar 21, 2026

apurva-meta force-pushed the export-D97530141 branch from cb6324c to 8249f53 Compare March 21, 2026 06:26

apurva-meta force-pushed the export-D97530141 branch 2 times, most recently from a03eb81 to 83d5a3e Compare March 21, 2026 06:29

apurva-meta force-pushed the export-D97530141 branch from 83d5a3e to 243eef6 Compare March 21, 2026 06:32

apurva-meta mentioned this pull request Mar 23, 2026

Add DWRF file format support for Iceberg data sink #16875

Closed

apurva-meta mentioned this pull request Mar 24, 2026

[native]: Iceberg V3 support prestodb/presto#27198

Open

18 tasks

jinchengchenghh reviewed Mar 25, 2026

View reviewed changes

apurva-meta force-pushed the export-D97530141 branch from 243eef6 to 5c04444 Compare March 27, 2026 05:51

meta-codesync bot changed the title ~~feat: [velox][iceberg] Add Iceberg equality delete file reader (#16871)~~ Add Iceberg equality delete file reader Mar 27, 2026

apurva-meta force-pushed the export-D97530141 branch from 5c04444 to 7de3176 Compare March 27, 2026 05:54

apurva-meta closed this Mar 30, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Iceberg equality delete file reader#16871

Add Iceberg equality delete file reader#16871
apurva-meta wants to merge 3 commits intofacebookincubator:mainfrom
apurva-meta:export-D97530141

apurva-meta commented Mar 21, 2026 •

edited by meta-codesync bot

Loading

Uh oh!

netlify bot commented Mar 21, 2026 •

edited

Loading

Uh oh!

meta-codesync bot commented Mar 21, 2026

Uh oh!

jinchengchenghh Mar 25, 2026

Uh oh!

jinchengchenghh Mar 25, 2026

Uh oh!

jinchengchenghh Mar 25, 2026

Uh oh!

jinchengchenghh Mar 25, 2026

Uh oh!

jinchengchenghh Mar 25, 2026

Uh oh!

jinchengchenghh Mar 25, 2026

Uh oh!

jinchengchenghh Mar 25, 2026

Uh oh!

nmahadevuni commented Mar 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

apurva-meta commented Mar 21, 2026 • edited by meta-codesync bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

netlify bot commented Mar 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for meta-velox canceled.

Uh oh!

meta-codesync bot commented Mar 21, 2026

Uh oh!

jinchengchenghh Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

jinchengchenghh Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

jinchengchenghh Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

jinchengchenghh Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

jinchengchenghh Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

jinchengchenghh Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

jinchengchenghh Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

nmahadevuni commented Mar 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

apurva-meta commented Mar 21, 2026 •

edited by meta-codesync bot

Loading

netlify bot commented Mar 21, 2026 •

edited

Loading