Support IF_NOT_CONTAINED filter type and loading inline deletion vectors for OSS delta [databricks] by jihoonson · Pull Request #15368 · NVIDIA/cudf-spark

jihoonson · 2026-07-23T21:03:31Z

Description

The CDF read with deletion vectors currently fails. Two things were missing to support this case:

The IF_NOT_CONTAINED row index filter type support. The Delta CDC reader can use this type of row index filter.
Inline deletion vector support. The CDC reader creates inline deletion vectors.

This PR adds those supports based on NVIDIA/cudf#23402 for OSS Delta. The plugin now can load inline deletion vectors and process the IF_NOT_CONTAINED filter properly with all 3 Delta readers.

Note that the issue exists only with the native readers (GpuDeltaParquetFileFormatBase2). The legacy reader (GpuDeltaParquetFileFormatBase) does not have this issue.

Databricks readers have the same issue, and will be fixed in #15365.

Checklists

Documentation

Updated for new or modified user-facing features or behaviors
No user-facing change

Testing

Added or modified tests to cover new code paths
Covered by existing tests
(Please provide the names of the existing tests in the PR description.)
Not required

Performance

Tests ran and results are added in the PR description
Issue filed with a link in the PR description
Not required

…ors for OSS delta Signed-off-by: Jihoon Son <ghoonson@gmail.com>

greptile-apps · 2026-07-23T21:17:52Z

Greptile Summary

This PR fixes CDF (Change Data Feed) reads with deletion vectors in the OSS Delta native readers (GpuDeltaParquetFileFormatBase2) by adding two missing capabilities: support for the IF_NOT_CONTAINED row-index filter type (used when the CDC reader builds a difference bitmap between an old and a new DV) and inline deletion vector loading (CDC creates inline DVs during certain commit operations).

IF_NOT_CONTAINED support: The SpillableDeletionVectorInfo, DeltaParquetExtraInfo, and PerFileDVEntry case classes now carry filterTypeOpt, replacing the hardcoded IF_CONTAINED assumption. computeNumRowsAlive is refactored into a shared helper that correctly returns numMarkedRows for IF_NOT_CONTAINED and totalRows - numMarkedRows for IF_CONTAINED.
Inline DV loading: RapidsDeletionVectorStoredBitmap.load is extended to dispatch to RapidsInMemoryDeletionVectorStore for inline DVs. The DeltaSerializedBitmapLoader gains a loadFromBytes path that skips the CRC step (inline DVs have no trailing checksum).
ByteBufferInputStream is promoted from a private class in ParquetCachedBatchSerializer to a public shared utility with proper argument validation, and a suite of unit tests is added alongside comprehensive integration tests covering PERFILE/COALESCING/MULTITHREADED readers and mixed filter type scenarios.

Confidence Score: 5/5

Safe to merge; the changes are well-scoped, all three reader paths are exercised by integration tests, and no existing callers are broken.

The two new behaviours — IF_NOT_CONTAINED row-count math and inline DV byte loading — are each straightforward and verified by dedicated integration tests using assert_gpu_and_cpu_are_equal_collect. The filterTypeOpt field threads cleanly through SpillableDeletionVectorInfo, DeltaParquetExtraInfo, and PerFileDVEntry without touching the GPU decode path. The ByteBufferInputStream extraction adds argument validation without changing semantics for existing callers. No GPU resource leaks, OOM-retry gaps, or cross-shim inconsistencies were found.

Files Needing Attention: No files require special attention.

Important Files Changed

Filename	Overview
delta-lake/common/src/main/delta-33x-41x/scala/com/nvidia/spark/rapids/delta/common/GpuDeltaParquetFileFormatBase2.scala	Core reader refactored to propagate filterTypeOpt through all three reader paths (single-file, coalescing, multithreaded); numRowsDeleted replaced by numRowsAlive in SpillableDeletionVectorInfo; require guard updated to accept IF_NOT_CONTAINED; @transient added to params.
delta-lake/common/src/main/delta-33x-41x/scala/com/nvidia/spark/rapids/delta/common/RapidsDeletionVectors.scala	loadDeletionVector and loadScalaBitmap updated to accept IF_NOT_CONTAINED; countDeletedRows renamed countMarkedRows (private); new computeNumRowsAlive correctly branches on filter type; isIfNotContainedRowIndexFilter helper added.
delta-lake/common/src/main/delta-33x-41x/scala/org/apache/spark/sql/delta/deletionvectors/RapidsDeletionVectorStore.scala	Added RapidsInMemoryDeletionVectorStore for inline DVs; DeltaSerializedBitmapLoader gains loadFromBytes (no CRC); loadAsStandardFormat crc parameter made Option to allow skipping checksum for inline format; loaders made private.
delta-lake/common/src/main/delta-33x-41x/scala/org/apache/spark/sql/delta/deletionvectors/RapidsStoredBitmap.scala	Removed isOnDisk-only restriction; load() now accepts RapidsFileIO and dispatches to inline, empty, or on-disk path; require(isOnDisk) added as defensive guard inside the on-disk branch.
delta-lake/common/src/main/scala/com/nvidia/spark/rapids/delta/RapidsDeletionVectorRowCountUtils.scala	countDeletedRows renamed countMarkedRows (neutral correctness change); legacy computeNumRowsAlive still present for the Databricks/legacy reader path and still assumes IF_CONTAINED semantics, which is correct for its only caller.
delta-lake/delta-spark400db173/src/main/scala/com/nvidia/spark/rapids/delta/RapidsDeletionVectors.scala	Databricks DV helper updated to call countMarkedRows (renamed); no semantic change.
sql-plugin/src/main/scala/com/nvidia/spark/rapids/ByteBufferInputStream.scala	New public utility class migrated from ParquetCachedBatchSerializer; adds proper argument validation (null dest, negative offset/length, zero-length early return) and uses hasRemaining() idiom.
sql-plugin/src/main/scala/com/nvidia/spark/rapids/parquet/ParquetCachedBatchSerializer.scala	Removed local private ByteBufferInputStream; now imports the shared public class. No behavioral change for existing serializer callers.
integration_tests/src/main/python/delta_lake_test.py	Adds test_delta_deletion_vector_read_with_cdf and three mixed-filter-type tests (different partitions, same partition, DV-to-DV transition), plus mixed DV/non-DV COUNT(*) test; all use assert_gpu_and_cpu_are_equal_collect for GPU verification.
sql-plugin/src/test/scala/com/nvidia/spark/rapids/ByteBufferInputStreamSuite.scala	New unit test suite for ByteBufferInputStream covering zero-length reads, argument validation, and skip clamping; no GPU resources involved.

Sequence Diagram

sequenceDiagram
    participant Reader as GPU Delta Reader
    participant RDV as RapidsDeletionVectors
    participant RDSB as RapidsDVStoredBitmap
    participant InMem as RapidsInMemoryDVStore
    participant OnDisk as RapidsHadoopDVStore
    participant Loader as DeltaSerializedBitmapLoader

    Reader->>RDV: loadDeletionVector(fileIO, dvDescOpt, filterTypeOpt, tablePath)
    RDV->>RDSB: storedBitmap.load(fileIO)
    alt isEmpty
        RDSB-->>RDV: serializedEmptyBitmap()
    else isInline
        RDSB->>InMem: load(dvDescriptor.inlineData)
        InMem->>Loader: loadFromBytes(bytes) [no CRC]
        Loader-->>InMem: HostMemoryBuffer
        InMem-->>RDSB: HostMemoryBuffer
    else isOnDisk
        RDSB->>OnDisk: load(path, offset, size)
        OnDisk->>Loader: load(stream, size) [with CRC]
        Loader-->>OnDisk: HostMemoryBuffer
        OnDisk-->>RDSB: HostMemoryBuffer
    end
    RDSB-->>RDV: HostMemoryBuffer (serialized bitmap)
    RDV-->>Reader: HostMemoryBuffer

    Reader->>RDV: computeNumRowsAlive(totalRows, bitmap, filterTypeOpt, offsets, numRows)
    alt IF_CONTAINED
        RDV-->>Reader: totalRows - numMarkedRows
    else IF_NOT_CONTAINED
        RDV-->>Reader: numMarkedRows
    else None (no DV)
        RDV-->>Reader: totalRows
    end

    Reader->>Reader: DeletionVectorInfo(bitmap, isIfNotContained, offsets, numRows)
    Reader->>Reader: cuDF Parquet read with DV filter

_{Reviews (9): Last reviewed commit: "address comments" | Re-trigger Greptile}

jihoonson · 2026-07-24T00:45:22Z

build

nvauto · 2026-07-27T06:53:15Z

NOTE: release/26.08 has been created from main. Please retarget your PR to release/26.08 if it should be included in the release.

…into fix-cdf-dv

Copilot

Pull request overview

Adds missing native-reader support needed for Delta Change Data Feed (CDF) reads on tables with deletion vectors (DVs), specifically handling IF_NOT_CONTAINED row-index filters and inline (in-log) DV payloads, plus targeted test coverage.

Changes:

Extend DV loading/row-count logic to accept both IF_CONTAINED and IF_NOT_CONTAINED semantics and propagate that intent into the cuDF DV reader path.
Add inline deletion vector loading support for Delta 33x–41x by parsing inline bitmap bytes into host buffers.
Add/extend unit + integration tests covering ByteBuffer-backed stream edge cases and Delta CDF scenarios involving mixed row-index filter types.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
sql-plugin/src/test/scala/com/nvidia/spark/rapids/ByteBufferInputStreamSuite.scala	Adds unit tests for ByteBuffer-backed InputStream argument/edge-case behavior.
sql-plugin/src/main/scala/com/nvidia/spark/rapids/parquet/ParquetCachedBatchSerializer.scala	Switches to shared `ByteBufferInputStream` implementation (removes local copy).
sql-plugin/src/main/scala/com/nvidia/spark/rapids/ByteBufferInputStream.scala	Introduces shared ByteBuffer-backed InputStream with Spark-aligned semantics.
integration_tests/src/main/python/delta_lake_test.py	Adds CDF + deletion-vector integration tests, including mixed filter-type scenarios.
delta-lake/delta-spark400db173/src/main/scala/com/nvidia/spark/rapids/delta/RapidsDeletionVectors.scala	Adapts DV row counting helper usage to shared “marked rows” semantics.
delta-lake/common/src/main/scala/com/nvidia/spark/rapids/delta/RapidsDeletionVectorRowCountUtils.scala	Renames/counts bitmap “marked rows” (vs “deleted rows”) to support both filter semantics.
delta-lake/common/src/main/delta-33x-41x/scala/org/apache/spark/sql/delta/deletionvectors/RapidsStoredBitmap.scala	Adds inline DV loading path and wires DV store creation through RapidsFileIO.
delta-lake/common/src/main/delta-33x-41x/scala/org/apache/spark/sql/delta/deletionvectors/RapidsDeletionVectorStore.scala	Adds inline bitmap loader + refactors loaders to support non-CRC inline parsing.
delta-lake/common/src/main/delta-33x-41x/scala/com/nvidia/spark/rapids/delta/common/RapidsDeletionVectors.scala	Accepts `IF_NOT_CONTAINED`, adds alive-row counting based on filter semantics, and exposes helper flagging the filter type.
delta-lake/common/src/main/delta-33x-41x/scala/com/nvidia/spark/rapids/delta/common/GpuDeltaParquetFileFormatBase2.scala	Propagates filter-type into DV metadata and uses new alive-row computation for partition routing and reader metadata.

…into fix-cdf-dv

jihoonson · 2026-07-28T17:53:43Z

build

jihoonson · 2026-07-28T20:13:11Z

This PR is currently blocked by #15408.

…into fix-cdf-dv

jihoonson · 2026-07-28T23:41:24Z

build

gerashegalov · 2026-07-29T02:20:24Z

@@ -110,17 +114,17 @@ object RapidsDeletionVectors extends Logging {
    if (dvDescriptorOpt.isDefined && filterTypeOpt.isDefined) {


Could we pattern-match on (dvDescriptorOpt, filterTypeOpt) here instead of checking isDefined and then calling .get? An exhaustive tuple match would encode the both-defined-or-both-absent invariant directly and avoid unsafe access. Since loadScalaBitmap repeats the same validation, a small shared helper could also remove that duplication.

Fixed as suggested.

gerashegalov · 2026-07-29T02:20:31Z

+   * Loads a bitmap payload and validates its trailing checksum. The CRC is initialized
+   * with the magic number before this method is called.
+   */
  def loadAsStandardFormat(input: DataInputStream, size: Int, crc: CRC32): HostMemoryBuffer


Since this is a private trait and both implementations immediately forward these overloads to an Option[CRC32] implementation, could the trait expose a single crcOpt: Option[CRC32] method? Callers can pass Some(crc) or None, eliminating the four forwarding methods.

Fixed as suggested.

gerashegalov · 2026-07-29T02:20:37Z

+    extends InputStream {
+
+  override def read(): Int = {
+    if (buffer == null || buffer.remaining() == 0) {


Nit: !buffer.hasRemaining is the idiomatic NIO spelling for buffer.remaining() == 0; likewise at lines 50 and 64.

gerashegalov · 2026-07-29T02:20:45Z

+
+  def loadFromBytes(bytes: Array[Byte]): HostMemoryBuffer = {
+    val bb = ByteBuffer.wrap(bytes)
+    bb.order(ByteOrder.LITTLE_ENDIAN)


Nit: this can be constructed in one expression: val bb = ByteBuffer.wrap(bytes).order(ByteOrder.LITTLE_ENDIAN).

gerashegalov · 2026-07-29T02:20:52Z

+    bb.order(ByteOrder.LITTLE_ENDIAN)
+    val magicNumber = bb.getInt()
+    val remainingSize = bb.remaining()
+    withResource(new ByteBufferInputStream(bb)) { bais =>


Nit: the outer resource wrapper can be removed by constructing new DataInputStream(new ByteBufferInputStream(bb)) in a single withResource; closing the DataInputStream closes its underlying stream.

gerashegalov

LGTM, nits

jihoonson · 2026-07-29T16:26:22Z

Thanks @gerashegalov. I addressed all your comments.

nartal1 · 2026-07-29T17:50:59Z

Just a question else LGTM.

gerashegalov

LGTM

jihoonson · 2026-07-29T23:19:15Z

build

Resolves the latest merge conflicts in #15412 after #15423 merged. ### Description Merge the current `release/26.08` head into the current `main` head. The initial conflicts were the root and Scala 2.13 project versions. Both were resolved by retaining main's `26.10.0-SNAPSHOT` version instead of the release branch's `26.08.0-SNAPSHOT` version. The branch was refreshed again after additional PRs landed on `release/26.08`. The current diff includes all release updates added after #15423, including: - #15413 — preserve Spark 4.2 BroadcastHashJoin `isSkewJoin` - #15422 — fix Iceberg REST S3 path regression coverage - #15368 — OSS Delta deletion-vector updates - #15411 — fix OSS Delta RTAS on Spark 4.x+ - #15416 — match Spark 4.2 `date_trunc` overflow behavior ### Checklists Documentation - [ ] Updated for new or modified user-facing features or behaviors - [x] No user-facing change Testing - [ ] Added or modified tests to cover new code paths - [x] Covered by existing tests (The included release commits retain their original tests.) - [ ] Not required Performance - [ ] Tests ran and results are added in the PR description - [ ] Issue filed with a link in the PR description - [x] Not required ### Validation - `git diff --check` - Parsed both initially resolved POM files as XML - `python3 -m py_compile` for the modified Iceberg, Delta, and date-time integration tests IMPORTANT: Merge this PR using **Create a merge commit** so the release commit ancestry is preserved and #15412 can close automatically. --------- Signed-off-by: Sameer Raheja <sraheja@.nvidia.com> Signed-off-by: Rahul Prabhu <raprabhu@nvidia.com> Signed-off-by: Chong Gao <chongg@nvidia.com> Signed-off-by: Firestarman <firestarmanllc@gmail.com> Signed-off-by: Ray Liu <liurenjie2008@gmail.com> Signed-off-by: liyuan <yuali@nvidia.com> Signed-off-by: Jihoon Son <ghoonson@gmail.com> Signed-off-by: Niranjan Artal <nartal@nvidia.com> Co-authored-by: Sameer Raheja <sameerz@users.noreply.github.com> Co-authored-by: Sameer Raheja <sraheja@.nvidia.com> Co-authored-by: Gary Shen <gashen@nvidia.com> Co-authored-by: Rahul Prabhu <100436830+sdrp713@users.noreply.github.com> Co-authored-by: Chong Gao <chongg@nvidia.com> Co-authored-by: Chong Gao <res_life@163.com> Co-authored-by: Liangcai Li <firestarmanllc@gmail.com> Co-authored-by: Renjie Liu <liurenjie2008@gmail.com> Co-authored-by: Jihoon Son <ghoonson@gmail.com> Co-authored-by: Niranjan Artal <50492963+nartal1@users.noreply.github.com>

Support IF_NOT_CONTAINED filter type and loading inline deletion vect…

a86d3d2

…ors for OSS delta Signed-off-by: Jihoon Son <ghoonson@gmail.com>

jihoonson requested a review from a team July 23, 2026 21:03

jihoonson changed the title ~~Support IF_NOT_CONTAINED filter type and loading inline deletion vectors for OSS delta~~ Support IF_NOT_CONTAINED filter type and loading inline deletion vectors for OSS delta [databricks] Jul 23, 2026

sameerz added the bug Something isn't working label Jul 24, 2026

Merge branch 'main' into fix-cdf-dv

00a56f0

nartal1 reviewed Jul 24, 2026

View reviewed changes

Comment thread integration_tests/src/main/python/delta_lake_test.py

jihoonson changed the base branch from main to release/26.08 July 27, 2026 16:45

Merge branch 'release/26.08' of https://github.com/NVIDIA/spark-rapids …

7cc0643

…into fix-cdf-dv

jihoonson requested a review from Copilot July 27, 2026 17:59

Copilot started reviewing on behalf of jihoonson July 27, 2026 18:00 View session

Copilot AI reviewed Jul 27, 2026

View reviewed changes

Comment thread ...main/delta-33x-41x/scala/org/apache/spark/sql/delta/deletionvectors/RapidsStoredBitmap.scala

Comment thread sql-plugin/src/main/scala/com/nvidia/spark/rapids/ByteBufferInputStream.scala

address comments

71a5529

greptile-apps Bot reviewed Jul 27, 2026

View reviewed changes

Comment thread ...elta-33x-41x/scala/com/nvidia/spark/rapids/delta/common/GpuDeltaParquetFileFormatBase2.scala

jihoonson added 3 commits July 27, 2026 18:40

Merge branch 'release/26.08' of https://github.com/NVIDIA/spark-rapids …

10cbd8b

…into fix-cdf-dv

Merge branch 'release/26.08' of https://github.com/NVIDIA/spark-rapids …

0a20dec

…into fix-cdf-dv

add a test for when reading files with dv and ones without dv

309e1b6

Merge branch 'release/26.08' of https://github.com/NVIDIA/spark-rapids …

8b96561

…into fix-cdf-dv

sameerz requested review from liurenjie1024 and nartal1 July 29, 2026 00:13

gerashegalov reviewed Jul 29, 2026

View reviewed changes

gerashegalov previously approved these changes Jul 29, 2026

View reviewed changes

address comments

c914a3c

jihoonson dismissed gerashegalov’s stale review via c914a3c July 29, 2026 16:25

nartal1 reviewed Jul 29, 2026

View reviewed changes

Comment thread ...main/delta-33x-41x/scala/org/apache/spark/sql/delta/deletionvectors/RapidsStoredBitmap.scala

gerashegalov approved these changes Jul 29, 2026

View reviewed changes

jihoonson merged commit 5efcd17 into NVIDIA:release/26.08 Jul 30, 2026
55 checks passed

nvliyuan mentioned this pull request Jul 30, 2026

Merge latest release/26.08 into main [skip ci] #15434

Merged

8 tasks

jihoonson mentioned this pull request Jul 30, 2026

[BUG] Delta CDF read of a table with deletion vector fails #15326

Closed

		@@ -110,17 +114,17 @@ object RapidsDeletionVectors extends Logging {
		if (dvDescriptorOpt.isDefined && filterTypeOpt.isDefined) {

Uh oh!

Conversation

jihoonson commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklists

Uh oh!

greptile-apps Bot commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Sequence Diagram

Uh oh!

jihoonson commented Jul 24, 2026

Uh oh!

Uh oh!

nvauto commented Jul 27, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jihoonson commented Jul 28, 2026

Uh oh!

jihoonson commented Jul 28, 2026

Uh oh!

jihoonson commented Jul 28, 2026

Uh oh!

gerashegalov Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

jihoonson Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

gerashegalov Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

jihoonson Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

gerashegalov Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

gerashegalov Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

gerashegalov Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

gerashegalov left a comment

Choose a reason for hiding this comment

Uh oh!

jihoonson commented Jul 29, 2026

Uh oh!

Uh oh!

nartal1 commented Jul 29, 2026

Uh oh!

gerashegalov left a comment

Choose a reason for hiding this comment

Uh oh!

jihoonson commented Jul 29, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

jihoonson commented Jul 23, 2026 •

edited

Loading

greptile-apps Bot commented Jul 23, 2026 •

edited

Loading