Add Spark master shim support[reduced-it][fast-ut] by firestarman · Pull Request #15517 · NVIDIA/cudf-spark

firestarman · 2026-08-04T06:42:35Z

Fixes #14547.

Description

Adds an opt-in sparkMaster profile for Spark 500/master so CI can explicitly track Spark master without depending on a release-profile name.
Extends shim detection, build packaging, and Spark 420-based shim coverage so Spark 500 compiles against Spark 5 API changes.
Adds Spark 500-specific shuffle, Python runner, Parquet unknown annotation, and test shims where Spark 5 changed runtime or compile-time contracts.
Updates Spark 500 integration test expectations for observed Spark 5 behavior, including timestamp overflow messages and exact percentile NaN/Inf ordering tracked by Track Spark 5 exact percentile NaN/Inf ordering differences #15516.
Validation run: SPARK_HOME=<Spark 5.0.0-SNAPSHOT binary> INTEGRATION_TEST_VERSION_OVERRIDE=spark500 ./run_pyspark_from_build.sh --test_oom_injection_mode=never passed with 39734 testcases, 0 failures, and 0 errors.
Validation run: SPARK_HOME=<Spark 5.0.0-SNAPSHOT binary> INTEGRATION_TEST_VERSION_OVERRIDE=spark500 mvn -f scala2.13/pom.xml -PsparkMaster -Dcuda.version=cuda13 clean verify passed.

Checklists

Documentation

Updated for new or modified user-facing features or behaviors
No user-facing change

Testing

Added or modified tests to cover new code paths
Covered by existing tests
(Please provide the names of the existing tests in the PR description.)
Not required

Performance

Tests ran and results are added in the PR description
Issue filed with a link in the PR description
Not required

Signed-off-by: Firestarman <firestarmanllc@gmail.com>

greptile-apps · 2026-08-04T06:42:39Z

Too many files changed for review (226 files, 100 file limit).

Bypass the limit by tagging @greptile-apps to review.

Resolve binary-dedupe conflicts for Spark 5 and keep the latest Spark 500 compatibility fixes on top of main. Signed-off-by: Firestarman <firestarmanllc@gmail.com>

Remove stale conflict markers from the merge resolution and keep Spark 5 shim matching on top of the new unshimming flow. Signed-off-by: Firestarman <firestarmanllc@gmail.com>

Include Spark 3.5.9 in the pre-Spark 5 partitioning shim headers and leave Spark 5 to its dedicated shim sources. Signed-off-by: Firestarman <firestarmanllc@gmail.com>

Resolve the SparkShims split by keeping Spark 4.2 support on the new Spark 4.2 shim files and carrying Spark 5.0 on the same baseline. Signed-off-by: Firestarman <firestarmanllc@gmail.com>

firestarman · 2026-08-06T02:26:55Z

build

firestarman · 2026-08-06T03:37:51Z

ci failuresa are due to missing the latest JNI jar on Artifactory.

firestarman · 2026-08-07T02:33:19Z

build

Keep the latest main changes and restore the Delta deletion-vector scan helper that the integration tests import. Signed-off-by: Firestarman <firestarmanllc@gmail.com>

firestarman · 2026-08-07T06:16:00Z

build

res-life

I found four issues in this change.

res-life · 2026-08-13T03:24:44Z

            'date_trunc("MICROSECOND", a)',
            'date_trunc("invalid", a)'))
-
-_LONG_MIN_TIMESTAMP_MICROS = -9223372036854775808


These two date_trunc Long.MinValue tests cover the two distinct overflow-check paths, including the scalar-timestamp/column-format case that previously caught a shape mismatch. Spark 5 continues to reuse the Spark 4.2 overflow shim, so deleting them leaves that behavior unprotected and is unrelated to the Spark master shim work. Could you restore both tests and the is_spark_420_or_later import?

Restored both date_trunc Long.MinValue tests and the Spark 4.2+ helper import.

res-life · 2026-08-13T03:24:44Z

    <artifactId>rapids-4-spark-parent_2.12</artifactId>
-    <name>NVIDIA cuDF plugin for Apache Spark Root Project</name>
-    <description>The root project of the NVIDIA cuDF plugin for Apache Spark</description>
+    <name>RAPIDS Accelerator for Apache Spark Root Project</name>


This reverts the completed cuDF Spark rename and publishes the former project name. The SCM block below is likewise changed from the canonical NVIDIA/cudf-spark repository back to the legacy NVIDIA/spark-rapids alias, and the same regression appears in scala2.13/pom.xml. Could you restore the cuDF project name, description, and canonical SCM coordinates in both POMs?

Restored the cuDF project metadata and canonical NVIDIA/cudf-spark SCM coordinates in both POMs.

res-life · 2026-08-13T03:24:44Z

  def unregisterShuffle(shuffleId: Int): Boolean = realImpl.unregisterShuffle(shuffleId)

-  def shuffleBlockResolver: ShuffleBlockResolver = realImpl.shuffleBlockResolver
+  def shuffleBlockResolver: ShuffleBlockResolver =


shuffleBlockResolver is accessed repeatedly on the local shuffle-block fetch path, so this now performs getMethod and invoke for every fetched block. The delegated resolver is stable and the real implementation stores it in a lazy val. Could you keep the reflective compatibility lookup but cache its result in a private lazy val?

Updated to keep the reflective compatibility path but cache the resolved ShuffleBlockResolver in a private lazy val.

res-life · 2026-08-13T03:24:44Z

     * Returns the executor-resolved PerfIO S3 enablement. This includes opportunistic
     * enablement when the configuration is unset and the required classes are available.
     */
    public static boolean isS3PerfEnabled() {


This method no longer returns the executor-resolved PerfIO state described by its contract. It independently re-evaluates the raw setting and classpath, and its S3Client/S3CrtAsyncClient probes do not match PerfIO's S3AsyncClient and selected HTTP-backend resolution. The two decisions can therefore disagree: this can select S3InputFile while PerfIO is disabled, causing the read to fail instead of falling back to S3A. Could this delegate to com.nvidia.spark.rapids.PerfIO$.MODULE$.isS3PerfEnabled(), matching the existing GCS accessor?

Updated isS3PerfEnabled to delegate to PerfIO, matching the GCS accessor.

nartal1 · 2026-08-13T23:37:00Z

        ))

 @pytest.mark.parametrize('data_gen', double_n_long_gens + _arith_decimal_gens_no_neg_scale + [DecimalGen(30, 15)], ids=idfn)
+@disable_ansi_mode


Any reason we are disabling the ansi mode here ? If it's to mask the test failures, how are we planning to track it?

Narrowed this so ANSI is disabled only for the double special-value floor/ceil cases; long and decimal coverage now keeps the default ANSI path.

Thanks @firestarman ! So looks like we have to fix the double special-value floor/ceil cases later in Spark-5.0 - apache/spark#57127 ?
I am confused should we let the tests fail and then have it fixed it later? Or how are we planning to track such tasks for Spark-5.0 once we merge this PR.

Signed-off-by: Firestarman <firestarmanllc@gmail.com>

firestarman · 2026-08-14T03:22:16Z

build

Signed-off-by: Firestarman <firestarmanllc@gmail.com>

firestarman · 2026-08-14T06:09:17Z

build

res-life · 2026-08-17T02:45:33Z

        or is_spark_420_or_later()
    ),
    reason="Requires Spark's partial-clustering correctness fix")
+@pytest.mark.xfail(


is there a issue to track? It's better to add a issue link in the "reason", e.g.:
reason="issue=https://xxx: Spark 5 requires KeyedPartitionings in a PartitioningCollection to share keys")

Added a dedicated tracking issue and updated the xfail reason to include #15688.

res-life · 2026-08-17T02:58:15Z

    )

+@pytest.mark.skipif(is_spark_500_or_later(),
+                    reason='Spark 5 exact percentile coverage uses the Spark 500 variant')


Add an issue link in the reason to track this case better.
Then it will be easy to use a script tool to gather blocking issues in skipif and xfail.
I remember Sameer have a script to search reason lines and then extract issues.

Updated the Spark 5 exact percentile skip reason to include #15516.

res-life · 2026-08-17T02:58:57Z

        lambda spark: exact_percentile_reduction(gen_df(spark, data_gen)))

+@pytest.mark.skipif(not is_spark_500_or_later(),
+                    reason='Spark 5 exact percentile result tolerance')


Add an issue link in the reason to track this case better.
Please check all the skipif and xfail to add corresponding issues.

Checked the related Spark 5 exact percentile skip/xfail reasons and routed the shared skip reasons through constants that include #15516.

res-life · 2026-08-17T03:18:03Z


-  def shuffleBlockResolver: ShuffleBlockResolver = realImpl.shuffleBlockResolver
+  private lazy val realShuffleBlockResolver: ShuffleBlockResolver =
+    realImpl.getClass.getMethod("shuffleBlockResolver")


We can introduce a shim to do the different behavior between Spark versions, it's better to remove the reflection, we should avoid using reflection as we can.

Removed the reflection path. The proxy now accesses the stable resolver through the version-independent shuffle delegate interface.

res-life · 2026-08-17T03:27:23Z

  def readFooterBufferFromInputFile(
      inputFile: RapidsInputFile,
      filePath: Path): HostMemoryBuffer = {
+    val fileLen = inputFile.getLength


Ray (Renjie Liu)'s PR #15384 already optimized this function, ParquetFooterUtils invokes PerfIO.readParquetFooterBuffer(...)
Seems this change in here is unnecessary? it's here because of merge conflict?

Reverted this back to PerfIO.readParquetFooterBuffer(...); this change was not needed for Spark 500.

res-life · 2026-08-17T03:40:14Z

And we need to consider the Iceberg and Delta supports.
Update #14547 to add sub tasks: Iceberg and Delta supports,
Or add two follow-up issues.

firestarman · 2026-08-17T05:17:58Z

Updated #14547 with Iceberg and Delta follow-up sub-issues.

Signed-off-by: Firestarman <firestarmanllc@gmail.com>

firestarman · 2026-08-17T06:07:11Z

build

res-life · 2026-08-19T07:15:36Z

~~So change Fixes issue to contributes to issue, because of sub-tasks for iceberg and delta are not done.~~

res-life

LGTM, but please test all the UTs and ITs

firestarman · 2026-08-20T02:00:50Z

LGTM, but please test all the UTs and ITs

Alreayd done, you can find the results in PR description.

NvTimLiu

LGTM from CICD point of view, +1

Add Spark master shim support

fb339ed

Signed-off-by: Firestarman <firestarmanllc@gmail.com>

firestarman requested a review from a team as a code owner August 4, 2026 06:42

firestarman requested a review from a team August 4, 2026 06:56

firestarman added 4 commits August 5, 2026 10:45

Merge main into Spark master shim branch

1141782

Resolve binary-dedupe conflicts for Spark 5 and keep the latest Spark 500 compatibility fixes on top of main. Signed-off-by: Firestarman <firestarmanllc@gmail.com>

Fix binary-dedupe merge resolution

a4e114a

Remove stale conflict markers from the merge resolution and keep Spark 5 shim matching on top of the new unshimming flow. Signed-off-by: Firestarman <firestarmanllc@gmail.com>

Fix Spark 3.5.9 partitioning shims

93365ad

Include Spark 3.5.9 in the pre-Spark 5 partitioning shim headers and leave Spark 5 to its dedicated shim sources. Signed-off-by: Firestarman <firestarmanllc@gmail.com>

Merge main into Spark master shim branch

ae5046d

Resolve the SparkShims split by keeping Spark 4.2 support on the new Spark 4.2 shim files and carrying Spark 5.0 on the same baseline. Signed-off-by: Firestarman <firestarmanllc@gmail.com>

sameerz added the task Work required that improves the product but is not user facing label Aug 7, 2026

Merge main and restore Delta DV helper

e858389

Keep the latest main changes and restore the Delta deletion-vector scan helper that the integration tests import. Signed-off-by: Firestarman <firestarmanllc@gmail.com>

res-life reviewed Aug 13, 2026

View reviewed changes

nartal1 reviewed Aug 13, 2026

View reviewed changes

firestarman added 4 commits August 14, 2026 10:22

Merge main and address review comments

cec62c8

Signed-off-by: Firestarman <firestarmanllc@gmail.com>

Clarify floor ceil ANSI test coverage

2ed56b2

Signed-off-by: Firestarman <firestarmanllc@gmail.com>

Restore Delta deletion vector helper docs

dadac77

Signed-off-by: Firestarman <firestarmanllc@gmail.com>

Narrow Spark 500 test relaxations

f2e7afd

Signed-off-by: Firestarman <firestarmanllc@gmail.com>

firestarman requested review from a team, nartal1 and res-life August 14, 2026 03:17

firestarman changed the title ~~Add Spark master shim support~~ Add Spark master shim support[reduced-it][fast-ut] Aug 14, 2026

Handle Spark 500 integration test failures

75ce009

Signed-off-by: Firestarman <firestarmanllc@gmail.com>

res-life reviewed Aug 17, 2026

View reviewed changes

Address Spark 500 review comments

807e8a9

Signed-off-by: Firestarman <firestarmanllc@gmail.com>

firestarman requested a review from res-life August 17, 2026 05:37

res-life approved these changes Aug 19, 2026

View reviewed changes

firestarman requested a review from NvTimLiu August 20, 2026 06:45

NvTimLiu approved these changes Aug 20, 2026

View reviewed changes

firestarman merged commit 1d8fa55 into NVIDIA:main Aug 20, 2026
54 checks passed

firestarman deleted the issue-14547-spark-master-shim-from420 branch August 20, 2026 08:06

Uh oh!

Conversation

firestarman commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklists

Uh oh!

greptile-apps Bot commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

firestarman commented Aug 6, 2026

Uh oh!

firestarman commented Aug 6, 2026

Uh oh!

firestarman commented Aug 7, 2026

Uh oh!

firestarman commented Aug 7, 2026

Uh oh!

res-life left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

firestarman commented Aug 14, 2026

Uh oh!

firestarman commented Aug 14, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

res-life Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

res-life commented Aug 17, 2026

Uh oh!

firestarman commented Aug 17, 2026

Uh oh!

firestarman commented Aug 17, 2026

Uh oh!

res-life commented Aug 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

res-life left a comment

Choose a reason for hiding this comment

firestarman commented Aug 4, 2026 •

edited

Loading

greptile-apps Bot commented Aug 4, 2026 •

edited

Loading

res-life Aug 17, 2026 •

edited

Loading

res-life commented Aug 19, 2026 •

edited

Loading