Skip Dataproc shuffle manager auto-configuration

[res-life]

Fixes #15418.

Description

#15285 changed behavior, now auto set shuffle manager. It impacts the behavior of Dataporc pipeline.

Dataproc's Spark 4.0.1 runtime uses a shuffle resolver ABI that differs from the Apache Spark ABI expected by the RAPIDS Shuffle Manager. After shuffle manager auto-configuration was enabled for Spark 4, workloads that did not explicitly configure spark.shuffle.manager began selecting the RAPIDS implementation and failed during shuffle output commit with NoSuchMethodError.

Skip RAPIDS Shuffle Manager auto-configuration when spark.dataproc.engine is present. This restores the previous behavior on Dataproc by leaving spark.shuffle.manager unset, while continuing to preserve any explicitly configured shuffle manager. The configuration documentation is updated to describe this exception.

Added RapidsPluginUtilsSuite coverage for the Dataproc guard.

Validation:

• Spark 4.0.1 / Scala 2.13: RapidsPluginUtilsSuite passed (6 tests).
• Scalastyle passed across 1,701 files with no errors or warnings.

Checklists

Documentation

• Updated for new or modified user-facing features or behaviors
• No user-facing change

Testing

• Added or modified tests to cover new code paths
• Covered by existing tests
  (Please provide the names of the existing tests in the PR description.)
• Not required

Performance

• Tests ran and results are added in the PR description
• Issue filed with a link in the PR description
• Not required

[res-life]

build

[greptile-apps]

Greptile Summary

The PR prevents automatic RAPIDS Shuffle Manager selection on Dataproc runtimes while preserving explicit shuffle-manager settings.

• Adds a spark.dataproc.engine presence guard to driver-side shuffle auto-configuration.
• Documents the Dataproc exception in generated and user-facing configuration descriptions.
• Adds unit coverage confirming that Dataproc configurations leave spark.shuffle.manager unset.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
sql-plugin/src/main/scala/com/nvidia/spark/rapids/Plugin.scala	Adds the intended presence-based Dataproc guard before automatic shuffle-manager selection.
sql-plugin/src/main/scala/com/nvidia/spark/rapids/RapidsConf.scala	Updates the configuration description to explain that Dataproc runtimes are excluded from automatic shuffle-manager setup.
sql-plugin/src/test/scala/com/nvidia/spark/rapids/RapidsPluginUtilsSuite.scala	Adds focused coverage verifying that a Dataproc marker prevents shuffle-manager auto-configuration.
docs/additional-functionality/advanced_configs.md	Documents the Dataproc auto-configuration exception consistently with the runtime behavior.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Driver plugin initialization] --> B{spark.dataproc.engine present?}
  B -->|Yes| C[Leave shuffle manager unchanged]
  B -->|No| D{Shuffle manager explicitly configured?}
  D -->|Yes| C
  D -->|No| E[Apply supported RAPIDS shuffle auto-configuration]

Loading

_{Reviews (2): Last reviewed commit: "Clarify Dataproc shuffle auto-configurat..." | Re-trigger Greptile}

[wjxiz1992]

One non-blocking documentation precision nit.

[firestarman]

lgtm

[res-life]

build