[HUDI-5632] Fix failure launching Spark jobs from hudi-cli-bundle

[yihua]

Change Logs

The Hudi CLI commands which require launching Spark cannot be executed in Hudi CLI shell with hudi-cli-bundle:

savepoint create --commit <latest-commit-timestamp> --sparkMaster local
savepoint delete --commit <latest-commit-timestamp> --sparkMaster local
savepoint create --commit <latest-commit-timestamp> --sparkMaster local
downgrade table --toVersion 3 --sparkMaster local
upgrade table --toVersion 5 --sparkMaster local
compaction schedule --hoodieConfigs hoodie.compact.inline.max.delta.commits=1

Sample error message:

30977 [Thread-4] INFO  org.apache.hudi.cli.utils.InputStreamConsumer [] - Error: Failed to load org.apache.hudi.cli.commands.SparkMain: org/apache/hudi/common/engine/HoodieEngineContext

The root cause is that the hudi-cli-bundle excludes the classes already in hudi-spark*-bundle, such as in hudi-common module, and the hudi-spark*-bundle is not added to the Spark launcher, so that the Spark job fails due to class not found.

This PR fixes the problem by adding the hudi-spark*-bundle specified by env variable SPARK_BUNDLE_JAR to the Spark launcher. Note that SPARK_BUNDLE_JAR is required when using hudi-cli-bundle.

Impact

Ensures that Hudi CLI commands which require launching Spark can be executed with hudi-cli-bundle. The above CLI commands are tested to be working locally with this fix.

Risk level

low

Documentation Update

N/A

Contributor's checklist

• Read through contributor's guide
• Change Logs and Impact were stated clearly
• Adequate tests were added if applicable
• CI passed

[yihua]

@rahil-c could you review this?

[hudi-bot]

CI report:

• 8e01aae Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build

[rahil-c]

Thanks for helping with this @yihua