[HUDI-3994] - Added support for initializing DeltaStreamer without a defined Spark Master

[Neuw84]

Tips

• Thank you very much for contributing to Apache Hudi.
• Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.

What is the purpose of the pull request

Added support for initializing DeltaStreamer without a defined Spark Master. That will enable the usage of DeltaStreamer on environments such as AWS Glue or other serverless environments where the spark master is inherited and we do not have access to it.

Brief change log

• Modify HoodieDeltaStreamer class in order to have an option to inherit Spark Master.
• Modify UtilHelpers class to have an option to start the Spark Context without defined Master.
• Modify HoodieMultiTableDeltaStreamer to have the same default Spark Master( although right now is not used).

Verify this pull request

(Please pick either of the following options)

This pull request is a trivial rework.

Committer checklist

• Has a corresponding JIRA in PR title & commit

• Commit message is descriptive of the change

• CI is green

• Necessary doc changes done or have another open PR

[Neuw84]

@hudi-bot run azure

[pratyakshsharma]

One high level comment, will it be possible to add test cases for this change?

[Neuw84]

Hi @pratyakshsharma,

I could try to add a test for initialising the Spark Context. However, the change is very simple and looking for the tests I did not see any that test that tests the other constructors.

The only thing is that now, if no Spark master is not defined it will inherit or the application will fail as no Spark context is defined. Previously it would run with local[2] as Spark Master.

What do you have in mind? (looking for some guidance).

[pratyakshsharma]

I am not having anything as of now. Let me check and get back to you. In the mean time, please check CI failures.

[Neuw84]

Hi,

Should I rebase it from latest stable version? Almost every time I merge somethings breaks ( and it´s not because my changes).

Thanks!

[pratyakshsharma]

To be able to merge, you need to rebase it on master itself.

[hudi-bot]

CI report:

• 0ed42d8 Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build