[FEATURE]: Allow customer spark conf and init script option in UCX to configure the workflow clusters

[HariGS-DB]

Is there an existing issue for this?

• I have searched the existing issues

Problem statement

The default installation of UCX creates a workflow with different kinds of job clusters. Customers often have a requirement where the clusters need to access data in storage using spn (for Azure), which is configured in the spark env property of the cluster. In addition, customers also may have an init script which needs to run for each job. These options are currently not available to input in the ucx installation, and customers have to do this manually by editing the workflow definition.
ucx has the option to scan existing policies and uses that, but that is mainly used for scanning external HMS policies.

Proposed Solution

While installing ucx, give the additional option for the customer to set custom spark env and any init script to run in the assessment job. These properties can be set in the UCX cluster policy (or directly in the cluster definition) and are picked by when the job is run. This minimises manual work to make the assessment job run

Additional Context

No response