Assessment for RunSubmit API usages

[renardeinside]

Problem statement

Existing processes that create RunSubmit Jobs (the so-called ephemeral jobs) with the following properties:

• DBR > 11.X
• [AWS-specific] Uses instance_profile
• Has no data_security_mode property specified in the job creation request
• [Azure-specific] some configurations in the spark_conf (To be confirmed)

May be broken when UC is enabled in a given workspace (meaning when workspace is assigned to the metastore).

Why this happens?

When UC is enabled, all RunSubmit Jobs with DBR 11.X+ will by default UC-enabled. If there is a conflict in permissions between UC and service principal (e.g. instance profile), the job will fail.

The reason and the change is described here.

How can we identify the identical RunSubmit?

This requires additional internal discussion.

TODO:

• get a list of all (persisted) jobs
• get a list of all job runs
• find job runs that have no persisted job (out of workflow)
• group all job runs from non-persisted jobs to approximate the number of unique airflow/azure data factory DAGs.
• identify job runs that do not include data_security_mode in the job creation request and are run against 11.x compute

[zpappa]

@nfx let's prioritize this one for next week

[nfx]

@tamilselvanveeramani Is assigned (working) on it, as you see.

Right, @tamilselvanveeramani ?

[nfx]

@tamilselvanveeramani could not make any progress on this one, up for the next pickup