[FEATURE]: Capture more details during cluster assessment to provide deeper insights into impact analysis

[dmoore247]

Is there an existing issue for this?

• I have searched the existing issues

Problem statement

The $inventory.clusters table contains just ~6 fields of information while the cluster contains dozens of attributes.
Determining code compatibility depends on knowing more about the current compute cluster. If the code is already running on a UC (shared) compute platform, if is already running as a single user compute or job compute then the code block from the audit log can be dropped from interactive assessment dashboard.

For example:

• data_security_mode will help eliminate false positives for clusters already converted or are of a Assigned type of cluster
• Certain tags indicate the cluster is a job cluster
• Cluster source will help identify clusters created via automation, possible job clusters

Proposed Solution

Adopt a "touch it, take it" approach when scanning clusters to satisfy future refinements to the assessment processes. (e.g. upgrade all clusters tagged with a certain department, group the assessment by a cluster tag to capture a group of workloads by business units)

Retain the existing $inventory.clusters table structure and columns for backwards compatibility.

Add an additional field into $inventory.clusters table, called attributes.
Within this table store the complex json obtained from the clusters api get method.

Ensure the attributes field can be queried using the SQL json notation for ease of use.

Additional Context

No response

[nfx]

@dmoore247 this issue is not concrete enough - it cannot be distributed among people without context. Please split this issue up to a more granular ones and provide extremely detailed input.

[nfx]

@vsevolodstep-db can you take a look at this issue, as it seems to align with your team's priorities

[nfx]

place it in the same shared state that currently tracked database is. Also parse the DATABRICKS_RUNTIME_VERSION env var

[nfx]

No longer necessary