[BUG]: Is UCX catalog now a pre-requisite for the table migration workflow to run?

[JCZuurmond]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

In #2929, the UCX catalog became a prequisite for the migrate-tables command. In #3239, it becomes a prerequisite for the table migration workflows, and the migration progress is tracked during the workflow too.

Expected Behavior

• Consistent prerequiste: both for the command and workflow. Or neither.
• If prerequisite, document it as such.

Steps To Reproduce

N.A.

Cloud

Azure

Operating System

macOS

Version

latest via Databricks CLI

Relevant log output

No response

[asnare]

The ucx catalog is where progress (across multiple installations/workspaces) is tracked for displaying in the progress-related dashboards. From a technical perspective it's intentional that the migration-related pieces therefore depend on it so that they can log their progress for it to be reported. From a technical point-of-view it would therefore be consistent for both the command and workflow to require it.

Stepping back a bit, the question in my mind is where should the ucx catalog be created. As far as I can tell:

• It only needs to be created once, even though multiple UCX installations might be using (sharing) it.
• The way to create it is with the create-ucx-catalog command, but this isn't documented nor is when it should it happen.

With this in mind, I think the following is needed:

• The create-ucx-catalog command should be documented.
• Arguably during attach-metastore we should also attempt to create it if it's not already present. (This needs to be documented.) If we ever need to upgrade this is also an obvious point to check for this. In the alternative, if we decide not to create the ucx catalog automatically here then if the catalog doesn't exist we should at least log a reminder that it needs to be created via create-ucx-catalog.
• If the user attaches the metastore themselves, it needs to be clear from the docs that the create-ucx-catalog is still needed.

[JCZuurmond]

Thanks for the good write up.

IMO we should make the UCX catalog required before the migrate tables workflow.

The way to create it is with the create-ucx-catalog command, but this isn't documented nor is when it should it happen.
The create-ucx-catalog command should be documented.

Already is document: https://github.com/databrickslabs/ucx?tab=readme-ov-file#create-ucx-catalog-command.
But not the when, I think it should be part of https://github.com/databrickslabs/ucx?tab=readme-ov-file#new-unity-catalog-resources

Arguably during attach-metastore

I like that, it fits also with when the UCX catalog should be created, i.e. right after attach metastore. At least a reminder should be logged.

If the user attaches the metastore themselves, it needs to be clear from the docs that the create-ucx-catalog is still needed.

Indeed, I think it should be part of https://github.com/databrickslabs/ucx?tab=readme-ov-file#new-unity-catalog-resources

[asnare]

Already is document: https://github.com/databrickslabs/ucx?tab=readme-ov-file#create-ucx-catalog-command. But not the when, I think it should be part of https://github.com/databrickslabs/ucx?tab=readme-ov-file#new-unity-catalog-resources

Ugh… my bad: I searched on create_ucx_catalog instead of create-ucx-catalog. Thanks for the correction.