[BUG]: Upgrade to (0.10.0) failed with 'installing components task failed'

[dmoore247]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

Failed to upgrade to version 0.10.0:

17:05:58 ERROR [d.l.blueprint.parallel][installing_components_2] installing components task failed: [UNRESOLVED_COLUMN.WITH_SUGGESTION] A column, variable, or function parameter with name `udf` cannot be resolved. Did you mean one of the following? [`view`, `table`, `any_file`, `catalog`, `database`].; line 9 pos 13: Traceback (most recent call last):
  File "/Users/first.last/.databricks/labs/ucx/state/venv/lib/python3.10/site-packages/databricks/labs/blueprint/parallel.py", line 134, in inner
    return func(*args, **kwargs), None

Expected Behavior

New wheel file installed, new workflows, dashboards, readme installed; Not an exception traceback

Steps To Reproduce

Using current ucx development console window on Mac OS, run command:

databricks labs upgrade ucx

Cloud

Azure

Operating System

macOS

Version

latest via Databricks CLI

Relevant log output

% databricks labs upgrade ucx --profile XXXXXXXX
17:05:42  INFO [d.l.ucx.install] Installing UCX v0.10.0
17:05:43  INFO [d.l.ucx.install] UCX is already configured. See https://adb-9999999999999999.14.azuredatabricks.net/#workspace/Users/[email protected]/.ucx/config.yml
17:05:43  INFO [d.l.ucx.install][installing_components_0] Enabling HMS Lineage: HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data.
17:05:43  INFO [d.l.ucx.install][installing_components_0] Checking if Global Init Script with Required Spark Config already exists and enabled.
17:05:43  INFO [d.l.ucx.install][installing_components_1] Creating dashboards...
17:05:43  INFO [d.l.ucx.install][installing_components_1] Fetching warehouse_id from a config
17:05:43  INFO [d.l.ucx.install][installing_components_0] Global Init Script already exists and enabled. Skipped creating a new one.
17:05:58 ERROR [d.l.blueprint.parallel][installing_components_2] installing components task failed: [UNRESOLVED_COLUMN.WITH_SUGGESTION] A column, variable, or function parameter with name `udf` cannot be resolved. Did you mean one of the following? [`view`, `table`, `any_file`, `catalog`, `database`].; line 9 pos 13: Traceback (most recent call last):
  File "/Users/first.last/.databricks/labs/ucx/state/venv/lib/python3.10/site-packages/databricks/labs/blueprint/parallel.py", line 134, in inner
    return func(*args, **kwargs), None
  File "/Users/first.last/.databricks/labs/ucx/lib/src/databricks/labs/ucx/install.py", line 220, in _create_database
    deploy_schema(self._sql_backend, self.current_config.inventory_database)
  File "/Users/first.last/.databricks/labs/ucx/lib/src/databricks/labs/ucx/install.py", line 163, in deploy_schema
    deployer.deploy_view("grant_detail", "queries/views/grant_detail.sql")
  File "/Users/first.last/.databricks/labs/ucx/lib/src/databricks/labs/ucx/framework/crawlers.py", line 325, in deploy_view
    self._sql_backend.execute(ddl)
  File "/Users/first.last/.databricks/labs/ucx/lib/src/databricks/labs/ucx/framework/crawlers.py", line 103, in execute
    self._sql.execute(self._warehouse_id, sql)
  File "/Users/first.last/.databricks/labs/ucx/lib/src/databricks/labs/ucx/mixins/sql.py", line 173, in execute
    self._raise_if_needed(status)
  File "/Users/first.last/.databricks/labs/ucx/lib/src/databricks/labs/ucx/mixins/sql.py", line 133, in _raise_if_needed
    raise error_class(error_message)
databricks.sdk.errors.mapping.BadRequest: [UNRESOLVED_COLUMN.WITH_SUGGESTION] A column, variable, or function parameter with name `udf` cannot be resolved. Did you mean one of the following? [`view`, `table`, `any_file`, `catalog`, `database`].; line 9 pos 13
17:05:59 ERROR [d.l.blueprint.parallel] More than half 'installing components' tasks failed: 0% results available (0/3). Took 0:00:16.732661
Traceback (most recent call last):
  File "/Users/first.last/.databricks/labs/ucx/lib/src/databricks/labs/ucx/install.py", line 1047, in <module>
    installer.run()
  File "/Users/first.last/.databricks/labs/ucx/lib/src/databricks/labs/ucx/install.py", line 201, in run
    self._run_configured()
  File "/Users/first.last/.databricks/labs/ucx/lib/src/databricks/labs/ucx/install.py", line 204, in _run_configured
    Threads.strict(
  File "/Users/first.last/.databricks/labs/ucx/state/venv/lib/python3.10/site-packages/databricks/labs/blueprint/parallel.py", line 56, in strict
    raise ManyError(errs)
databricks.labs.blueprint.parallel.ManyError: Detected 1 failures: BadRequest: [UNRESOLVED_COLUMN.WITH_SUGGESTION] A column, variable, or function parameter with name `udf` cannot be resolved. Did you mean one of the following? [`view`, `table`, `any_file`, `catalog`, `database`].; line 9 pos 13
Error: installer: exit status 1

[dmoore247]

databricks labs uninstall ucx
databricks labs install ucx

worked as expected.

[nfx]

Until #471 is implemented, the solution is to

databricks labs uninstall ucx
databricks labs install ucx