From e3c05c1796e4f3e9bf8c52d1a2445ec34c79c3aa Mon Sep 17 00:00:00 2001 From: mohanab-db <121183444+mohanab-db@users.noreply.github.com> Date: Wed, 10 Jul 2024 10:08:16 -0500 Subject: [PATCH] Adding documentation for common challenges and solutions (#1940) List of common challenges and their solutions --- README.md | 219 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 219 insertions(+) diff --git a/README.md b/README.md index c96ee92ede..0158018176 100644 --- a/README.md +++ b/README.md @@ -98,6 +98,14 @@ See [contributing instructions](CONTRIBUTING.md) to help improve this project. * [`validate-groups-membership` command](#validate-groups-membership-command) * [`cluster-remap` command](#cluster-remap-command) * [`revert-cluster-remap` command](#revert-cluster-remap-command) +* [Common Challenges and the Solutions](#common-challenges-and-the-solutions) + * [Network Connectivity Issues](#network-connectivity-issues) + * [Insufficient Privileges](#insufficient-privileges) + * [Version Issues](#version-issues) + * [Multiple Profiles in Databricks CLI](#multiple-profiles-in-databricks-cli) + * [Authentication Issues](#authentication-issues) + * [Workspace has an external Hive Metastore (HMS)](#workspace-has-an-external-hive-metastore-hms) + * [Verify the Installation](#verify-the-installation) * [Star History](#star-history) * [Project Support](#project-support) @@ -1323,6 +1331,217 @@ By default, it will revert all the clusters present in the backup folder [[back to top](#databricks-labs-ucx)] +# Common Challenges and the Solutions +Users might encounter some challenges while installing and executing UCX. Please find the listing of some common challenges and the solutions below. + +[[back to top](#databricks-labs-ucx)] + +### Network Connectivity Issues + +**From local machine to the Databricks Account and Workspace:** UCX +installation process has to be run from the local laptop using +Databricks CLI and it will deploy the latest version of UCX into the +Databricks workspace. For this reason, the Databricks account and +workspace needs to be accessible from the laptop. Sometimes, the +workspace might have a network isolation, like it can only be reached +from a VPC, or from a specific IP range. + +**Solution:** Please check that your laptop has network connectivity to +the Databricks account and workspace. If not, you might need to be +connected to a VPN to access your workspace. + +**From local machine to GitHub:** UCX needs internet access to connect +to [github.com](https://github.com) (to download the tool) from +the machine running the installation. The installation will fail if +there is no internet connectivity. + +**Solution:** Ensure that GitHub is reachable from the local machine. If +not, make necessary changes to the network/firewall settings. + +**From Databricks workspace to PyPi:** There are some dependent +libraries which need to be installed from +[pypi.org](https://pypi.org/) to run the UCX workflows from the +Databricks workspace. If the workspace doesn’t have network +connectivity, then the job might fail with +NO_MATCHING_DISTRIBUTION_ERROR. + +**Solution:** Version 0.24.0 of UCX supports workspace with no internet +access. Please upgrade UCX and rerun the installation. Reply *yes* to +the question “Does the given workspace block Internet access?” asked +during installation. It will then upload all necessary dependencies to +the workspace. Also, please note that UCX uses both UC and non-UC +enabled clusters. If you have different proxy settings for each, then +please update the necessary proxies (eg. with init scripts) for each +cluster type. + +[[back to top](#databricks-labs-ucx)] + +### Insufficient Privileges + +**User is not a Databricks workspace administrator:** User running the +installation needs to be a workspace administrator as the CLI will +deploy the UCX tool into the workspace, create jobs, and dashboards. + +**Solution:** Identify a workspace admin from your team and ask them to +install UCX with their authentication, or request a workspace +administrator to grant you temporary administrator privileges to run the +installation. More details on the issues that you can run into if +you are not an admin (and some possible solutions) can be found +[here](docs/troubleshooting.md#resolving-common-errors-on-ucx-install). + +**User is not a Cloud IAM Administrator:** Cloud CLI needs to be +installed in the local machine for certain cloud related activities, +like creating an [uber +principal](#create-uber-principal-command). +For this, the user needs Cloud IAM Administrator privileges. + +**Solution:** Work with a cloud administrator in your organization to +run the commands that need cloud administrator rights. + +Admin privileges required for commands: + +| **CLI command** | **Admin privileges** | +|----|----| +| [install](#install-ucx) | Workspace Admin | +| [account install](#advanced-installing-ucx-on-all-workspaces-within-a-databricks-account) | Account Admin | +| [create-account-groups](#create-account-groups-command) | Account Admin | +| [validate-groups-membership](#validate-groups-membership-command) | Account Admin | +| [create-uber-principal](#create-uber-principal-command) | Cloud Admin | +| [principal-prefix-access](#principal-prefix-access-command) | Cloud Admin | +| [create-missing-principals](#create-missing-pricipals-command-aws-only) | Cloud Admin | +| [migrate-credentials](#migrate-credentials-command) | Cloud Admin, Account Admin / Metastore Admin / CREATE STORAGE CREDENTIAL privilege | +| [migrate-location](#migrate-locations-command) | Metastore Admin / CREATE EXTERNAL LOCATION privilege | +| [create-catalogs-schemas](#create-catalogs-schemas-command) | Metastore Admin / CREATE CATALOG privilege | +| [sync-workspace-info](#sync-workspace-info-command) | Account Admin | +| [manual-workspace-info](#manual-workspace-info-command) | Workspace Admin | + +[[back to top](#databricks-labs-ucx)] + +### Version Issues + +**Python:** UCX needs Python version 3.10 or later. + +**Solution:** Check the current version using “python --version”. If the +version is lower than 3.10, upgrade the local Python version to 3.10 or +higher. + +**Databricks CLI:** Databricks CLI v0.213 or higher is needed. + +**Solution:** Check the current version with “databricks --version”. For +lower versions of CLI, +[update](https://docs.databricks.com/en/dev-tools/cli/install.html#update) +the Databricks CLI on the local machine. + +**UCX:** When you install UCX, you get the latest version. But since UCX +is being actively developed, new versions are released frequently. There +might be issues if you have run the assessment with a much earlier +version, and then trying to run the migration workflows with the latest +UCX version. + +**Solution:** Upgrade UCX, and rerun the assessment job before running +the migration workflows. For some reason, if you want to install a +specific version of UCX, you can do it using the command +“databricks labs install ucx@\”, for example, +“databricks labs install ucx@v0.21.0”. + +[[back to top](#databricks-labs-ucx)] + +### Authentication Issues + +**Workspace Level:** If you are facing authentication issues while +setting up Databricks CLI, please refer to the +[Cryptic errors on authentication](docs/troubleshooting.md#cryptic-errors-on-authentication) +section to resolve the common errors related to authentication, +profiles, and tokens. + +**Account Level:** Not only workspace, but account level authentication +is also needed for installing UCX. If you do not have an account +configured in .databrickscfg, you will get an error message +“.databrickscfg does not contain account profiles; please create one +first”. + +**Solution:** To authenticate with a Databricks account, consider using +one of the following authentication types: [OAuth machine-to-machine +(M2M) +authentication](https://docs.databricks.com/en/dev-tools/cli/authentication.html#m2m-auth), +[OAuth user-to-machine (U2M) +authentication](https://docs.databricks.com/en/dev-tools/cli/authentication.html#u2m-auth), +[Basic authentication +(legacy)](https://docs.databricks.com/en/dev-tools/cli/authentication.html#basic-auth). + +[[back to top](#databricks-labs-ucx)] + +### Multiple Profiles in Databricks CLI + +**Workspace Level:** More than one workspace profile can be configured +in the .databrickscfg file. For example, you can have profiles set for +Dev and Prod workspaces. You want to install UCX only for the Prod +workspace. + +**Solution:** The Databricks CLI provides an option to select the +[profile](https://docs.databricks.com/en/dev-tools/cli/profiles.html) +using “*--profile \*” or “*-p \*”. You can +test that the correct workspace is getting selected by running any +Databricks CLI command. For example, you can run “*databricks clusters +list -p prod*” and check that the Prod clusters are being returned. Once +the profile is verified, you can run UCX install for that specific +profile: “*databricks labs install ucx -p prod*”. + +**Account Level:** Multiple account level profiles are set in the +.databrickscfg file. + +**Solution:** The installation command “*databricks labs install ucx*” +will provide an option to select one account profile. + +[[back to top](#databricks-labs-ucx)] + +### Workspace has an external Hive Metastore (HMS) + +**External HMS connectivity from UCX clusters:** If the workspace has an +external HMS, the clusters running the UCX jobs need to have specific +configurations to connect to the external HMS. Otherwise, UCX assessment +will not be able to assess the tables on HMS. + +**Solution:** Use a cluster policy before installation to set the +required Spark config for connecting to the external HMS, or manually +edit the cluster post-installation to have the correct configurations. +Detailed steps can be found +[here](docs/external_hms_glue.md). + +**External HMS connectivity from UCX SQL warehouse:** UCX requires a SQL +warehouse to create tables, run queries, create and refresh dashboards. +If you already have a Pro or Serverless warehouse connected to the +external HMS, you can select the same warehouse for UCX. You will also +be given an option (during installation) to create a new warehouse for +UCX. If you have never used a warehouse before, the new warehouse +created might not have proper configuration set to connect to the +external HMS. + +**Solution:** Set Spark configuration for connecting to external HMS in +the Admin Settings of SQL warehouse. This will only be needed if the +admin settings do not have the configurations already set. For example, +add *spark.hadoop.javax.jdo.option.ConnectionURL \* +under Data Access Configuration of SQL Warehouse Admin Settings. + +[[back to top](#databricks-labs-ucx)] + +### Verify the Installation + +Once the UCX command “*databricks labs install ucx*” has completed +successfully, the installation can be verified with the following steps: + +1. Go to the Databricks Catalog Explorer and check if a new schema for + ucx is available in Hive Metastore with all empty tables. + +2. Check that the UCX jobs are visible under Workflows. + +3. Run the assessment. This will start the UCX clusters, crawl through + the workspace, and display results in the UCX dashboards. In case of + external HMS, verify from the results that the assessment has + analyzed the external HMS tables. + +[[back to top](#databricks-labs-ucx)] + # Star History [![Star History Chart](https://api.star-history.com/svg?repos=databrickslabs/ucx&type=Date)](https://star-history.com/#databrickslabs/ucx)