Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding documentation for common challenges and solutions #1940

Merged
merged 3 commits into from
Jul 10, 2024
Merged

Adding documentation for common challenges and solutions #1940

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
42cfb34
Adding documentation for common challenges and solutions
mohanab-db Jun 26, 2024
d1dee4d
Adding documentation for common challenges and solutions
mohanab-db Jul 9, 2024
109a6a7
Adding documentation for common challenges and solutions
mohanab-db Jul 10, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
219 changes: 219 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,14 @@ See [contributing instructions](CONTRIBUTING.md) to help improve this project.
* [`validate-groups-membership` command](#validate-groups-membership-command)
* [`cluster-remap` command](#cluster-remap-command)
* [`revert-cluster-remap` command](#revert-cluster-remap-command)
* [Common Challenges and the Solutions](#common-challenges-and-the-solutions)
* [Network Connectivity Issues](#network-connectivity-issues)
* [Insufficient Privileges](#insufficient-privileges)
* [Version Issues](#version-issues)
* [Multiple Profiles in Databricks CLI](#multiple-profiles-in-databricks-cli)
* [Authentication Issues](#authentication-issues)
* [Workspace has an external Hive Metastore (HMS)](#workspace-has-an-external-hive-metastore-hms)
* [Verify the Installation](#verify-the-installation)
* [Star History](#star-history)
* [Project Support](#project-support)
<!-- TOC -->
Expand Down Expand Up @@ -1301,6 +1309,217 @@ By default, it will revert all the clusters present in the backup folder

[[back to top](#databricks-labs-ucx)]

# Common Challenges and the Solutions
Users might encounter some challenges while installing and executing UCX. Please find the listing of some common challenges and the solutions below.
nfx marked this conversation as resolved.
Show resolved Hide resolved

[[back to top](#databricks-labs-ucx)]

### Network Connectivity Issues

**From local machine to the Databricks Account and Workspace:** UCX
installation process has to be run from the local laptop using
Databricks CLI and it will deploy the latest version of UCX into the
Databricks workspace. For this reason, the Databricks account and
workspace needs to be accessible from the laptop. Sometimes, the
workspace might have a network isolation, like it can only be reached
from a VPC, or from a specific IP range.

**Solution:** Please check that your laptop has network connectivity to
the Databricks account and workspace. If not, you might need to be
connected to a VPN to access your workspace.

**From local machine to GitHub:** UCX needs internet access to connect
to [<u>github.com</u>](https://github.com) (to download the tool) from
the machine running the installation. The installation will fail if
there is no internet connectivity.

**Solution:** Ensure that GitHub is reachable from the local machine. If
not, make necessary changes to the network/firewall settings.

**From Databricks workspace to PyPi:** There are some dependent
libraries which need to be installed from
[<u>pypi.org</u>](https://pypi.org/) to run the UCX workflows from the
Databricks workspace. If the workspace doesn’t have network
connectivity, then the job might fail with
NO_MATCHING_DISTRIBUTION_ERROR.

**Solution:** Version 0.24.0 of UCX supports workspace with no internet
access. Please upgrade UCX and rerun the installation. Reply *yes* to
the question “Does the given workspace block Internet access?” asked
during installation. It will then upload all necessary dependencies to
the workspace. Also, please note that UCX uses both UC and non-UC
enabled clusters. If you have different proxy settings for each, then
please update the necessary proxies (eg. with init scripts) for each
cluster type.

[[back to top](#databricks-labs-ucx)]

### Insufficient Privileges

**User is not a Databricks workspace administrator:** User running the
installation needs to be a workspace administrator as the CLI will
deploy the UCX tool into the workspace, create jobs, and dashboards.

**Solution:** Identify a workspace admin from your team and ask them to
install UCX with their authentication, or request a workspace
administrator to grant you temporary administrator privileges to run the
installation. More details on the issues that you can run into if
you are not an admin (and some possible solutions) can be found
[here](docs/troubleshooting.md#resolving-common-errors-on-ucx-install).

**User is not a Cloud IAM Administrator:** Cloud CLI needs to be
installed in the local machine for certain cloud related activities,
like creating an [<u>uber
principal</u>](#create-uber-principal-command).
For this, the user needs Cloud IAM Administrator privileges.

**Solution:** Work with a cloud administrator in your organization to
run the commands that need cloud administrator rights.

Admin privileges required for commands:
nfx marked this conversation as resolved.
Show resolved Hide resolved

| **CLI command** | **Admin privileges** |
|----|----|
| [<u>install</u>](#install-ucx) | Workspace Admin |
| [<u>account install</u>](#advanced-installing-ucx-on-all-workspaces-within-a-databricks-account) | Account Admin |
| [<u>create-account-groups</u>](#create-account-groups-command) | Account Admin |
| [<u>validate-groups-membership</u>](#validate-groups-membership-command) | Account Admin |
| [<u>create-uber-principal</u>](#create-uber-principal-command) | Cloud Admin |
| [<u>principal-prefix-access</u>](#principal-prefix-access-command) | Cloud Admin |
| [<u>create-missing-principals</u>](#create-missing-pricipals-command-aws-only) | Cloud Admin |
| [<u>migrate-credentials</u>](#migrate-credentials-command) | Cloud Admin, Account Admin / Metastore Admin / CREATE STORAGE CREDENTIAL privilege |
| [<u>migrate-location</u>](#migrate-locations-command) | Metastore Admin / CREATE EXTERNAL LOCATION privilege |
| [<u>create-catalogs-schemas</u>](#create-catalogs-schemas-command) | Metastore Admin / CREATE CATALOG privilege |
| [<u>sync-workspace-info</u>](#sync-workspace-info-command) | Account Admin |
| [<u>manual-workspace-info</u>](#manual-workspace-info-command) | Workspace Admin |

[[back to top](#databricks-labs-ucx)]

### Version Issues

**Python:** UCX needs Python version 3.10 or later.

**Solution:** Check the current version using “python --version”. If the
version is lower than 3.10, upgrade the local Python version to 3.10 or
higher.

**Databricks CLI:** Databricks CLI v0.213 or higher is needed.

**Solution:** Check the current version with “databricks --version”. For
lower versions of CLI,
[<u>update</u>](https://docs.databricks.com/en/dev-tools/cli/install.html#update)
the Databricks CLI on the local machine.

**UCX:** When you install UCX, you get the latest version. But since UCX
is being actively developed, new versions are released frequently. There
might be issues if you have run the assessment with a much earlier
version, and then trying to run the migration workflows with the latest
UCX version.

**Solution:** Upgrade UCX, and rerun the assessment job before running
the migration workflows. For some reason, if you want to install a
specific version of UCX, you can do it using the command
“databricks labs install ucx@\<version\>”, for example,
“databricks labs install [email protected]”.

[[back to top](#databricks-labs-ucx)]

### Authentication Issues

**Workspace Level:** If you are facing authentication issues while
setting up Databricks CLI, please refer to the
[Cryptic errors on authentication](docs/troubleshooting.md#cryptic-errors-on-authentication)
section to resolve the common errors related to authentication,
profiles, and tokens.

**Account Level:** Not only workspace, but account level authentication
is also needed for installing UCX. If you do not have an account
configured in .databrickscfg, you will get an error message
“.databrickscfg does not contain account profiles; please create one
first”.

**Solution:** To authenticate with a Databricks account, consider using
one of the following authentication types: [<u>OAuth machine-to-machine
(M2M)
authentication</u>](https://docs.databricks.com/en/dev-tools/cli/authentication.html#m2m-auth),
[<u>OAuth user-to-machine (U2M)
authentication</u>](https://docs.databricks.com/en/dev-tools/cli/authentication.html#u2m-auth),
[<u>Basic authentication
(legacy)</u>](https://docs.databricks.com/en/dev-tools/cli/authentication.html#basic-auth).

[[back to top](#databricks-labs-ucx)]

### Multiple Profiles in Databricks CLI

**Workspace Level:** More than one workspace profile can be configured
in the .databrickscfg file. For example, you can have profiles set for
Dev and Prod workspaces. You want to install UCX only for the Prod
workspace.

**Solution:** The Databricks CLI provides an option to select the
[<u>profile</u>](https://docs.databricks.com/en/dev-tools/cli/profiles.html)
using “*--profile \<profile_name\>*” or “*-p \<profile_name\>*”. You can
test that the correct workspace is getting selected by running any
Databricks CLI command. For example, you can run “*databricks clusters
list -p prod*” and check that the Prod clusters are being returned. Once
the profile is verified, you can run UCX install for that specific
profile: “*databricks labs install ucx -p prod*”.

**Account Level:** Multiple account level profiles are set in the
.databrickscfg file.

**Solution:** The installation command “*databricks labs install ucx*”
will provide an option to select one account profile.

[[back to top](#databricks-labs-ucx)]

### Workspace has an external Hive Metastore (HMS)

**External HMS connectivity from UCX clusters:** If the workspace has an
external HMS, the clusters running the UCX jobs need to have specific
configurations to connect to the external HMS. Otherwise, UCX assessment
will not be able to assess the tables on HMS.

**Solution:** Use a cluster policy before installation to set the
required Spark config for connecting to the external HMS, or manually
edit the cluster post-installation to have the correct configurations.
Detailed steps can be found
[<u>here</u>](docs/external_hms_glue.md).

**External HMS connectivity from UCX SQL warehouse:** UCX requires a SQL
warehouse to create tables, run queries, create and refresh dashboards.
If you already have a Pro or Serverless warehouse connected to the
external HMS, you can select the same warehouse for UCX. You will also
be given an option (during installation) to create a new warehouse for
UCX. If you have never used a warehouse before, the new warehouse
created might not have proper configuration set to connect to the
external HMS.

**Solution:** Set Spark configuration for connecting to external HMS in
the Admin Settings of SQL warehouse. This will only be needed if the
admin settings do not have the configurations already set. For example,
add *spark.hadoop.javax.jdo.option.ConnectionURL \<connectionstring\>*
under Data Access Configuration of SQL Warehouse Admin Settings.

[[back to top](#databricks-labs-ucx)]

### Verify the Installation

Once the UCX command “*databricks labs install ucx*” has completed
successfully, the installation can be verified with the following steps:

1. Go to the Databricks Catalog Explorer and check if a new schema for
ucx is available in Hive Metastore with all empty tables.

2. Check that the UCX jobs are visible under Workflows.

3. Run the assessment. This will start the UCX clusters, crawl through
the workspace, and display results in the UCX dashboards. In case of
external HMS, verify from the results that the assessment has
analyzed the external HMS tables.

[[back to top](#databricks-labs-ucx)]

# Star History

[![Star History Chart](https://api.star-history.com/svg?repos=databrickslabs/ucx&type=Date)](https://star-history.com/#databrickslabs/ucx)
Expand Down
Loading