Define table migration process #307

FastLee · 2023-09-27T16:22:12Z

Added a table migration doc.
Let's discuss the migration process.

docs/table-upgrade.md

nfx · 2023-09-27T16:57:22Z

docs/table-upgrade.md

+1. The assessment generates "databases" tables or an outline for a configuration file.
+1. It can be used by the user to configure the database upgrade.
+
+This is the structure we recommend:


Assume we don't have this table, what happens?

We create it.

nfx · 2023-09-27T16:58:35Z

docs/table-upgrade.md

+| upgrade_status | int | 0-Not Upgraded <br/> 1-Failed Upgrade <br/> 2-Partial Upgrade <br/> 3-Full Upgrade | 0 |
+| upgrade_messages | string | Json with a list of all the upgrade errors. | empty |
+
+1. We can use a notebook with IPYWidgets to update the table (or configuration file).


Why not a job? What cluster configuration does this job run on? What about thousands of workspaces?

Agree, this should be a job - and it should operate on multiple tables

I'm looking for the interactive step.
What UI do we provide the user to update the table/manifest

i'm still challenging if we need a UI. How can we do it without the UI at all and have it fully automated?

I suggested we have a fast-track option to have it fully automated that uses sensible defaults.

In the event a user wants to modify the mapping, we can generate an output file for the user to review and modify in lieu of a UI, csv, tsv, excel, etc. User would make any custom mapping overrides in this file.

This file could be uploaded back to the workspace via the tool, and the migration could proceed.

docs/table-upgrade.md

nfx

nice work! could you answer the question I've left in this and the previous reviews?

docs/table-upgrade.md

nfx · 2023-09-28T13:06:58Z

docs/table-upgrade.md

+
+We don't expect this process to be a "one and done" process. This typically is an iterative process and may require a few runs.
+
+We suggest to keep track of the migration and provide the user a continuous feedback of the progress and status of the upgrade.


please elaborate how this continuous feedback should be delivered to the customer, if we're migrating 300 or 3000 workspaces.

docs/table-upgrade.md

nfx · 2023-09-28T13:08:27Z

docs/table-upgrade.md

+1. The assessment runs and capture all the tables in HMS (Done).
+   1. Each table is categorized based on the type and storage
+   1. We use the "upgraded_to" table property to determine if the table was already upgraded to UC
+1. The assessment generates a list of "recommended External Locations". These locations are used by the external tables and are required for an "in place" upgrade.


what is done with that list? Please elaborate

nfx · 2023-09-28T13:10:36Z

docs/table-upgrade.md

+| upgrade_status | int | 0-Not Upgraded <br/> 1-Failed Upgrade <br/> 2-Partial Upgrade <br/> 3-Full Upgrade | 0 |
+| upgrade_messages | string | Json with a list of all the upgrade errors. | empty |
+
+1. We can use a notebook with IPYWidgets to update the table (or configuration file).


i'm still challenging if we need a UI. How can we do it without the UI at all and have it fully automated?

nfx

Now for tables, there also needs to be a report on table/db inconsistency - like
A: db1.tbl1, db1.tbl3
B: db1.tbl2

And the team(s) that are driving UC Migration within account would make a decision after some time in review (of excel spreadsheet). By the way, we can split UCX installation across different Azure Subscriptions. And every installation would just focus on defining target catalog mapping per database. But here are unanswered questions:

two workspaces, same dbs, all different tables and columns (all managed tables, effectively)
two workspaces, same dbs, 90% same tables, 10% are different tables
two workspaces, two different dbs
We can technically support both db_to_catalog and workspace_to_catalog, and even at the same time, but db_to_catalog will override workspace_to_catalog. We also need default_catalog_for_workspace, if workspace_to_catalog is set (default catalog for all workspaces is set per metastore)..

We can also do another override for tables, but we have unanswered questions:

what if same db, same workspace, same table, but different columns/order/types? Ignore and keep in hive metastore? And then rerun the scan for tables and grants?
what if during migration catalog/database/table were deleted either from hms and/or uc?
Speaking of metastores, in the beginning, there needs to be workspace_to_metastore mapping with default_metastore_for_workspace. Can we come up with a good default mapping here? Coarse or fine grained? Select between the two? Ask for inline input? How many conflicts we expect to justify the need to create/support custom mapping?

the last very important question is what future-proof configuration format might we need for this mapping. That's why I don't want any configuration after the assessment step.

codecov · 2023-09-28T21:02:59Z

Codecov Report

Merging #307 (7759580) into main (035320c) will increase coverage by 0.11%.
The diff coverage is 87.71%.

❗ Current head 7759580 differs from pull request most recent head e75e4c9. Consider uploading reports for the commit e75e4c9 to get more accurate results

@@            Coverage Diff             @@
##             main     #307      +/-   ##
==========================================
+ Coverage   83.47%   83.58%   +0.11%     
==========================================
  Files          30       30              
  Lines        2269     2254      -15     
  Branches      395      394       -1     
==========================================
- Hits         1894     1884      -10     
+ Misses        290      285       -5     
  Partials       85       85

Files	Coverage Δ
src/databricks/labs/ucx/__about__.py	`100.00% <100.00%> (ø)`
src/databricks/labs/ucx/config.py	`86.50% <100.00%> (ø)`
src/databricks/labs/ucx/install.py	`80.82% <ø> (ø)`
src/databricks/labs/ucx/hive_metastore/tables.py	`94.30% <90.56%> (ø)`
src/databricks/labs/ucx/assessment/crawlers.py	`73.29% <84.48%> (-0.22%)`	⬇️

... and 3 files with indirect coverage changes

nfx · 2023-09-29T14:11:41Z

docs/table-upgrade.md

+   |finance| de_dev | finance |
+   |hr | de_dev | human_resources|
+   |sales | ucx-dev_ws | sales |
+1. We need to provide a UI or other option to override these values.


we don't have capacity to build and maintain UI. How can we express these overrides as YAML configuration? How would command-line interface could look for it? please specify the sample format, try to be very explicit with corner-cases.

pohlposition · 2023-10-02T18:55:44Z

docs/table-upgrade.md

+1. We should migrate ACLs for the tables (where applicable). We should highlight cases where we can't (no direct translation/conflicts)
+1. We should consider automating ACLs based on Instance Profiles / Service Principals and other legacy security mechanisms
+
+## Tables (Parquet/Delta) on DBFS root


Can we keep Managed Tables separate from External Tables. This is how it maps to our upgrade steps.

External tables are easier and lower risk. We should be able to deliver on those faster.

nfx

Please address other comments and properly rebase the branch, so that it has changes only to docs/table-upgrade.md.

Added a table migration doc. Let's discuss the migration process.

FastLee requested a review from william-conti September 27, 2023 16:22

FastLee requested review from nfx and larsgeorge-db as code owners September 27, 2023 16:22

nfx requested changes Sep 27, 2023

View reviewed changes

nfx added the to be discussed label Sep 27, 2023

nfx changed the title ~~Created a table-migration.md~~ Define table migration process Sep 27, 2023

nfx reviewed Sep 27, 2023

View reviewed changes

docs/table-upgrade.md Outdated Show resolved Hide resolved

docs/table-upgrade.md Outdated Show resolved Hide resolved

docs/table-upgrade.md Outdated Show resolved Hide resolved

nfx self-requested a review September 27, 2023 17:28

This was referenced Sep 27, 2023

Create mapping document from HMS tables to UC tables. #304

Closed

If a view uses a function, we need to expose whether or not that function CAN be migrated #315

Closed

If Grants are used, we should migrate them. #316

Closed

william-conti reviewed Sep 28, 2023

View reviewed changes

docs/table-upgrade.md Outdated Show resolved Hide resolved

nfx reviewed Sep 28, 2023

View reviewed changes

nfx requested changes Sep 29, 2023

View reviewed changes

pohlposition mentioned this pull request Oct 2, 2023

Mapping Flow from Old [HMS] to new [UC] For External Tables #354

Closed

nfx added the feat/migration-index mapping of databases to catalog or potentially other databases label Oct 2, 2023

nfx added this to the 1 week milestone Oct 2, 2023

zpappa removed this from the 1 week milestone Oct 2, 2023

pohlposition reviewed Oct 2, 2023

View reviewed changes

nfx requested changes Oct 3, 2023

View reviewed changes

Updated per Serge/Zeashan comments

e75e4c9

FastLee force-pushed the feature/table_upgrade_doc branch from 7759580 to e75e4c9 Compare October 3, 2023 14:17

FastLee requested a review from nfx October 3, 2023 14:18

nfx approved these changes Oct 3, 2023

View reviewed changes

nfx added this pull request to the merge queue Oct 3, 2023

Merged via the queue into main with commit 10abd0c Oct 3, 2023

zpappa pushed a commit that referenced this pull request Oct 4, 2023

Define table migration process (#307)

9b21fab

Added a table migration doc. Let's discuss the migration process.

nfx deleted the feature/table_upgrade_doc branch October 17, 2023 22:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Define table migration process #307

Define table migration process #307

FastLee commented Sep 27, 2023

nfx Sep 27, 2023

FastLee Sep 28, 2023

nfx Sep 27, 2023

zpappa Sep 27, 2023

FastLee Sep 28, 2023

nfx Sep 28, 2023

zpappa Oct 1, 2023

nfx left a comment

nfx Sep 28, 2023

nfx Sep 28, 2023

nfx Sep 28, 2023

nfx left a comment

codecov bot commented Sep 28, 2023 •

edited

Loading

nfx Sep 29, 2023

pohlposition Oct 2, 2023

nfx left a comment


		We don't expect this process to be a "one and done" process. This typically is an iterative process and may require a few runs.

		We suggest to keep track of the migration and provide the user a continuous feedback of the progress and status of the upgrade.

Define table migration process #307

Define table migration process #307

Conversation

FastLee commented Sep 27, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nfx left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nfx left a comment

Choose a reason for hiding this comment

codecov bot commented Sep 28, 2023 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nfx left a comment

Choose a reason for hiding this comment

codecov bot commented Sep 28, 2023 •

edited

Loading