Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create mapping document from HMS tables to UC tables. #304

Closed
FastLee opened this issue Sep 27, 2023 · 6 comments
Closed

Create mapping document from HMS tables to UC tables. #304

FastLee opened this issue Sep 27, 2023 · 6 comments
Assignees
Labels
enhancement New feature or request step/assessment go/uc/upgrade - Assessment Step

Comments

@FastLee
Copy link
Contributor

FastLee commented Sep 27, 2023

Create a "databases" table as part of the assessment.
This table will be used to track database migrations and be used to configure database targets for migration.

Column Name Type Description Default Value
database string Original schema of the table
upgrade_assessment int 0-Manual
1-In Place
2-CTAS
3-Mixed
target_catalog string Name of the Target Catalog ucx_<workspace_id>
target_datbase string Name of the Target Database original database name
workspace_id string A workspace ID of the HMS (Optional)
views int 0-No Views
1-Views present
upgrade_status int 0-Not Upgraded
1-Failed Upgrade
2-Partial Upgrade
3-Full Upgrade
0
upgrade_messages string Json with a list of all the upgrade errors. empty
@nfx
Copy link
Collaborator

nfx commented Sep 27, 2023

@FastLee this configuration cannot be a table, because we don't have a way to edit/add/validate it from UI. But I see the important points on some of the logic. let's discuss them -

  1. what creates upgrade_assessment?
  2. what defines the target_catalog? what is the source? which process?
  3. why do we need target_datbase? why can't it be the original database name?
  4. why do we need upgrade_status and upgrade_messages?

@FastLee
Copy link
Contributor Author

FastLee commented Sep 27, 2023

  1. The assessment process
  2. It defaults to a default catalog for the migration, manually overriden.
  3. If we want to redirect
  4. To report back from the database migration step. And rerun as needed.

@FastLee
Copy link
Contributor Author

FastLee commented Sep 27, 2023

Let's consider a notebook with UI to allow admins to update this table.

@nfx
Copy link
Collaborator

nfx commented Sep 27, 2023

Let's close this issue in favor of resolving the #307.

@pohlposition pohlposition added the step/assessment go/uc/upgrade - Assessment Step label Sep 28, 2023
@nfx nfx added this to UCX Oct 3, 2023
@nfx nfx moved this to Triage in UCX Oct 3, 2023
@FastLee
Copy link
Contributor Author

FastLee commented Oct 19, 2023

I'm reopening this discussion. As we addressed the process in #307 we need to address this issue to make progress on table migrations.

@FastLee
Copy link
Contributor Author

FastLee commented Oct 19, 2023

I would suggest a following.
We create a default mapping for the HMS.
By default all the databases and tables are mapped to a single catalog.
You can override the mapping at the HMS level/Database/Table.
We should annotate just the overrides.
The Structure Should be something like that:

Element Target Notes
<Root> UCX_Dev Default mapping
Sales Sales_Dev.Sales Mapping to a different catalog
Production_Yield Production.Yield Overriding the catalog and the database name
Production_Yield.YOY_Yield Production.Yield.YOY Overriding the catalog database and table name
Playground <Skip> Marking a resource to be skipped

The installer will create a configuration file with a single entry for the workspace. We will create a notebook that will provide a simple interface to override entries.

@FastLee FastLee changed the title Create "databases" tables as part of the assessment Create mapping document from HMS tables to UC tables. Oct 19, 2023
@nfx nfx added the enhancement New feature or request label Nov 7, 2023
@nfx nfx moved this from Triage to Active Backlog in UCX Dec 6, 2023
@nfx nfx self-assigned this Dec 6, 2023
@nfx nfx closed this as completed Dec 15, 2023
@github-project-automation github-project-automation bot moved this from Active Backlog to Archive in UCX Dec 15, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request step/assessment go/uc/upgrade - Assessment Step
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

3 participants