Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE]: Add CTAS table migration function #1340

Closed
1 task done
qziyuan opened this issue Apr 9, 2024 · 2 comments · Fixed by #1434
Closed
1 task done

[FEATURE]: Add CTAS table migration function #1340

qziyuan opened this issue Apr 9, 2024 · 2 comments · Fixed by #1434
Assignees
Labels
migrate/external go/uc/upgrade SYNC EXTERNAL TABLES step

Comments

@qziyuan
Copy link
Contributor

qziyuan commented Apr 9, 2024

Is there an existing issue for this?

  • I have searched the existing issues

Problem statement

Parent issues:

When migrate managed dbfs root non-delta tables, and certain external tables, we need to run CTAS.

Proposed Solution

  • Create a function that can create a UC table using CTAS.
  • The CTAS should preserve the partition columns if any.
  • The function should be able to be used in either managed or external table migration. In case, external table migration, it should be able to create the table in a provided table location. The location will be passed in by the migration function who calls this CTAS function.

Additional Context

No response

@nfx
Copy link
Collaborator

nfx commented Apr 10, 2024

@qziyuan isn't it a duplicate of #332?..

@nfx nfx added migrate/external go/uc/upgrade SYNC EXTERNAL TABLES step and removed enhancement New feature or request needs-triage labels Apr 10, 2024
@qziyuan
Copy link
Contributor Author

qziyuan commented Apr 10, 2024

@qziyuan isn't it a duplicate of #332?..

@nfx I put the #332 as a parent issue. This issue is meant to create a CTAS function that can be reused by both #332 and #889. Once this issue is completed, what left in #332 is to add a workflow task and calls this CTAS function.

@nfx nfx unassigned FastLee Apr 15, 2024
@nfx nfx added the migrate/managed go/uc/upgrade Upgrade Managed Tables and Jobs label Apr 15, 2024
@nfx nfx removed the migrate/managed go/uc/upgrade Upgrade Managed Tables and Jobs label Apr 15, 2024
nfx pushed a commit that referenced this issue Apr 18, 2024
… Sync) (#1432)

relates to #1340
A new feature allows migration of external tables in Databricks' Hive
metastore using "Create Table (No Sync)" method, with new methods
`_migrate_non_sync_table` and `_get_create_in_place_sql` for migration
and SQL query generation. The `_migrate_dbfs_root_table` and
`_migrate_acl` methods have also been updated. A new test case
demonstrates migration of external tables while preserving their
location and properties.
@nfx nfx closed this as completed in #1434 Apr 18, 2024
nfx pushed a commit that referenced this issue Apr 18, 2024
…nvert such tables to Delta tables. (#1434)

Migrate Non-Delta DBFS root tables to Managed Delta tables.
closes #1340
This release enhances migration for non-delta DBFS tables, supporting
more table types and configurations. New methods improve CTAS
functionality with safer SQL statement generation. Additional table
format compatibility, new creation methods, and updated integration
tests are included. Grant assignments during migration are now
supported.
@github-project-automation github-project-automation bot moved this from Active Backlog to Archive in UCX Apr 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
migrate/external go/uc/upgrade SYNC EXTERNAL TABLES step
Projects
Archived in project
3 participants