Add "what" property for migration to scope down table migrations. #856
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
FastLee
had a problem deploying
to
account-admin
GitHub Actions
Failure
FastLee
had a problem deploying
to
account-admin
GitHub Actions
Failure
FastLee
had a problem deploying
to
account-admin
GitHub Actions
Failure
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## main #856 +/- ##
==========================================
+ Coverage 85.75% 85.85% +0.10%
==========================================
Files 42 42
Lines 5369 5394 +25
Branches 971 978 +7
==========================================
+ Hits 4604 4631 +27
+ Misses 547 546 -1
+ Partials 218 217 -1 ☔ View full report in Codecov by Sentry. |
nfx
requested changes
Jan 29, 2024
FastLee
changed the title
FastLee
force-pushed
the
feature/migrate_ext_table_333
branch
from
fe5d51b to
17be05d
Compare
FastLee
had a problem deploying
to
account-admin
GitHub Actions
Failure
FastLee
force-pushed
the
feature/migrate_ext_table_333
branch
from
17be05d to
ebb0e87
Compare
FastLee
had a problem deploying
to
account-admin
GitHub Actions
Failure
FastLee
had a problem deploying
to
account-admin
GitHub Actions
Failure
FastLee
had a problem deploying
to
account-admin
GitHub Actions
Failure
FastLee
had a problem deploying
to
account-admin
GitHub Actions
Failure
nfx
reviewed
Jan 30, 2024
| table_a = Table("a", "b", "c", "MANAGED", "DELTA", location="dbfs:/somelocation/tablename") | ||
| assert table_a.is_dbfs_root | ||
| assert table_a.what == What.DBFS_ROOT_DELTA | ||
| table_b = Table("a", "b", "c", "MANAGED", "PARQUET", location="dbfs:/somelocation/tablename") |
There was a problem hiding this comment.
what about
- PARQUET & external?
- JSON & external?
- CSV & external?
- TEXT & external?
- avro (whatever other formats) & external?
- what about the same formats & DBFS?
There was a problem hiding this comment.
External tables are divided to supported and unsupported.
All the unsupported are marked "unknown"
It may be a good idea to introduce EXTERNAL_SYNC and EXTERNAL_NO_SYNC or something like that.
There was a problem hiding this comment.
I don't want to replace the table_format I want the field to highlight what kind of table it is when it comes to migration.
There was a problem hiding this comment.
I added EXTERNAL_SYNC and EXTERNAL_NO_SYNC to correspond to the supported/unsupported formats.
FastLee
had a problem deploying
to
account-admin
GitHub Actions
Failure
nfx
added a commit
that referenced
this pull request
Feb 1, 2024
* Added "what" property for migration to scope down table migrations ([#856](#856)). * Added job count in the assessment dashboard ([#858](#858)). * Adopted `installation` package from `databricks-labs-blueprint` ([#860](#860)). * Debug logs to print only the first 96 bytes of SQL query by default, tunable by `debug_truncate_bytes` SDK configuration property ([#859](#859)). * Extract command codes and unify the checks for spark_conf, cluster_policy, init_scripts ([#855](#855)). * Improved installation failure with actionable message ([#840](#840)). * Improved validating groups membership cli command ([#816](#816)). Dependency updates: * Updated databricks-labs-blueprint requirement from ~=0.1.0 to ~=0.2.4 ([#867](#867)).
Merged
nfx
added a commit
that referenced
this pull request
Feb 1, 2024
* Added "what" property for migration to scope down table migrations ([#856](#856)). * Added job count in the assessment dashboard ([#858](#858)). * Adopted `installation` package from `databricks-labs-blueprint` ([#860](#860)). * Debug logs to print only the first 96 bytes of SQL query by default, tunable by `debug_truncate_bytes` SDK configuration property ([#859](#859)). * Extract command codes and unify the checks for spark_conf, cluster_policy, init_scripts ([#855](#855)). * Improved installation failure with actionable message ([#840](#840)). * Improved validating groups membership cli command ([#816](#816)). Dependency updates: * Updated databricks-labs-blueprint requirement from ~=0.1.0 to ~=0.2.4 ([#867](#867)).
dmoore247
pushed a commit
that referenced
this pull request
Mar 23, 2024
) ## Changes <!-- Summary of your changes that are easy to understand. Add screenshots when necessary --> ### Linked issues related to #333 Resolves #.. ### Functionality - [ ] added relevant user documentation - [ ] added new CLI command - [ ] modified existing command: `databricks labs ucx ...` - [ ] added a new workflow - [ ] modified existing workflow: `...` - [ ] added a new table - [ ] modified existing table: `...` ### Tests <!-- How is this tested? Please see the checklist below and also describe any other relevant tests --> - [ ] manually tested - [ ] added unit tests - [ ] added integration tests - [ ] verified on staging environment (screenshot attached)
dmoore247
pushed a commit
that referenced
this pull request
Mar 23, 2024
* Added "what" property for migration to scope down table migrations ([#856](#856)). * Added job count in the assessment dashboard ([#858](#858)). * Adopted `installation` package from `databricks-labs-blueprint` ([#860](#860)). * Debug logs to print only the first 96 bytes of SQL query by default, tunable by `debug_truncate_bytes` SDK configuration property ([#859](#859)). * Extract command codes and unify the checks for spark_conf, cluster_policy, init_scripts ([#855](#855)). * Improved installation failure with actionable message ([#840](#840)). * Improved validating groups membership cli command ([#816](#816)). Dependency updates: * Updated databricks-labs-blueprint requirement from ~=0.1.0 to ~=0.2.4 ([#867](#867)).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes
Linked issues
related to #333
Resolves #..
Functionality
databricks labs ucx .........Tests