v0.3.0

• Added inventory_database name check during installation (#275).
• Added a column to $inventory.tables to specify if a table might have been synchronised to Unity Catalog already or not (#306).
• Added a migration state to skip already migrated tables (#325).
• Fixed appending to tables by adding filtering of None rows (#356).
• Fixed handling of missing but linked cluster policies. (#361).
• Ignore errors for Redash widgets and queries redeployment during installation (#367).
• Remove exception and added proper logging for groups in the list that… (#357).
• Skip group migration when no groups are available after preparation step. (#363).
• Update databricks-sdk requirement from ~=0.9.0 to ~=0.10.0 (#362).

Kudos to @william-conti @FastLee @larsgeorge-db @nfx @dmoore247

v0.2.0

• Added retrieving for all account-level groups with matching names to workspace-level groups in case no explicit configuration (#277).
• Added crawler for Azure Service principals used for direct storage access (#305).
• Added more SQL queries to the assessment step dashboard (#269).
• Added filtering out for job clusters in the clusters crawler (#298).
• Added recording errors from crawl_tables step in $inventory.table_failures table and display counter on the dashboard (#300).
• Added comprehensive introduction user manual (#273).
• Added interactive tutorial for local group migration readme (#291).
• Added tutorial links to the landing page of documentation (#290).
• Added (internal) support for account-level configuration and multi-cloud workspace list (#264).
• Improved order of tasks in the README notebook (#286).
• Improved installation script to run in a Windows Git Bash terminal (#282).
• Improved installation script by setting log level to uppercase by default (#271).
• Improved installation finish messages within installer script (#267).
• Improved automation for MANAGED table migration and continued building tables migration component (#295).
• Fixed debug notebook code with refactored package structure (#250) (#265).
• Fixed replacement of custom configured database to replicate in the report for external locations (#296).
• Removed redundant notebooks top-level folder (#263).
• Split checking for test failures and linting errors into independent GitHub Actions checks (#287).
• Verify query metadata for assessment dashboards during unit tests (#294).

Kudos to @tamilselvanveeramani @larsgeorge-db @saraivdbx @dmoore247 @nfx @FastLee @william-conti @dipankarkush-db

v0.1.1

• Added batched iteration for INSERT INTO queries in StatementExecutionBackend with default max_records_per_batch=1000 (#237).
• Added crawler for mount points (#209).
• Added crawlers for compatibility of jobs and clusters, along with basic recommendations for external locations (#244).
• Added safe return on grants (#246).
• Added ability to specify empty group filter in the installer script (#216) (#217).
• Added ability to install application by multiple different users on the same workspace (#235).
• Added dashboard creation on installation and a requirement for warehouse_id in config, so that the assessment dashboards are refreshed automatically after job runs (#214).
• Added reliance on rate limiting from Databricks SDK for listing workspace (#258).
• Fixed errors in corner cases where Azure Service Principal Credentials were not available in Spark context (#254).
• Fixed DESCRIBE TABLE throwing errors when listing Legacy Table ACLs (#238).
• Fixed file already exists error in the installer script (#219) (#222).
• Fixed guess_external_locations failure with AttributeError: as_dict and added an integration test (#259).
• Fixed error handling edge cases in crawl_tables task (#243) (#251).
• Fixed crawl_permissions task failure on folder names containing a forward slash (#234).
• Improved README notebook documentation (#260, #228, #252, #223, #225).
• Removed redundant .python-version file (#221).
• Removed discovery of account groups from crawl_permissions task (#240).
• Updated databricks-sdk requirement from ~=0.8.0 to ~=0.9.0 (#245).

Kudos to @larsgeorge-db @william-conti @dmoore247 @tamilselvanveeramani @nfx @FastLee

v0.1.0

Version changelog

0.1.0

Features

• Added interactive installation wizard (#184, #117).
• Added schedule of jobs as part of install.sh flow and created some documentation (#187).
• Added debug notebook companion to troubleshoot the installation (#191).
• Added support for Hive Metastore Table ACLs inventory from all databases (#78, #122, #151).
• Created $inventory.tables from Scala notebook (#207).
• Added local group migration support for ML-related objects (#56).
• Added local group migration support for SQL warehouses (#57).
• Added local group migration support for all compute-related resources (#53).
• Added local group migration support for security-related objects (#58).
• Added local group migration support for workflows (#54).
• Added local group migration support for workspace-level objects (#59).
• Added local group migration support for dashboards, queries, and alerts (#144).

Stability

• Added codecov.io publishing (#204).
• Added more tests to group.py (#148).
• Added tests for group state (#133).
• Added tests for inventorizer and typed (#125).
• Added tests WorkspaceListing (#110).
• Added make_*_permissions fixtures (#159).
• Added reusable fixtures module (#119).
• Added testing for permissions (#126).
• Added inventory table manager tests (#153).
• Added product_info to track as SDK integration (#76).
• Added failsafe permission get operations (#65).
• Always install the latest pip version in ./install.sh (#201).
• Always store inventory in hive_metastore and make only inventory_database configurable (#178).
• Changed default logging level from TRACE to DEBUG log level (#124).
• Consistently use WorkspaceClient from databricks.sdk (#120).
• Convert pipeline code to use fixtures. (#166).
• Exclude mixins from coverage (#130).
• Fixed codecov.io reporting (#212).
• Fixed configuration path in job task install code (#210).
• Fixed a bug with dependency definitions (#70).
• Fixed failing test_jobs (#140).
• Fixed the issues with experiment listing (#64).
• Fixed integration testing configuration (#77).
• Make project runnable on nightly testing infrastructure (#75).
• Migrated cluster policies to new fixtures (#174).
• Migrated clusters to the new fixture framework (#162).
• Migrated instance pool to the new fixture framework (#161).
• Migrated to databricks.labs.ucx package (#90).
• Migrated token authorization to new fixtures (#175).
• Migrated experiment fixture to standard one (#168).
• Migrated jobs test to fixture based one. (#167).
• Migrated model fixture to the standard fixtures (#169).
• Migrated warehouse fixture to standard one (#170).
• Organise modules by domain (#197).
• Prefetch all account-level and workspace-level groups (#192).
• Programmatically create a dashboard (#121).
• Properly integrate Python logging facility (#118).
• Refactored code to use Databricks SDK for Python (#27).
• Refactored configuration and remove global provider state (#71).
• Removed pydantic dependency (#138).
• Removed redundant pyspark, databricks-connect, delta-spark, and pandas dependencies (#193).
• Removed redundant typer[all] dependency and its usages (#194).
• Renamed MigrationGroupsProvider to GroupMigrationState (#81).
• Replaced ratelimit and tenacity dependencies with simpler implementations (#195).
• Reorganised integration tests to align more with unit tests (#206).
• Run build workflow also on main branch (#211).
• Run integration test with a single group (#152).
• Simplify SqlBackend and table creation logic (#203).
• Updated migration_config.yml (#179).
• Updated legal information (#196).
• Use make_secret_scope fixture (#163).
• Use fixture factory for make_table, make_schema, and make_catalog (#189).
• Use new fixtures for notebooks and folders (#176).
• Validate toolkit notebook test (#183).

Contributing

• Added a note on external dependencies (#139).
• Added ability to run SQL queries on Spark when in Databricks Runtime (#108).
• Added some ground rules for contributing (#82).
• Added contributing instructions link from main readme (#109).
• Added info about environment refreshes (#155).
• Clarified documentation (#137).
• Enabled merge queue (#146).
• Improved CONTRIBUTING.md guide (#135, #145).

Kudos to @dependabot @nsenno-dbr @renardeinside @nfx @william-conti @larsgeorge-db @HariGS-DB @saraivdbx

v0.0.2

Lates version before SDK version

v0.0.1

With this release we introduce the first public version of the UC Upgrade repo. There will be a lot of changes further, therefore anybody who is looking for the latest working version shall use release 0.0.1 unless a different release comes out.