Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

whitelist databricks-feature-engineering #1976

Closed
wants to merge 4 commits into from
Closed

whitelist databricks-feature-engineering #1976

wants to merge 4 commits into from

Conversation

ericvergnaud
Copy link
Contributor

Changes

whitelist databricks-feature-engineering

Linked issues

Progresses #1901

Functionality

None

Tests

  • manually tested

@ericvergnaud ericvergnaud requested review from a team and dleiva04 July 2, 2024 15:02
@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 2, 2024 •
edited
Loading

❌ 87/91 passed, 4 failed, 21 skipped, 5h23m9s total

❌ test_create_catalog_schema_with_principal_acl_CLOUD_ENV: TimeoutError: Timed out after 0:02:00 (5m46.451s) 
TimeoutError: Timed out after 0:02:00
16:47 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/Z14w
16:48 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_s90eq: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_s90eq
16:48 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_s90eq', metastore_id=None, name='ucx_s90eq', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
16:48 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.ucx_s90eq.ucx_tydsj: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_s90eq/ucx_tydsj
16:48 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.CSV: 'CSV'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.ucx_s90eq.ucx_tydsj', metastore_id=None, name='ucx_tydsj', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070218'}, row_filter=None, schema_name='ucx_s90eq', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/b/Z14w', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
16:48 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=<CatalogType.MANAGED_CATALOG: 'MANAGED_CATALOG'>, comment='', connection_name=None, created_at=1719938933692, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_c16lb', isolation_mode=<CatalogIsolationMode.OPEN: 'OPEN'>, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_c16lb', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=<CatalogInfoSecurableKind.CATALOG_STANDARD: 'CATALOG_STANDARD'>, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1719938933692, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d')
16:49 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_c16lb.ucx_s90eq: https://DATABRICKS_HOST/explore/data/ucx_c16lb/ucx_s90eq
16:49 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_c16lb', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_c16lb.ucx_s90eq', metastore_id=None, name='ucx_s90eq', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
[gw8] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python
16:47 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/Z14w
16:48 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_s90eq: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_s90eq
16:48 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_s90eq', metastore_id=None, name='ucx_s90eq', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
16:48 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.ucx_s90eq.ucx_tydsj: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_s90eq/ucx_tydsj
16:48 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.CSV: 'CSV'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.ucx_s90eq.ucx_tydsj', metastore_id=None, name='ucx_tydsj', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070218'}, row_filter=None, schema_name='ucx_s90eq', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/b/Z14w', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
16:48 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=<CatalogType.MANAGED_CATALOG: 'MANAGED_CATALOG'>, comment='', connection_name=None, created_at=1719938933692, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_c16lb', isolation_mode=<CatalogIsolationMode.OPEN: 'OPEN'>, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_c16lb', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=<CatalogInfoSecurableKind.CATALOG_STANDARD: 'CATALOG_STANDARD'>, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1719938933692, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d')
16:49 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_c16lb.ucx_s90eq: https://DATABRICKS_HOST/explore/data/ucx_c16lb/ucx_s90eq
16:49 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_c16lb', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_c16lb.ucx_s90eq', metastore_id=None, name='ucx_s90eq', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
16:49 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster fixture: <databricks.sdk.service._internal.Wait object at 0x7efcc61c46a0>
16:49 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace user fixture: User(active=True, display_name='[email protected]', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='[email protected]')], entitlements=[], external_id=None, groups=[], id='1962998821917202', name=Name(family_name=None, given_name='[email protected]'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='[email protected]')
16:49 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster permissions fixture: 0702-164921-x8h66ybl [group_name admins CAN_MANAGE] -> [user_name [email protected] CAN_ATTACH_TO]
16:49 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sfhmy: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sfhmy
16:49 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sfhmy', metastore_id=None, name='ucx_sfhmy', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 INFO [databricks.labs.ucx.hive_metastore.catalog_schema] Listing existing UC catalogs and schemas
16:50 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster fixture: <databricks.sdk.service._internal.Wait object at 0x7efcc62b41c0>
16:50 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace user fixture: User(active=True, display_name='[email protected]', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='[email protected]')], entitlements=[], external_id=None, groups=[], id='8888737530109253', name=Name(family_name=None, given_name='[email protected]'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='[email protected]')
16:50 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster permissions fixture: 0702-165019-pmmrn61t [group_name admins CAN_MANAGE] -> [user_name [email protected] CAN_ATTACH_TO]
16:50 INFO [databricks.labs.ucx.hive_metastore.catalog_schema] Listing existing UC catalogs and schemas
16:50 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster fixture: <databricks.sdk.service._internal.Wait object at 0x7efcc622be80>
16:50 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace user fixture: User(active=True, display_name='[email protected]', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='[email protected]')], entitlements=[], external_id=None, groups=[], id='7888406658453865', name=Name(family_name=None, given_name='[email protected]'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='[email protected]')
16:51 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster permissions fixture: 0702-165051-7v3hdvio [group_name admins CAN_MANAGE] -> [user_name [email protected] CAN_ATTACH_TO]
16:51 INFO [databricks.labs.ucx.hive_metastore.catalog_schema] Listing existing UC catalogs and schemas
16:47 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/Z14w
16:48 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_s90eq: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_s90eq
16:48 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_s90eq', metastore_id=None, name='ucx_s90eq', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
16:48 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.ucx_s90eq.ucx_tydsj: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_s90eq/ucx_tydsj
16:48 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.CSV: 'CSV'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.ucx_s90eq.ucx_tydsj', metastore_id=None, name='ucx_tydsj', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070218'}, row_filter=None, schema_name='ucx_s90eq', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/b/Z14w', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
16:48 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=<CatalogType.MANAGED_CATALOG: 'MANAGED_CATALOG'>, comment='', connection_name=None, created_at=1719938933692, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_c16lb', isolation_mode=<CatalogIsolationMode.OPEN: 'OPEN'>, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_c16lb', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=<CatalogInfoSecurableKind.CATALOG_STANDARD: 'CATALOG_STANDARD'>, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1719938933692, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d')
16:49 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_c16lb.ucx_s90eq: https://DATABRICKS_HOST/explore/data/ucx_c16lb/ucx_s90eq
16:49 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_c16lb', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_c16lb.ucx_s90eq', metastore_id=None, name='ucx_s90eq', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
16:49 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster fixture: <databricks.sdk.service._internal.Wait object at 0x7efcc61c46a0>
16:49 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace user fixture: User(active=True, display_name='[email protected]', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='[email protected]')], entitlements=[], external_id=None, groups=[], id='1962998821917202', name=Name(family_name=None, given_name='[email protected]'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='[email protected]')
16:49 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster permissions fixture: 0702-164921-x8h66ybl [group_name admins CAN_MANAGE] -> [user_name [email protected] CAN_ATTACH_TO]
16:49 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sfhmy: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sfhmy
16:49 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sfhmy', metastore_id=None, name='ucx_sfhmy', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 WARNING [databricks.labs.ucx.assessment.secrets] Secret in config stored in plaintext.
16:49 INFO [databricks.labs.ucx.hive_metastore.catalog_schema] Listing existing UC catalogs and schemas
16:50 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster fixture: <databricks.sdk.service._internal.Wait object at 0x7efcc62b41c0>
16:50 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace user fixture: User(active=True, display_name='[email protected]', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='[email protected]')], entitlements=[], external_id=None, groups=[], id='8888737530109253', name=Name(family_name=None, given_name='[email protected]'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='[email protected]')
16:50 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster permissions fixture: 0702-165019-pmmrn61t [group_name admins CAN_MANAGE] -> [user_name [email protected] CAN_ATTACH_TO]
16:50 INFO [databricks.labs.ucx.hive_metastore.catalog_schema] Listing existing UC catalogs and schemas
16:50 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster fixture: <databricks.sdk.service._internal.Wait object at 0x7efcc622be80>
16:50 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace user fixture: User(active=True, display_name='[email protected]', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='[email protected]')], entitlements=[], external_id=None, groups=[], id='7888406658453865', name=Name(family_name=None, given_name='[email protected]'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='[email protected]')
16:51 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster permissions fixture: 0702-165051-7v3hdvio [group_name admins CAN_MANAGE] -> [user_name [email protected] CAN_ATTACH_TO]
16:51 INFO [databricks.labs.ucx.hive_metastore.catalog_schema] Listing existing UC catalogs and schemas
16:51 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 3 cluster permissions fixtures
16:51 DEBUG [databricks.labs.ucx.mixins.fixtures] removing cluster permissions fixture: 0702-164921-x8h66ybl [group_name admins CAN_MANAGE] -> [user_name [email protected] CAN_ATTACH_TO]
16:51 DEBUG [databricks.labs.ucx.mixins.fixtures] removing cluster permissions fixture: 0702-165019-pmmrn61t [group_name admins CAN_MANAGE] -> [user_name [email protected] CAN_ATTACH_TO]
16:51 DEBUG [databricks.labs.ucx.mixins.fixtures] removing cluster permissions fixture: 0702-165051-7v3hdvio [group_name admins CAN_MANAGE] -> [user_name [email protected] CAN_ATTACH_TO]
16:52 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 catalog fixtures
16:52 DEBUG [databricks.labs.ucx.mixins.fixtures] removing catalog fixture: CatalogInfo(browse_only=False, catalog_type=<CatalogType.MANAGED_CATALOG: 'MANAGED_CATALOG'>, comment='', connection_name=None, created_at=1719938933692, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_c16lb', isolation_mode=<CatalogIsolationMode.OPEN: 'OPEN'>, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_c16lb', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=<CatalogInfoSecurableKind.CATALOG_STANDARD: 'CATALOG_STANDARD'>, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1719938933692, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d')
16:52 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 make_dbfs_data_copy fixtures
16:52 DEBUG [databricks.labs.ucx.mixins.fixtures] removing make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/Z14w
16:52 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 3 cluster fixtures
16:52 DEBUG [databricks.labs.ucx.mixins.fixtures] removing cluster fixture: <databricks.sdk.service._internal.Wait object at 0x7efcc61c46a0>
16:52 DEBUG [databricks.labs.ucx.mixins.fixtures] removing cluster fixture: <databricks.sdk.service._internal.Wait object at 0x7efcc62b41c0>
16:52 DEBUG [databricks.labs.ucx.mixins.fixtures] removing cluster fixture: <databricks.sdk.service._internal.Wait object at 0x7efcc622be80>
16:52 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 workspace group fixtures
16:52 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures
16:52 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 table fixtures
16:52 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.CSV: 'CSV'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.ucx_s90eq.ucx_tydsj', metastore_id=None, name='ucx_tydsj', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070218'}, row_filter=None, schema_name='ucx_s90eq', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/b/Z14w', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
16:52 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 3 schema fixtures
16:52 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_s90eq', metastore_id=None, name='ucx_s90eq', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
16:52 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_c16lb', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_c16lb.ucx_s90eq', metastore_id=None, name='ucx_s90eq', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
16:53 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sfhmy', metastore_id=None, name='ucx_sfhmy', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
16:53 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 3 workspace user fixtures
16:53 DEBUG [databricks.labs.ucx.mixins.fixtures] removing workspace user fixture: User(active=True, display_name='[email protected]', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='[email protected]')], entitlements=[], external_id=None, groups=[], id='1962998821917202', name=Name(family_name=None, given_name='[email protected]'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='[email protected]')
16:53 DEBUG [databricks.labs.ucx.mixins.fixtures] removing workspace user fixture: User(active=True, display_name='[email protected]', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='[email protected]')], entitlements=[], external_id=None, groups=[], id='8888737530109253', name=Name(family_name=None, given_name='[email protected]'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='[email protected]')
16:53 DEBUG [databricks.labs.ucx.mixins.fixtures] removing workspace user fixture: User(active=True, display_name='[email protected]', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='[email protected]')], entitlements=[], external_id=None, groups=[], id='7888406658453865', name=Name(family_name=None, given_name='[email protected]'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='[email protected]')
[gw8] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python
❌ test_table_migration_job_refreshes_migration_status[hiveserde-migrate-external-tables-ctas]: TimeoutError: Timed out after 0:05:00 (19m35.491s) 
TimeoutError: Timed out after 0:05:00
16:54 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/8Tyy
16:55 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.hiveserde_in_place_bu5hf: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_bu5hf
16:55 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.hiveserde_in_place_bu5hf', metastore_id=None, name='hiveserde_in_place_bu5hf', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
16:55 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_storage_dir fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_bu5hf
16:55 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_bu5hf.parquet_serde_bu5hf: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_bu5hf/parquet_serde_bu5hf
16:55 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_bu5hf.parquet_serde_bu5hf', metastore_id=None, name='parquet_serde_bu5hf', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070218'}, row_filter=None, schema_name='hiveserde_in_place_bu5hf', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_bu5hf/parquet_serde_bu5hf', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
16:56 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_bu5hf.orc_serde_bu5hf: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_bu5hf/orc_serde_bu5hf
16:56 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_bu5hf.orc_serde_bu5hf', metastore_id=None, name='orc_serde_bu5hf', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070218'}, row_filter=None, schema_name='hiveserde_in_place_bu5hf', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_bu5hf/orc_serde_bu5hf', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
16:56 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_bu5hf.avro_serde_bu5hf: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_bu5hf/avro_serde_bu5hf
16:56 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_bu5hf.avro_serde_bu5hf', metastore_id=None, name='avro_serde_bu5hf', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070218'}, row_filter=None, schema_name='hiveserde_in_place_bu5hf', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_bu5hf/avro_serde_bu5hf', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
16:56 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=<CatalogType.MANAGED_CATALOG: 'MANAGED_CATALOG'>, comment='', connection_name=None, created_at=1719939409344, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_chh16', isolation_mode=<CatalogIsolationMode.OPEN: 'OPEN'>, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_chh16', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=<CatalogInfoSecurableKind.CATALOG_STANDARD: 'CATALOG_STANDARD'>, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1719939409344, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d')
16:57 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_chh16.hiveserde_in_place_bu5hf: https://DATABRICKS_HOST/explore/data/ucx_chh16/hiveserde_in_place_bu5hf
16:57 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_chh16', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_chh16.hiveserde_in_place_bu5hf', metastore_id=None, name='hiveserde_in_place_bu5hf', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
16:57 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_src3c: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_src3c
16:57 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_src3c', metastore_id=None, name='ucx_src3c', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
[gw6] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python
16:54 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/8Tyy
16:55 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.hiveserde_in_place_bu5hf: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_bu5hf
16:55 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.hiveserde_in_place_bu5hf', metastore_id=None, name='hiveserde_in_place_bu5hf', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
16:55 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_storage_dir fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_bu5hf
16:55 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_bu5hf.parquet_serde_bu5hf: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_bu5hf/parquet_serde_bu5hf
16:55 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_bu5hf.parquet_serde_bu5hf', metastore_id=None, name='parquet_serde_bu5hf', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070218'}, row_filter=None, schema_name='hiveserde_in_place_bu5hf', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_bu5hf/parquet_serde_bu5hf', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
16:56 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_bu5hf.orc_serde_bu5hf: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_bu5hf/orc_serde_bu5hf
16:56 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_bu5hf.orc_serde_bu5hf', metastore_id=None, name='orc_serde_bu5hf', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070218'}, row_filter=None, schema_name='hiveserde_in_place_bu5hf', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_bu5hf/orc_serde_bu5hf', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
16:56 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_bu5hf.avro_serde_bu5hf: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_bu5hf/avro_serde_bu5hf
16:56 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_bu5hf.avro_serde_bu5hf', metastore_id=None, name='avro_serde_bu5hf', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070218'}, row_filter=None, schema_name='hiveserde_in_place_bu5hf', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_bu5hf/avro_serde_bu5hf', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
16:56 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=<CatalogType.MANAGED_CATALOG: 'MANAGED_CATALOG'>, comment='', connection_name=None, created_at=1719939409344, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_chh16', isolation_mode=<CatalogIsolationMode.OPEN: 'OPEN'>, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_chh16', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=<CatalogInfoSecurableKind.CATALOG_STANDARD: 'CATALOG_STANDARD'>, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1719939409344, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d')
16:57 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_chh16.hiveserde_in_place_bu5hf: https://DATABRICKS_HOST/explore/data/ucx_chh16/hiveserde_in_place_bu5hf
16:57 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_chh16', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_chh16.hiveserde_in_place_bu5hf', metastore_id=None, name='hiveserde_in_place_bu5hf', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
16:57 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_src3c: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_src3c
16:57 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_src3c', metastore_id=None, name='ucx_src3c', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
17:01 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.Twdb/config.yml) doesn't exist.
17:01 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration
17:01 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data.
17:01 INFO [databricks.labs.ucx.install] Fetching installations...
17:01 WARNING [databricks.labs.ucx.install] Existing installation at /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.Twdb is corrupted. Skipping...
17:01 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy.
17:01 DEBUG [tests.integration.conftest] Waiting for clusters to start...
17:01 DEBUG [tests.integration.conftest] Waiting for clusters to start...
17:02 INFO [databricks.labs.ucx.install] Installing UCX v0.27.2+6020240702170252
17:02 INFO [databricks.labs.ucx.install] Creating ucx schemas...
17:06 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental
17:06 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
17:07 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter
17:07 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
17:07 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental
17:07 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation
17:07 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
17:07 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
17:07 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
17:07 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
17:07 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
17:07 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
17:07 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
17:07 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.Twdb/README for the next steps.
17:07 DEBUG [databricks.labs.ucx.installer.workflows] starting migrate-external-tables-ctas job: https://DATABRICKS_HOST#job/199414338629540
16:54 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/8Tyy
16:55 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.hiveserde_in_place_bu5hf: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_bu5hf
16:55 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.hiveserde_in_place_bu5hf', metastore_id=None, name='hiveserde_in_place_bu5hf', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
16:55 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_storage_dir fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_bu5hf
16:55 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_bu5hf.parquet_serde_bu5hf: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_bu5hf/parquet_serde_bu5hf
16:55 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_bu5hf.parquet_serde_bu5hf', metastore_id=None, name='parquet_serde_bu5hf', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070218'}, row_filter=None, schema_name='hiveserde_in_place_bu5hf', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_bu5hf/parquet_serde_bu5hf', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
16:56 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_bu5hf.orc_serde_bu5hf: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_bu5hf/orc_serde_bu5hf
16:56 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_bu5hf.orc_serde_bu5hf', metastore_id=None, name='orc_serde_bu5hf', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070218'}, row_filter=None, schema_name='hiveserde_in_place_bu5hf', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_bu5hf/orc_serde_bu5hf', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
16:56 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.hiveserde_in_place_bu5hf.avro_serde_bu5hf: https://DATABRICKS_HOST/explore/data/hive_metastore/hiveserde_in_place_bu5hf/avro_serde_bu5hf
16:56 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_bu5hf.avro_serde_bu5hf', metastore_id=None, name='avro_serde_bu5hf', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070218'}, row_filter=None, schema_name='hiveserde_in_place_bu5hf', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_bu5hf/avro_serde_bu5hf', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
16:56 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=<CatalogType.MANAGED_CATALOG: 'MANAGED_CATALOG'>, comment='', connection_name=None, created_at=1719939409344, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_chh16', isolation_mode=<CatalogIsolationMode.OPEN: 'OPEN'>, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_chh16', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=<CatalogInfoSecurableKind.CATALOG_STANDARD: 'CATALOG_STANDARD'>, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1719939409344, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d')
16:57 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_chh16.hiveserde_in_place_bu5hf: https://DATABRICKS_HOST/explore/data/ucx_chh16/hiveserde_in_place_bu5hf
16:57 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_chh16', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_chh16.hiveserde_in_place_bu5hf', metastore_id=None, name='hiveserde_in_place_bu5hf', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
16:57 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_src3c: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_src3c
16:57 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_src3c', metastore_id=None, name='ucx_src3c', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
17:01 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.Twdb/config.yml) doesn't exist.
17:01 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration
17:01 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data.
17:01 INFO [databricks.labs.ucx.install] Fetching installations...
17:01 WARNING [databricks.labs.ucx.install] Existing installation at /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.Twdb is corrupted. Skipping...
17:01 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy.
17:01 DEBUG [tests.integration.conftest] Waiting for clusters to start...
17:01 DEBUG [tests.integration.conftest] Waiting for clusters to start...
17:02 INFO [databricks.labs.ucx.install] Installing UCX v0.27.2+6020240702170252
17:02 INFO [databricks.labs.ucx.install] Creating ucx schemas...
17:06 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental
17:06 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
17:07 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter
17:07 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
17:07 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental
17:07 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation
17:07 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
17:07 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
17:07 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
17:07 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
17:07 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
17:07 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
17:07 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
17:07 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.Twdb/README for the next steps.
17:07 DEBUG [databricks.labs.ucx.installer.workflows] starting migrate-external-tables-ctas job: https://DATABRICKS_HOST#job/199414338629540
17:11 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 make_storage_dir fixtures
17:11 DEBUG [databricks.labs.ucx.mixins.fixtures] removing make_storage_dir fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_bu5hf
17:11 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 make_dbfs_data_copy fixtures
17:11 DEBUG [databricks.labs.ucx.mixins.fixtures] removing make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/8Tyy
17:12 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 cluster fixtures
17:12 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 catalog fixtures
17:12 DEBUG [databricks.labs.ucx.mixins.fixtures] removing catalog fixture: CatalogInfo(browse_only=False, catalog_type=<CatalogType.MANAGED_CATALOG: 'MANAGED_CATALOG'>, comment='', connection_name=None, created_at=1719939409344, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_chh16', isolation_mode=<CatalogIsolationMode.OPEN: 'OPEN'>, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_chh16', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=<CatalogInfoSecurableKind.CATALOG_STANDARD: 'CATALOG_STANDARD'>, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1719939409344, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d')
17:12 INFO [databricks.labs.ucx.install] Deleting UCX v0.27.2+6020240702170252 from https://DATABRICKS_HOST
17:12 INFO [databricks.labs.ucx.install] Deleting inventory database ucx_src3c
17:12 INFO [databricks.labs.ucx.install] Deleting jobs
17:12 INFO [databricks.labs.ucx.install] Deleting scan-tables-in-mounts-experimental job_id=98464748957468.
17:12 INFO [databricks.labs.ucx.install] Deleting migrate-external-hiveserde-tables-in-place-experimental job_id=697908783517541.
17:12 INFO [databricks.labs.ucx.install] Deleting experimental-workflow-linter job_id=699829952266564.
17:12 INFO [databricks.labs.ucx.install] Deleting migrate-tables job_id=1019664110950990.
17:12 INFO [databricks.labs.ucx.install] Deleting migrate-groups-experimental job_id=213699773501601.
17:12 INFO [databricks.labs.ucx.install] Deleting migrate-data-reconciliation job_id=160523731286056.
17:12 INFO [databricks.labs.ucx.install] Deleting failing job_id=244865312564026.
17:12 INFO [databricks.labs.ucx.install] Deleting remove-workspace-local-backup-groups job_id=230331030866844.
17:12 INFO [databricks.labs.ucx.install] Deleting assessment job_id=853610090304833.
17:12 INFO [databricks.labs.ucx.install] Deleting migrate-groups job_id=510320754075154.
17:12 INFO [databricks.labs.ucx.install] Deleting migrate-tables-in-mounts-experimental job_id=952510595764189.
17:12 INFO [databricks.labs.ucx.install] Deleting migrate-external-tables-ctas job_id=199414338629540.
17:12 INFO [databricks.labs.ucx.install] Deleting validate-groups-permissions job_id=217443090601129.
17:12 INFO [databricks.labs.ucx.install] Deleting cluster policy
17:12 INFO [databricks.labs.ucx.install] Deleting secret scope
17:12 INFO [databricks.labs.ucx.install] UnInstalling UCX complete
17:12 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 workspace user fixtures
17:12 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 account group fixtures
17:12 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 workspace group fixtures
17:12 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures
17:12 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 3 table fixtures
17:12 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_bu5hf.parquet_serde_bu5hf', metastore_id=None, name='parquet_serde_bu5hf', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070218'}, row_filter=None, schema_name='hiveserde_in_place_bu5hf', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_bu5hf/parquet_serde_bu5hf', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
17:12 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_bu5hf.orc_serde_bu5hf', metastore_id=None, name='orc_serde_bu5hf', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070218'}, row_filter=None, schema_name='hiveserde_in_place_bu5hf', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_bu5hf/orc_serde_bu5hf', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
17:13 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.hiveserde_in_place_bu5hf.avro_serde_bu5hf', metastore_id=None, name='avro_serde_bu5hf', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070218'}, row_filter=None, schema_name='hiveserde_in_place_bu5hf', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/hiveserde_in_place_bu5hf/avro_serde_bu5hf', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
17:13 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 3 schema fixtures
17:13 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.hiveserde_in_place_bu5hf', metastore_id=None, name='hiveserde_in_place_bu5hf', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
17:13 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_chh16', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_chh16.hiveserde_in_place_bu5hf', metastore_id=None, name='hiveserde_in_place_bu5hf', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
17:14 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_src3c', metastore_id=None, name='ucx_src3c', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
[gw6] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python
❌ test_running_real_workflow_linter_job: TimeoutError: Timed out after 0:02:00 (7m53.998s) 
TimeoutError: Timed out after 0:02:00
[gw8] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python
17:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added directory fixture: /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/sdk-TZkQ
17:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added notebook fixture: /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/sdk-TZkQ/notebook.ipynb
17:15 INFO [databricks.labs.ucx.mixins.fixtures] Job: https://DATABRICKS_HOST#job/760292123173154
17:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added job fixture: CreateResponse(job_id=760292123173154)
17:16 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_s3anz: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_s3anz
17:16 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_s3anz', metastore_id=None, name='ucx_s3anz', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
17:16 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.wbk2/config.yml) doesn't exist.
17:16 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration
17:16 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data.
17:16 INFO [databricks.labs.ucx.install] Fetching installations...
17:16 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy.
17:16 DEBUG [tests.integration.conftest] Waiting for clusters to start...
17:16 DEBUG [tests.integration.conftest] Waiting for clusters to start...
17:16 INFO [databricks.labs.ucx.install] Installing UCX v0.27.2+6020240702171627
17:16 INFO [databricks.labs.ucx.install] Creating ucx schemas...
17:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
17:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
17:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental
17:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
17:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter
17:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation
17:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
17:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
17:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
17:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
17:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental
17:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
17:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
17:18 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.wbk2/README for the next steps.
17:18 DEBUG [databricks.labs.ucx.installer.workflows] starting experimental-workflow-linter job: https://DATABRICKS_HOST#job/977100948186972
17:23 DEBUG [databricks.labs.ucx.installer.workflows] Validating experimental-workflow-linter workflow: https://DATABRICKS_HOST#job/977100948186972
17:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added directory fixture: /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/sdk-TZkQ
17:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added notebook fixture: /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/sdk-TZkQ/notebook.ipynb
17:15 INFO [databricks.labs.ucx.mixins.fixtures] Job: https://DATABRICKS_HOST#job/760292123173154
17:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added job fixture: CreateResponse(job_id=760292123173154)
17:16 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_s3anz: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_s3anz
17:16 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_s3anz', metastore_id=None, name='ucx_s3anz', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
17:16 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.wbk2/config.yml) doesn't exist.
17:16 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration
17:16 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data.
17:16 INFO [databricks.labs.ucx.install] Fetching installations...
17:16 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy.
17:16 DEBUG [tests.integration.conftest] Waiting for clusters to start...
17:16 DEBUG [tests.integration.conftest] Waiting for clusters to start...
17:16 INFO [databricks.labs.ucx.install] Installing UCX v0.27.2+6020240702171627
17:16 INFO [databricks.labs.ucx.install] Creating ucx schemas...
17:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
17:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
17:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental
17:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
17:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter
17:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation
17:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
17:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
17:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
17:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
17:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental
17:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
17:18 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
17:18 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.wbk2/README for the next steps.
17:18 DEBUG [databricks.labs.ucx.installer.workflows] starting experimental-workflow-linter job: https://DATABRICKS_HOST#job/977100948186972
17:23 DEBUG [databricks.labs.ucx.installer.workflows] Validating experimental-workflow-linter workflow: https://DATABRICKS_HOST#job/977100948186972
17:23 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 job fixtures
17:23 DEBUG [databricks.labs.ucx.mixins.fixtures] removing job fixture: CreateResponse(job_id=760292123173154)
17:23 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 directory fixtures
17:23 DEBUG [databricks.labs.ucx.mixins.fixtures] removing directory fixture: /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/sdk-TZkQ
17:23 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 notebook fixtures
17:23 DEBUG [databricks.labs.ucx.mixins.fixtures] removing notebook fixture: /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/sdk-TZkQ/notebook.ipynb
17:23 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while notebook /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/sdk-TZkQ/notebook.ipynb teardown: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/sdk-TZkQ/notebook.ipynb) doesn't exist.
17:23 INFO [databricks.labs.ucx.install] Deleting UCX v0.27.2+6020240702171627 from https://DATABRICKS_HOST
17:23 INFO [databricks.labs.ucx.install] Deleting inventory database ucx_s3anz
17:23 INFO [databricks.labs.ucx.install] Deleting jobs
17:23 INFO [databricks.labs.ucx.install] Deleting migrate-tables job_id=723231228145789.
17:23 INFO [databricks.labs.ucx.install] Deleting migrate-external-tables-ctas job_id=347909732218227.
17:23 INFO [databricks.labs.ucx.install] Deleting scan-tables-in-mounts-experimental job_id=845361999255028.
17:23 INFO [databricks.labs.ucx.install] Deleting validate-groups-permissions job_id=572077493140177.
17:23 INFO [databricks.labs.ucx.install] Deleting experimental-workflow-linter job_id=977100948186972.
17:23 INFO [databricks.labs.ucx.install] Deleting migrate-data-reconciliation job_id=878211440419680.
17:23 INFO [databricks.labs.ucx.install] Deleting assessment job_id=8483047074347.
17:23 INFO [databricks.labs.ucx.install] Deleting remove-workspace-local-backup-groups job_id=818702192280558.
17:23 INFO [databricks.labs.ucx.install] Deleting migrate-groups job_id=1059290601950185.
17:23 INFO [databricks.labs.ucx.install] Deleting failing job_id=852213740503740.
17:23 INFO [databricks.labs.ucx.install] Deleting migrate-groups-experimental job_id=767664490868990.
17:23 INFO [databricks.labs.ucx.install] Deleting migrate-tables-in-mounts-experimental job_id=497441898961117.
17:23 INFO [databricks.labs.ucx.install] Deleting migrate-external-hiveserde-tables-in-place-experimental job_id=584800356606911.
17:23 INFO [databricks.labs.ucx.install] Deleting cluster policy
17:23 INFO [databricks.labs.ucx.install] Deleting secret scope
17:23 INFO [databricks.labs.ucx.install] UnInstalling UCX complete
17:23 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 workspace user fixtures
17:23 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 account group fixtures
17:23 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 workspace group fixtures
17:23 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures
17:23 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures
17:23 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 schema fixtures
17:23 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_s3anz', metastore_id=None, name='ucx_s3anz', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
[gw8] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python
❌ test_table_migration_job_refreshes_migration_status[regular-migrate-tables]: TimeoutError: Timed out after 0:05:00 (12m36.021s) 
TimeoutError: Timed out after 0:05:00
17:14 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/DSaW
17:14 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.migrate_75zn9: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_75zn9
17:14 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.migrate_75zn9', metastore_id=None, name='migrate_75zn9', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
17:15 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_75zn9.ucx_tnzeu: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_75zn9/ucx_tnzeu
17:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.DELTA: 'DELTA'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_75zn9.ucx_tnzeu', metastore_id=None, name='ucx_tnzeu', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070218'}, row_filter=None, schema_name='migrate_75zn9', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/migrate_75zn9/ucx_tnzeu', table_constraints=None, table_id=None, table_type=<TableType.MANAGED: 'MANAGED'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
17:15 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_75zn9.ucx_togpb: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_75zn9/ucx_togpb
17:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.JSON: 'JSON'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_75zn9.ucx_togpb', metastore_id=None, name='ucx_togpb', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070218'}, row_filter=None, schema_name='migrate_75zn9', sql_path=None, storage_credential_name=None, storage_location='dbfs:/tmp/ucx_test_U8sA', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
17:15 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_75zn9.ucx_tq0di: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_75zn9/ucx_tq0di
17:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.CSV: 'CSV'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_75zn9.ucx_tq0di', metastore_id=None, name='ucx_tq0di', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070218'}, row_filter=None, schema_name='migrate_75zn9', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/b/DSaW', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
17:16 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_75zn9.ucx_tafdm: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_75zn9/ucx_tafdm
17:16 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_75zn9.ucx_tafdm', metastore_id=None, name='ucx_tafdm', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070218'}, row_filter=None, schema_name='migrate_75zn9', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=<TableType.VIEW: 'VIEW'>, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_75zn9.ucx_tnzeu', view_dependencies=None)
17:16 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_75zn9.ucx_tkjfm: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_75zn9/ucx_tkjfm
17:16 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_75zn9.ucx_tkjfm', metastore_id=None, name='ucx_tkjfm', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070218'}, row_filter=None, schema_name='migrate_75zn9', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=<TableType.VIEW: 'VIEW'>, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_75zn9.ucx_tafdm', view_dependencies=None)
17:17 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=<CatalogType.MANAGED_CATALOG: 'MANAGED_CATALOG'>, comment='', connection_name=None, created_at=1719940623841, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cdrns', isolation_mode=<CatalogIsolationMode.OPEN: 'OPEN'>, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cdrns', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=<CatalogInfoSecurableKind.CATALOG_STANDARD: 'CATALOG_STANDARD'>, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1719940623841, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d')
17:17 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_cdrns.migrate_75zn9: https://DATABRICKS_HOST/explore/data/ucx_cdrns/migrate_75zn9
17:17 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cdrns', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cdrns.migrate_75zn9', metastore_id=None, name='migrate_75zn9', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
17:17 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_svvej: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_svvej
17:17 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_svvej', metastore_id=None, name='ucx_svvej', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
[gw7] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python
17:14 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/DSaW
17:14 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.migrate_75zn9: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_75zn9
17:14 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.migrate_75zn9', metastore_id=None, name='migrate_75zn9', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
17:15 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_75zn9.ucx_tnzeu: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_75zn9/ucx_tnzeu
17:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.DELTA: 'DELTA'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_75zn9.ucx_tnzeu', metastore_id=None, name='ucx_tnzeu', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070218'}, row_filter=None, schema_name='migrate_75zn9', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/migrate_75zn9/ucx_tnzeu', table_constraints=None, table_id=None, table_type=<TableType.MANAGED: 'MANAGED'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
17:15 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_75zn9.ucx_togpb: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_75zn9/ucx_togpb
17:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.JSON: 'JSON'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_75zn9.ucx_togpb', metastore_id=None, name='ucx_togpb', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070218'}, row_filter=None, schema_name='migrate_75zn9', sql_path=None, storage_credential_name=None, storage_location='dbfs:/tmp/ucx_test_U8sA', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
17:15 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_75zn9.ucx_tq0di: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_75zn9/ucx_tq0di
17:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.CSV: 'CSV'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_75zn9.ucx_tq0di', metastore_id=None, name='ucx_tq0di', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070218'}, row_filter=None, schema_name='migrate_75zn9', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/b/DSaW', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
17:16 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_75zn9.ucx_tafdm: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_75zn9/ucx_tafdm
17:16 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_75zn9.ucx_tafdm', metastore_id=None, name='ucx_tafdm', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070218'}, row_filter=None, schema_name='migrate_75zn9', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=<TableType.VIEW: 'VIEW'>, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_75zn9.ucx_tnzeu', view_dependencies=None)
17:16 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_75zn9.ucx_tkjfm: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_75zn9/ucx_tkjfm
17:16 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_75zn9.ucx_tkjfm', metastore_id=None, name='ucx_tkjfm', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070218'}, row_filter=None, schema_name='migrate_75zn9', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=<TableType.VIEW: 'VIEW'>, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_75zn9.ucx_tafdm', view_dependencies=None)
17:17 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=<CatalogType.MANAGED_CATALOG: 'MANAGED_CATALOG'>, comment='', connection_name=None, created_at=1719940623841, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cdrns', isolation_mode=<CatalogIsolationMode.OPEN: 'OPEN'>, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cdrns', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=<CatalogInfoSecurableKind.CATALOG_STANDARD: 'CATALOG_STANDARD'>, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1719940623841, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d')
17:17 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_cdrns.migrate_75zn9: https://DATABRICKS_HOST/explore/data/ucx_cdrns/migrate_75zn9
17:17 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cdrns', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cdrns.migrate_75zn9', metastore_id=None, name='migrate_75zn9', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
17:17 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_svvej: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_svvej
17:17 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_svvej', metastore_id=None, name='ucx_svvej', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
17:18 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.rRsr/config.yml) doesn't exist.
17:18 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration
17:19 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data.
17:19 INFO [databricks.labs.ucx.install] Fetching installations...
17:19 WARNING [databricks.labs.ucx.install] Existing installation at /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.rRsr is corrupted. Skipping...
17:19 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy.
17:19 DEBUG [tests.integration.conftest] Waiting for clusters to start...
17:19 DEBUG [tests.integration.conftest] Waiting for clusters to start...
17:19 INFO [databricks.labs.ucx.install] Installing UCX v0.27.2+6020240702171924
17:19 INFO [databricks.labs.ucx.install] Creating ucx schemas...
17:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
17:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
17:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental
17:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation
17:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental
17:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
17:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
17:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
17:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
17:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter
17:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
17:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
17:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
17:20 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.rRsr/README for the next steps.
17:20 DEBUG [databricks.labs.ucx.installer.workflows] starting migrate-tables job: https://DATABRICKS_HOST#job/213377662691383
17:14 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/DSaW
17:14 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.migrate_75zn9: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_75zn9
17:14 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.migrate_75zn9', metastore_id=None, name='migrate_75zn9', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
17:15 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_75zn9.ucx_tnzeu: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_75zn9/ucx_tnzeu
17:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.DELTA: 'DELTA'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_75zn9.ucx_tnzeu', metastore_id=None, name='ucx_tnzeu', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070218'}, row_filter=None, schema_name='migrate_75zn9', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/migrate_75zn9/ucx_tnzeu', table_constraints=None, table_id=None, table_type=<TableType.MANAGED: 'MANAGED'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
17:15 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_75zn9.ucx_togpb: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_75zn9/ucx_togpb
17:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.JSON: 'JSON'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_75zn9.ucx_togpb', metastore_id=None, name='ucx_togpb', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070218'}, row_filter=None, schema_name='migrate_75zn9', sql_path=None, storage_credential_name=None, storage_location='dbfs:/tmp/ucx_test_U8sA', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
17:15 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_75zn9.ucx_tq0di: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_75zn9/ucx_tq0di
17:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.CSV: 'CSV'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_75zn9.ucx_tq0di', metastore_id=None, name='ucx_tq0di', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070218'}, row_filter=None, schema_name='migrate_75zn9', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/b/DSaW', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
17:16 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_75zn9.ucx_tafdm: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_75zn9/ucx_tafdm
17:16 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_75zn9.ucx_tafdm', metastore_id=None, name='ucx_tafdm', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070218'}, row_filter=None, schema_name='migrate_75zn9', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=<TableType.VIEW: 'VIEW'>, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_75zn9.ucx_tnzeu', view_dependencies=None)
17:16 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_75zn9.ucx_tkjfm: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_75zn9/ucx_tkjfm
17:16 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_75zn9.ucx_tkjfm', metastore_id=None, name='ucx_tkjfm', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070218'}, row_filter=None, schema_name='migrate_75zn9', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=<TableType.VIEW: 'VIEW'>, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_75zn9.ucx_tafdm', view_dependencies=None)
17:17 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=<CatalogType.MANAGED_CATALOG: 'MANAGED_CATALOG'>, comment='', connection_name=None, created_at=1719940623841, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cdrns', isolation_mode=<CatalogIsolationMode.OPEN: 'OPEN'>, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cdrns', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=<CatalogInfoSecurableKind.CATALOG_STANDARD: 'CATALOG_STANDARD'>, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1719940623841, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d')
17:17 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_cdrns.migrate_75zn9: https://DATABRICKS_HOST/explore/data/ucx_cdrns/migrate_75zn9
17:17 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cdrns', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cdrns.migrate_75zn9', metastore_id=None, name='migrate_75zn9', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
17:17 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_svvej: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_svvej
17:17 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_svvej', metastore_id=None, name='ucx_svvej', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
17:18 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.rRsr/config.yml) doesn't exist.
17:18 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration
17:19 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data.
17:19 INFO [databricks.labs.ucx.install] Fetching installations...
17:19 WARNING [databricks.labs.ucx.install] Existing installation at /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.rRsr is corrupted. Skipping...
17:19 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy.
17:19 DEBUG [tests.integration.conftest] Waiting for clusters to start...
17:19 DEBUG [tests.integration.conftest] Waiting for clusters to start...
17:19 INFO [databricks.labs.ucx.install] Installing UCX v0.27.2+6020240702171924
17:19 INFO [databricks.labs.ucx.install] Creating ucx schemas...
17:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
17:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
17:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental
17:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation
17:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental
17:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
17:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
17:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
17:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
17:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter
17:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
17:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
17:20 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
17:20 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.rRsr/README for the next steps.
17:20 DEBUG [databricks.labs.ucx.installer.workflows] starting migrate-tables job: https://DATABRICKS_HOST#job/213377662691383
17:26 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 make_storage_dir fixtures
17:26 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 make_dbfs_data_copy fixtures
17:26 DEBUG [databricks.labs.ucx.mixins.fixtures] removing make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/DSaW
17:26 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 cluster fixtures
17:26 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 catalog fixtures
17:26 DEBUG [databricks.labs.ucx.mixins.fixtures] removing catalog fixture: CatalogInfo(browse_only=False, catalog_type=<CatalogType.MANAGED_CATALOG: 'MANAGED_CATALOG'>, comment='', connection_name=None, created_at=1719940623841, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cdrns', isolation_mode=<CatalogIsolationMode.OPEN: 'OPEN'>, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cdrns', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=<CatalogInfoSecurableKind.CATALOG_STANDARD: 'CATALOG_STANDARD'>, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1719940623841, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d')
17:26 INFO [databricks.labs.ucx.install] Deleting UCX v0.27.2+6020240702171924 from https://DATABRICKS_HOST
17:26 INFO [databricks.labs.ucx.install] Deleting inventory database ucx_svvej
17:26 INFO [databricks.labs.ucx.install] Deleting jobs
17:26 INFO [databricks.labs.ucx.install] Deleting failing job_id=378876543871563.
17:26 INFO [databricks.labs.ucx.install] Deleting migrate-groups job_id=495580131839094.
17:26 INFO [databricks.labs.ucx.install] Deleting scan-tables-in-mounts-experimental job_id=901471361410736.
17:26 INFO [databricks.labs.ucx.install] Deleting migrate-data-reconciliation job_id=686915452022449.
17:26 INFO [databricks.labs.ucx.install] Deleting migrate-groups-experimental job_id=678407480146698.
17:26 INFO [databricks.labs.ucx.install] Deleting migrate-external-tables-ctas job_id=888168290551622.
17:26 INFO [databricks.labs.ucx.install] Deleting migrate-tables-in-mounts-experimental job_id=514017254878961.
17:26 INFO [databricks.labs.ucx.install] Deleting assessment job_id=678036895199687.
17:26 INFO [databricks.labs.ucx.install] Deleting remove-workspace-local-backup-groups job_id=436468672430603.
17:26 INFO [databricks.labs.ucx.install] Deleting experimental-workflow-linter job_id=959957621721702.
17:26 INFO [databricks.labs.ucx.install] Deleting migrate-tables job_id=213377662691383.
17:26 INFO [databricks.labs.ucx.install] Deleting validate-groups-permissions job_id=1032649123038499.
17:26 INFO [databricks.labs.ucx.install] Deleting migrate-external-hiveserde-tables-in-place-experimental job_id=41227504722938.
17:26 INFO [databricks.labs.ucx.install] Deleting cluster policy
17:26 INFO [databricks.labs.ucx.install] Deleting secret scope
17:26 INFO [databricks.labs.ucx.install] UnInstalling UCX complete
17:26 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 workspace user fixtures
17:26 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 account group fixtures
17:26 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 workspace group fixtures
17:26 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures
17:26 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 5 table fixtures
17:26 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.DELTA: 'DELTA'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_75zn9.ucx_tnzeu', metastore_id=None, name='ucx_tnzeu', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070218'}, row_filter=None, schema_name='migrate_75zn9', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/migrate_75zn9/ucx_tnzeu', table_constraints=None, table_id=None, table_type=<TableType.MANAGED: 'MANAGED'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
17:26 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.JSON: 'JSON'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_75zn9.ucx_togpb', metastore_id=None, name='ucx_togpb', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070218'}, row_filter=None, schema_name='migrate_75zn9', sql_path=None, storage_credential_name=None, storage_location='dbfs:/tmp/ucx_test_U8sA', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
17:27 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.CSV: 'CSV'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_75zn9.ucx_tq0di', metastore_id=None, name='ucx_tq0di', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070218'}, row_filter=None, schema_name='migrate_75zn9', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/b/DSaW', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
17:27 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_75zn9.ucx_tafdm', metastore_id=None, name='ucx_tafdm', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070218'}, row_filter=None, schema_name='migrate_75zn9', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=<TableType.VIEW: 'VIEW'>, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_75zn9.ucx_tnzeu', view_dependencies=None)
17:27 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while table TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_75zn9.ucx_tafdm', metastore_id=None, name='ucx_tafdm', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070218'}, row_filter=None, schema_name='migrate_75zn9', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=<TableType.VIEW: 'VIEW'>, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_75zn9.ucx_tnzeu', view_dependencies=None) teardown: [WRONG_COMMAND_FOR_OBJECT_TYPE] The operation DROP TABLE requires a EXTERNAL or MANAGED. But hive_metastore.migrate_75zn9.ucx_tafdm is a VIEW. Use DROP VIEW instead. SQLSTATE: 42809
17:27 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_75zn9.ucx_tkjfm', metastore_id=None, name='ucx_tkjfm', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070218'}, row_filter=None, schema_name='migrate_75zn9', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=<TableType.VIEW: 'VIEW'>, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_75zn9.ucx_tafdm', view_dependencies=None)
17:27 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while table TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_75zn9.ucx_tkjfm', metastore_id=None, name='ucx_tkjfm', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070218'}, row_filter=None, schema_name='migrate_75zn9', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=<TableType.VIEW: 'VIEW'>, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_75zn9.ucx_tafdm', view_dependencies=None) teardown: [WRONG_COMMAND_FOR_OBJECT_TYPE] The operation DROP TABLE requires a EXTERNAL or MANAGED. But hive_metastore.migrate_75zn9.ucx_tkjfm is a VIEW. Use DROP VIEW instead. SQLSTATE: 42809
17:27 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 3 schema fixtures
17:27 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.migrate_75zn9', metastore_id=None, name='migrate_75zn9', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
17:27 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cdrns', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cdrns.migrate_75zn9', metastore_id=None, name='migrate_75zn9', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
17:27 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_svvej', metastore_id=None, name='ucx_svvej', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
[gw7] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python

Running from acceptance #4202

nfx
nfx requested changes Jul 2, 2024
View reviewed changes
Copy link
Collaborator

@nfx nfx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please remove noise from analysis

src/databricks/labs/ucx/source_code/known.json Outdated
},
{
"code": "table-migrate",
"message": "Can't migrate table_name argument in 'self._spark.sql(SparkClient.drop_pk_statement(full_table_name))' because its value cannot be computed"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

noise

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed

src/databricks/labs/ucx/source_code/known.json Outdated
},
{
"code": "table-migrate",
"message": "Can't migrate table_name argument in 'self._spark.sql(SparkClient.set_pk_tk_statement(full_table_name=full_table_name, pk_cols=pk_cols, tk_col=tk_col, pk_name=pk_name))' because its value cannot be computed"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

noise

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed

@ericvergnaud
Copy link
Contributor Author

@nfx noise removed

nfx
nfx requested changes Jul 2, 2024
View reviewed changes
src/databricks/labs/ucx/source_code/known.json Outdated
},
{
"code": "table-migrate",
"message": "SQL query is not supported yet: COMMENT ON TABLE IS NULL"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is noise as well

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed

@ericvergnaud
Merge branch 'main' into whitelist-databricks-feature-engineering
ba2cf29 
# Conflicts:
#	src/databricks/labs/ucx/source_code/known.json
@ericvergnaud
Copy link
Contributor Author

already merged somehow...

@ericvergnaud ericvergnaud deleted the whitelist-databricks-feature-engineering branch July 10, 2024 09:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nfx nfx nfx approved these changes

@dleiva04 dleiva04 Awaiting requested review from dleiva04

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@ericvergnaud @nfx