Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test failure: test_table_migration_job[regular] #1930

Closed
github-actions bot opened this issue Jun 25, 2024 · 1 comment · Fixed by #2129
Closed

Test failure: test_table_migration_job[regular] #1930

github-actions bot opened this issue Jun 25, 2024 · 1 comment · Fixed by #2129
Assignees

Comments

@github-actions
Copy link

❌ test_table_migration_job[regular]: TimeoutError: timed out after 0:20:00: (21m38.029s)
TimeoutError: timed out after 0:20:00:
07:14 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/ahcd
07:14 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.migrate_ohhn3: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_ohhn3
07:14 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.migrate_ohhn3', metastore_id=None, name='migrate_ohhn3', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:14 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_ohhn3.ucx_t250s: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_ohhn3/ucx_t250s
07:14 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.DELTA: 'DELTA'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_ohhn3.ucx_t250s', metastore_id=None, name='ucx_t250s', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024062508'}, row_filter=None, schema_name='migrate_ohhn3', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/migrate_ohhn3/ucx_t250s', table_constraints=None, table_id=None, table_type=<TableType.MANAGED: 'MANAGED'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:14 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_ohhn3.ucx_t9aqt: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_ohhn3/ucx_t9aqt
07:14 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.JSON: 'JSON'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_ohhn3.ucx_t9aqt', metastore_id=None, name='ucx_t9aqt', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024062508'}, row_filter=None, schema_name='migrate_ohhn3', sql_path=None, storage_credential_name=None, storage_location='dbfs:/tmp/ucx_test_Plfl', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:14 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_ohhn3.ucx_tdvqv: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_ohhn3/ucx_tdvqv
07:14 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.CSV: 'CSV'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_ohhn3.ucx_tdvqv', metastore_id=None, name='ucx_tdvqv', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024062508'}, row_filter=None, schema_name='migrate_ohhn3', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/b/ahcd', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:14 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_ohhn3.ucx_thm8y: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_ohhn3/ucx_thm8y
07:14 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_ohhn3.ucx_thm8y', metastore_id=None, name='ucx_thm8y', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024062508'}, row_filter=None, schema_name='migrate_ohhn3', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=<TableType.VIEW: 'VIEW'>, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_ohhn3.ucx_t250s', view_dependencies=None)
07:14 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_ohhn3.ucx_txoio: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_ohhn3/ucx_txoio
07:14 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_ohhn3.ucx_txoio', metastore_id=None, name='ucx_txoio', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024062508'}, row_filter=None, schema_name='migrate_ohhn3', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=<TableType.VIEW: 'VIEW'>, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_ohhn3.ucx_thm8y', view_dependencies=None)
07:14 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=<CatalogType.MANAGED_CATALOG: 'MANAGED_CATALOG'>, comment='', connection_name=None, created_at=1719299673339, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cx459', isolation_mode=<IsolationMode.OPEN: 'OPEN'>, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cx459', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=<CatalogInfoSecurableKind.CATALOG_STANDARD: 'CATALOG_STANDARD'>, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1719299673339, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d')
07:14 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_cx459.migrate_ohhn3: https://DATABRICKS_HOST/explore/data/ucx_cx459/migrate_ohhn3
07:14 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cx459', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cx459.migrate_ohhn3', metastore_id=None, name='migrate_ohhn3', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:14 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_srutz: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_srutz
07:14 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_srutz', metastore_id=None, name='ucx_srutz', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
[gw4] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python
07:14 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/ahcd
07:14 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.migrate_ohhn3: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_ohhn3
07:14 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.migrate_ohhn3', metastore_id=None, name='migrate_ohhn3', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:14 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_ohhn3.ucx_t250s: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_ohhn3/ucx_t250s
07:14 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.DELTA: 'DELTA'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_ohhn3.ucx_t250s', metastore_id=None, name='ucx_t250s', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024062508'}, row_filter=None, schema_name='migrate_ohhn3', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/migrate_ohhn3/ucx_t250s', table_constraints=None, table_id=None, table_type=<TableType.MANAGED: 'MANAGED'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:14 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_ohhn3.ucx_t9aqt: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_ohhn3/ucx_t9aqt
07:14 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.JSON: 'JSON'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_ohhn3.ucx_t9aqt', metastore_id=None, name='ucx_t9aqt', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024062508'}, row_filter=None, schema_name='migrate_ohhn3', sql_path=None, storage_credential_name=None, storage_location='dbfs:/tmp/ucx_test_Plfl', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:14 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_ohhn3.ucx_tdvqv: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_ohhn3/ucx_tdvqv
07:14 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.CSV: 'CSV'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_ohhn3.ucx_tdvqv', metastore_id=None, name='ucx_tdvqv', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024062508'}, row_filter=None, schema_name='migrate_ohhn3', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/b/ahcd', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:14 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_ohhn3.ucx_thm8y: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_ohhn3/ucx_thm8y
07:14 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_ohhn3.ucx_thm8y', metastore_id=None, name='ucx_thm8y', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024062508'}, row_filter=None, schema_name='migrate_ohhn3', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=<TableType.VIEW: 'VIEW'>, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_ohhn3.ucx_t250s', view_dependencies=None)
07:14 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_ohhn3.ucx_txoio: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_ohhn3/ucx_txoio
07:14 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_ohhn3.ucx_txoio', metastore_id=None, name='ucx_txoio', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024062508'}, row_filter=None, schema_name='migrate_ohhn3', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=<TableType.VIEW: 'VIEW'>, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_ohhn3.ucx_thm8y', view_dependencies=None)
07:14 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=<CatalogType.MANAGED_CATALOG: 'MANAGED_CATALOG'>, comment='', connection_name=None, created_at=1719299673339, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cx459', isolation_mode=<IsolationMode.OPEN: 'OPEN'>, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cx459', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=<CatalogInfoSecurableKind.CATALOG_STANDARD: 'CATALOG_STANDARD'>, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1719299673339, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d')
07:14 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_cx459.migrate_ohhn3: https://DATABRICKS_HOST/explore/data/ucx_cx459/migrate_ohhn3
07:14 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cx459', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cx459.migrate_ohhn3', metastore_id=None, name='migrate_ohhn3', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:14 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_srutz: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_srutz
07:14 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_srutz', metastore_id=None, name='ucx_srutz', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:14 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.rSqp/config.yml) doesn't exist.
07:14 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration
07:14 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data.
07:14 INFO [databricks.labs.ucx.install] Fetching installations...
07:14 WARNING [databricks.labs.ucx.install] Existing installation at /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.rSqp is corrupted. Skipping...
07:14 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy.
07:14 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:14 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:14 INFO [databricks.labs.ucx.install] Installing UCX v0.27.2+1120240625071456
07:14 INFO [databricks.labs.ucx.install] Creating ucx schemas...
07:15 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
07:15 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental
07:15 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
07:15 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
07:15 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
07:15 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
07:15 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter
07:15 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
07:15 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation
07:15 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
07:15 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
07:15 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental
07:15 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
07:15 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.rSqp/README for the next steps.
07:15 DEBUG [databricks.labs.ucx.installer.workflows] starting migrate-tables job: https://DATABRICKS_HOST#job/991667432963772
07:14 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/ahcd
07:14 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.migrate_ohhn3: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_ohhn3
07:14 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.migrate_ohhn3', metastore_id=None, name='migrate_ohhn3', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:14 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_ohhn3.ucx_t250s: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_ohhn3/ucx_t250s
07:14 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.DELTA: 'DELTA'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_ohhn3.ucx_t250s', metastore_id=None, name='ucx_t250s', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024062508'}, row_filter=None, schema_name='migrate_ohhn3', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/migrate_ohhn3/ucx_t250s', table_constraints=None, table_id=None, table_type=<TableType.MANAGED: 'MANAGED'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:14 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_ohhn3.ucx_t9aqt: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_ohhn3/ucx_t9aqt
07:14 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.JSON: 'JSON'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_ohhn3.ucx_t9aqt', metastore_id=None, name='ucx_t9aqt', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024062508'}, row_filter=None, schema_name='migrate_ohhn3', sql_path=None, storage_credential_name=None, storage_location='dbfs:/tmp/ucx_test_Plfl', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:14 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_ohhn3.ucx_tdvqv: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_ohhn3/ucx_tdvqv
07:14 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.CSV: 'CSV'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_ohhn3.ucx_tdvqv', metastore_id=None, name='ucx_tdvqv', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024062508'}, row_filter=None, schema_name='migrate_ohhn3', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/b/ahcd', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:14 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_ohhn3.ucx_thm8y: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_ohhn3/ucx_thm8y
07:14 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_ohhn3.ucx_thm8y', metastore_id=None, name='ucx_thm8y', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024062508'}, row_filter=None, schema_name='migrate_ohhn3', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=<TableType.VIEW: 'VIEW'>, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_ohhn3.ucx_t250s', view_dependencies=None)
07:14 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_ohhn3.ucx_txoio: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_ohhn3/ucx_txoio
07:14 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_ohhn3.ucx_txoio', metastore_id=None, name='ucx_txoio', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024062508'}, row_filter=None, schema_name='migrate_ohhn3', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=<TableType.VIEW: 'VIEW'>, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_ohhn3.ucx_thm8y', view_dependencies=None)
07:14 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=<CatalogType.MANAGED_CATALOG: 'MANAGED_CATALOG'>, comment='', connection_name=None, created_at=1719299673339, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cx459', isolation_mode=<IsolationMode.OPEN: 'OPEN'>, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cx459', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=<CatalogInfoSecurableKind.CATALOG_STANDARD: 'CATALOG_STANDARD'>, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1719299673339, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d')
07:14 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_cx459.migrate_ohhn3: https://DATABRICKS_HOST/explore/data/ucx_cx459/migrate_ohhn3
07:14 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cx459', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cx459.migrate_ohhn3', metastore_id=None, name='migrate_ohhn3', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:14 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_srutz: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_srutz
07:14 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_srutz', metastore_id=None, name='ucx_srutz', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:14 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.rSqp/config.yml) doesn't exist.
07:14 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration
07:14 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data.
07:14 INFO [databricks.labs.ucx.install] Fetching installations...
07:14 WARNING [databricks.labs.ucx.install] Existing installation at /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.rSqp is corrupted. Skipping...
07:14 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy.
07:14 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:14 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:14 INFO [databricks.labs.ucx.install] Installing UCX v0.27.2+1120240625071456
07:14 INFO [databricks.labs.ucx.install] Creating ucx schemas...
07:15 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
07:15 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental
07:15 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
07:15 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
07:15 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
07:15 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
07:15 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter
07:15 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
07:15 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation
07:15 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
07:15 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
07:15 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental
07:15 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
07:15 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.rSqp/README for the next steps.
07:15 DEBUG [databricks.labs.ucx.installer.workflows] starting migrate-tables job: https://DATABRICKS_HOST#job/991667432963772
07:35 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 make_storage_dir fixtures
07:35 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 make_dbfs_data_copy fixtures
07:35 DEBUG [databricks.labs.ucx.mixins.fixtures] removing make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/ahcd
07:35 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 cluster fixtures
07:35 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 catalog fixtures
07:35 DEBUG [databricks.labs.ucx.mixins.fixtures] removing catalog fixture: CatalogInfo(browse_only=False, catalog_type=<CatalogType.MANAGED_CATALOG: 'MANAGED_CATALOG'>, comment='', connection_name=None, created_at=1719299673339, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cx459', isolation_mode=<IsolationMode.OPEN: 'OPEN'>, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cx459', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=<CatalogInfoSecurableKind.CATALOG_STANDARD: 'CATALOG_STANDARD'>, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1719299673339, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d')
07:35 INFO [databricks.labs.ucx.install] Deleting UCX v0.27.2+1120240625071456 from https://DATABRICKS_HOST
07:35 INFO [databricks.labs.ucx.install] Deleting inventory database ucx_srutz
07:35 INFO [databricks.labs.ucx.install] Deleting jobs
07:35 INFO [databricks.labs.ucx.install] Deleting migrate-groups job_id=1050128164411083.
07:35 INFO [databricks.labs.ucx.install] Deleting migrate-groups-experimental job_id=143260075558555.
07:35 INFO [databricks.labs.ucx.install] Deleting migrate-external-hiveserde-tables-in-place-experimental job_id=727894841111343.
07:35 INFO [databricks.labs.ucx.install] Deleting migrate-tables-in-mounts-experimental job_id=675554858758246.
07:35 INFO [databricks.labs.ucx.install] Deleting failing job_id=537078797864767.
07:35 INFO [databricks.labs.ucx.install] Deleting remove-workspace-local-backup-groups job_id=1047856248446924.
07:35 INFO [databricks.labs.ucx.install] Deleting experimental-workflow-linter job_id=266837503193869.
07:35 INFO [databricks.labs.ucx.install] Deleting validate-groups-permissions job_id=1001838687281421.
07:35 INFO [databricks.labs.ucx.install] Deleting migrate-data-reconciliation job_id=124070441881511.
07:35 INFO [databricks.labs.ucx.install] Deleting migrate-external-tables-ctas job_id=772012863748265.
07:35 INFO [databricks.labs.ucx.install] Deleting migrate-tables job_id=991667432963772.
07:35 INFO [databricks.labs.ucx.install] Deleting scan-tables-in-mounts-experimental job_id=978816194719827.
07:35 INFO [databricks.labs.ucx.install] Deleting assessment job_id=692370381514253.
07:35 INFO [databricks.labs.ucx.install] Deleting cluster policy
07:35 INFO [databricks.labs.ucx.install] Deleting secret scope
07:35 INFO [databricks.labs.ucx.install] UnInstalling UCX complete
07:35 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 workspace user fixtures
07:35 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 account group fixtures
07:35 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 workspace group fixtures
07:35 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures
07:35 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 5 table fixtures
07:35 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.DELTA: 'DELTA'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_ohhn3.ucx_t250s', metastore_id=None, name='ucx_t250s', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024062508'}, row_filter=None, schema_name='migrate_ohhn3', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/migrate_ohhn3/ucx_t250s', table_constraints=None, table_id=None, table_type=<TableType.MANAGED: 'MANAGED'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:35 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.JSON: 'JSON'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_ohhn3.ucx_t9aqt', metastore_id=None, name='ucx_t9aqt', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024062508'}, row_filter=None, schema_name='migrate_ohhn3', sql_path=None, storage_credential_name=None, storage_location='dbfs:/tmp/ucx_test_Plfl', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:35 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.CSV: 'CSV'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_ohhn3.ucx_tdvqv', metastore_id=None, name='ucx_tdvqv', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024062508'}, row_filter=None, schema_name='migrate_ohhn3', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/b/ahcd', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:35 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_ohhn3.ucx_thm8y', metastore_id=None, name='ucx_thm8y', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024062508'}, row_filter=None, schema_name='migrate_ohhn3', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=<TableType.VIEW: 'VIEW'>, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_ohhn3.ucx_t250s', view_dependencies=None)
07:35 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while table TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_ohhn3.ucx_thm8y', metastore_id=None, name='ucx_thm8y', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024062508'}, row_filter=None, schema_name='migrate_ohhn3', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=<TableType.VIEW: 'VIEW'>, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_ohhn3.ucx_t250s', view_dependencies=None) teardown: [WRONG_COMMAND_FOR_OBJECT_TYPE] The operation DROP TABLE requires a EXTERNAL or MANAGED. But hive_metastore.migrate_ohhn3.ucx_thm8y is a VIEW. Use DROP VIEW instead. SQLSTATE: 42809
07:35 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_ohhn3.ucx_txoio', metastore_id=None, name='ucx_txoio', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024062508'}, row_filter=None, schema_name='migrate_ohhn3', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=<TableType.VIEW: 'VIEW'>, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_ohhn3.ucx_thm8y', view_dependencies=None)
07:35 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while table TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_ohhn3.ucx_txoio', metastore_id=None, name='ucx_txoio', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024062508'}, row_filter=None, schema_name='migrate_ohhn3', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=<TableType.VIEW: 'VIEW'>, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_ohhn3.ucx_thm8y', view_dependencies=None) teardown: [WRONG_COMMAND_FOR_OBJECT_TYPE] The operation DROP TABLE requires a EXTERNAL or MANAGED. But hive_metastore.migrate_ohhn3.ucx_txoio is a VIEW. Use DROP VIEW instead. SQLSTATE: 42809
07:35 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 3 schema fixtures
07:35 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.migrate_ohhn3', metastore_id=None, name='migrate_ohhn3', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:35 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cx459', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cx459.migrate_ohhn3', metastore_id=None, name='migrate_ohhn3', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:36 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_srutz', metastore_id=None, name='ucx_srutz', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
[gw4] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python

Running from nightly #102

@github-actions github-actions bot added the bug label Jun 25, 2024
@asnare asnare self-assigned this Jun 27, 2024
Copy link
Author

github-actions bot commented Jul 1, 2024

❌ test_table_migration_job[regular]: TimeoutError: timed out after 0:20:00: (21m52.317s)
TimeoutError: timed out after 0:20:00:
07:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/Vul0
07:15 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.migrate_qw4qe: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_qw4qe
07:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.migrate_qw4qe', metastore_id=None, name='migrate_qw4qe', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:15 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_qw4qe.ucx_tisrk: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_qw4qe/ucx_tisrk
07:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.DELTA: 'DELTA'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qw4qe.ucx_tisrk', metastore_id=None, name='ucx_tisrk', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070108'}, row_filter=None, schema_name='migrate_qw4qe', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/migrate_qw4qe/ucx_tisrk', table_constraints=None, table_id=None, table_type=<TableType.MANAGED: 'MANAGED'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:15 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_qw4qe.ucx_txfih: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_qw4qe/ucx_txfih
07:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.JSON: 'JSON'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qw4qe.ucx_txfih', metastore_id=None, name='ucx_txfih', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070108'}, row_filter=None, schema_name='migrate_qw4qe', sql_path=None, storage_credential_name=None, storage_location='dbfs:/tmp/ucx_test_N144', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:15 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_qw4qe.ucx_te9ik: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_qw4qe/ucx_te9ik
07:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.CSV: 'CSV'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qw4qe.ucx_te9ik', metastore_id=None, name='ucx_te9ik', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070108'}, row_filter=None, schema_name='migrate_qw4qe', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/b/Vul0', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:15 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_qw4qe.ucx_tlskj: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_qw4qe/ucx_tlskj
07:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qw4qe.ucx_tlskj', metastore_id=None, name='ucx_tlskj', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070108'}, row_filter=None, schema_name='migrate_qw4qe', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=<TableType.VIEW: 'VIEW'>, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_qw4qe.ucx_tisrk', view_dependencies=None)
07:15 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_qw4qe.ucx_tvirs: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_qw4qe/ucx_tvirs
07:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qw4qe.ucx_tvirs', metastore_id=None, name='ucx_tvirs', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070108'}, row_filter=None, schema_name='migrate_qw4qe', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=<TableType.VIEW: 'VIEW'>, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_qw4qe.ucx_tlskj', view_dependencies=None)
07:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=<CatalogType.MANAGED_CATALOG: 'MANAGED_CATALOG'>, comment='', connection_name=None, created_at=1719818131757, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cwwto', isolation_mode=<IsolationMode.OPEN: 'OPEN'>, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cwwto', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=<CatalogInfoSecurableKind.CATALOG_STANDARD: 'CATALOG_STANDARD'>, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1719818131757, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d')
07:15 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_cwwto.migrate_qw4qe: https://DATABRICKS_HOST/explore/data/ucx_cwwto/migrate_qw4qe
07:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cwwto', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cwwto.migrate_qw4qe', metastore_id=None, name='migrate_qw4qe', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:15 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_s9bvq: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_s9bvq
07:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_s9bvq', metastore_id=None, name='ucx_s9bvq', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
[gw6] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python
07:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/Vul0
07:15 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.migrate_qw4qe: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_qw4qe
07:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.migrate_qw4qe', metastore_id=None, name='migrate_qw4qe', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:15 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_qw4qe.ucx_tisrk: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_qw4qe/ucx_tisrk
07:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.DELTA: 'DELTA'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qw4qe.ucx_tisrk', metastore_id=None, name='ucx_tisrk', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070108'}, row_filter=None, schema_name='migrate_qw4qe', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/migrate_qw4qe/ucx_tisrk', table_constraints=None, table_id=None, table_type=<TableType.MANAGED: 'MANAGED'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:15 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_qw4qe.ucx_txfih: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_qw4qe/ucx_txfih
07:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.JSON: 'JSON'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qw4qe.ucx_txfih', metastore_id=None, name='ucx_txfih', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070108'}, row_filter=None, schema_name='migrate_qw4qe', sql_path=None, storage_credential_name=None, storage_location='dbfs:/tmp/ucx_test_N144', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:15 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_qw4qe.ucx_te9ik: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_qw4qe/ucx_te9ik
07:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.CSV: 'CSV'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qw4qe.ucx_te9ik', metastore_id=None, name='ucx_te9ik', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070108'}, row_filter=None, schema_name='migrate_qw4qe', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/b/Vul0', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:15 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_qw4qe.ucx_tlskj: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_qw4qe/ucx_tlskj
07:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qw4qe.ucx_tlskj', metastore_id=None, name='ucx_tlskj', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070108'}, row_filter=None, schema_name='migrate_qw4qe', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=<TableType.VIEW: 'VIEW'>, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_qw4qe.ucx_tisrk', view_dependencies=None)
07:15 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_qw4qe.ucx_tvirs: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_qw4qe/ucx_tvirs
07:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qw4qe.ucx_tvirs', metastore_id=None, name='ucx_tvirs', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070108'}, row_filter=None, schema_name='migrate_qw4qe', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=<TableType.VIEW: 'VIEW'>, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_qw4qe.ucx_tlskj', view_dependencies=None)
07:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=<CatalogType.MANAGED_CATALOG: 'MANAGED_CATALOG'>, comment='', connection_name=None, created_at=1719818131757, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cwwto', isolation_mode=<IsolationMode.OPEN: 'OPEN'>, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cwwto', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=<CatalogInfoSecurableKind.CATALOG_STANDARD: 'CATALOG_STANDARD'>, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1719818131757, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d')
07:15 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_cwwto.migrate_qw4qe: https://DATABRICKS_HOST/explore/data/ucx_cwwto/migrate_qw4qe
07:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cwwto', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cwwto.migrate_qw4qe', metastore_id=None, name='migrate_qw4qe', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:15 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_s9bvq: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_s9bvq
07:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_s9bvq', metastore_id=None, name='ucx_s9bvq', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:15 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.XpIz/config.yml) doesn't exist.
07:15 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration
07:15 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data.
07:15 INFO [databricks.labs.ucx.install] Fetching installations...
07:16 WARNING [databricks.labs.ucx.install] Existing installation at /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.XpIz is corrupted. Skipping...
07:16 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy.
07:16 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:16 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:16 INFO [databricks.labs.ucx.install] Installing UCX v0.27.2+1320240701071608
07:16 INFO [databricks.labs.ucx.install] Creating ucx schemas...
07:16 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation
07:16 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
07:16 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
07:16 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
07:16 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental
07:16 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter
07:16 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
07:16 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
07:16 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
07:16 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental
07:16 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
07:16 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
07:16 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
07:16 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.XpIz/README for the next steps.
07:16 DEBUG [databricks.labs.ucx.installer.workflows] starting migrate-tables job: https://DATABRICKS_HOST#job/191647837048877
07:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/Vul0
07:15 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.migrate_qw4qe: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_qw4qe
07:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.migrate_qw4qe', metastore_id=None, name='migrate_qw4qe', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:15 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_qw4qe.ucx_tisrk: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_qw4qe/ucx_tisrk
07:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.DELTA: 'DELTA'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qw4qe.ucx_tisrk', metastore_id=None, name='ucx_tisrk', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070108'}, row_filter=None, schema_name='migrate_qw4qe', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/migrate_qw4qe/ucx_tisrk', table_constraints=None, table_id=None, table_type=<TableType.MANAGED: 'MANAGED'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:15 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_qw4qe.ucx_txfih: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_qw4qe/ucx_txfih
07:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.JSON: 'JSON'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qw4qe.ucx_txfih', metastore_id=None, name='ucx_txfih', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070108'}, row_filter=None, schema_name='migrate_qw4qe', sql_path=None, storage_credential_name=None, storage_location='dbfs:/tmp/ucx_test_N144', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:15 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_qw4qe.ucx_te9ik: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_qw4qe/ucx_te9ik
07:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.CSV: 'CSV'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qw4qe.ucx_te9ik', metastore_id=None, name='ucx_te9ik', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070108'}, row_filter=None, schema_name='migrate_qw4qe', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/b/Vul0', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:15 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_qw4qe.ucx_tlskj: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_qw4qe/ucx_tlskj
07:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qw4qe.ucx_tlskj', metastore_id=None, name='ucx_tlskj', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070108'}, row_filter=None, schema_name='migrate_qw4qe', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=<TableType.VIEW: 'VIEW'>, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_qw4qe.ucx_tisrk', view_dependencies=None)
07:15 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.migrate_qw4qe.ucx_tvirs: https://DATABRICKS_HOST/explore/data/hive_metastore/migrate_qw4qe/ucx_tvirs
07:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qw4qe.ucx_tvirs', metastore_id=None, name='ucx_tvirs', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070108'}, row_filter=None, schema_name='migrate_qw4qe', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=<TableType.VIEW: 'VIEW'>, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_qw4qe.ucx_tlskj', view_dependencies=None)
07:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added catalog fixture: CatalogInfo(browse_only=False, catalog_type=<CatalogType.MANAGED_CATALOG: 'MANAGED_CATALOG'>, comment='', connection_name=None, created_at=1719818131757, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cwwto', isolation_mode=<IsolationMode.OPEN: 'OPEN'>, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cwwto', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=<CatalogInfoSecurableKind.CATALOG_STANDARD: 'CATALOG_STANDARD'>, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1719818131757, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d')
07:15 INFO [databricks.labs.ucx.mixins.fixtures] Schema ucx_cwwto.migrate_qw4qe: https://DATABRICKS_HOST/explore/data/ucx_cwwto/migrate_qw4qe
07:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cwwto', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cwwto.migrate_qw4qe', metastore_id=None, name='migrate_qw4qe', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:15 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_s9bvq: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_s9bvq
07:15 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_s9bvq', metastore_id=None, name='ucx_s9bvq', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:15 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.XpIz/config.yml) doesn't exist.
07:15 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration
07:15 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data.
07:15 INFO [databricks.labs.ucx.install] Fetching installations...
07:16 WARNING [databricks.labs.ucx.install] Existing installation at /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.XpIz is corrupted. Skipping...
07:16 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy.
07:16 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:16 DEBUG [tests.integration.conftest] Waiting for clusters to start...
07:16 INFO [databricks.labs.ucx.install] Installing UCX v0.27.2+1320240701071608
07:16 INFO [databricks.labs.ucx.install] Creating ucx schemas...
07:16 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation
07:16 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
07:16 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
07:16 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
07:16 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental
07:16 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter
07:16 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
07:16 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
07:16 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
07:16 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental
07:16 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
07:16 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
07:16 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
07:16 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.XpIz/README for the next steps.
07:16 DEBUG [databricks.labs.ucx.installer.workflows] starting migrate-tables job: https://DATABRICKS_HOST#job/191647837048877
07:36 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 make_storage_dir fixtures
07:36 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 make_dbfs_data_copy fixtures
07:36 DEBUG [databricks.labs.ucx.mixins.fixtures] removing make_dbfs_data_copy fixture: dbfs:/mnt/TEST_MOUNT_NAME/a/b/Vul0
07:36 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 cluster fixtures
07:36 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 catalog fixtures
07:36 DEBUG [databricks.labs.ucx.mixins.fixtures] removing catalog fixture: CatalogInfo(browse_only=False, catalog_type=<CatalogType.MANAGED_CATALOG: 'MANAGED_CATALOG'>, comment='', connection_name=None, created_at=1719818131757, created_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cwwto', isolation_mode=<IsolationMode.OPEN: 'OPEN'>, metastore_id='8952c1e3-b265-4adf-98c3-6f755e2e1453', name='ucx_cwwto', options=None, owner='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', properties=None, provider_name=None, provisioning_info=None, securable_kind=<CatalogInfoSecurableKind.CATALOG_STANDARD: 'CATALOG_STANDARD'>, securable_type='CATALOG', share_name=None, storage_location=None, storage_root=None, updated_at=1719818131757, updated_by='0a330eb5-dd51-4d97-b6e4-c474356b1d5d')
07:36 INFO [databricks.labs.ucx.install] Deleting UCX v0.27.2+1320240701071608 from https://DATABRICKS_HOST
07:36 INFO [databricks.labs.ucx.install] Deleting inventory database ucx_s9bvq
07:36 INFO [databricks.labs.ucx.install] Deleting jobs
07:36 INFO [databricks.labs.ucx.install] Deleting migrate-data-reconciliation job_id=116638703378763.
07:36 INFO [databricks.labs.ucx.install] Deleting migrate-tables job_id=191647837048877.
07:36 INFO [databricks.labs.ucx.install] Deleting migrate-tables-in-mounts-experimental job_id=436242349524147.
07:36 INFO [databricks.labs.ucx.install] Deleting migrate-external-tables-ctas job_id=946006745284361.
07:36 INFO [databricks.labs.ucx.install] Deleting migrate-groups-experimental job_id=1018717195622275.
07:36 INFO [databricks.labs.ucx.install] Deleting experimental-workflow-linter job_id=438185648883023.
07:36 INFO [databricks.labs.ucx.install] Deleting failing job_id=87228517371129.
07:36 INFO [databricks.labs.ucx.install] Deleting remove-workspace-local-backup-groups job_id=1037126949287739.
07:36 INFO [databricks.labs.ucx.install] Deleting migrate-external-hiveserde-tables-in-place-experimental job_id=90409082879275.
07:36 INFO [databricks.labs.ucx.install] Deleting scan-tables-in-mounts-experimental job_id=240166311459885.
07:36 INFO [databricks.labs.ucx.install] Deleting assessment job_id=175475860435545.
07:36 INFO [databricks.labs.ucx.install] Deleting validate-groups-permissions job_id=982452748750080.
07:36 INFO [databricks.labs.ucx.install] Deleting migrate-groups job_id=232449754778088.
07:36 INFO [databricks.labs.ucx.install] Deleting cluster policy
07:36 INFO [databricks.labs.ucx.install] Deleting secret scope
07:37 INFO [databricks.labs.ucx.install] UnInstalling UCX complete
07:37 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 workspace user fixtures
07:37 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 account group fixtures
07:37 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 workspace group fixtures
07:37 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures
07:37 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 5 table fixtures
07:37 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.DELTA: 'DELTA'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qw4qe.ucx_tisrk', metastore_id=None, name='ucx_tisrk', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070108'}, row_filter=None, schema_name='migrate_qw4qe', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/migrate_qw4qe/ucx_tisrk', table_constraints=None, table_id=None, table_type=<TableType.MANAGED: 'MANAGED'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:37 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.JSON: 'JSON'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qw4qe.ucx_txfih', metastore_id=None, name='ucx_txfih', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070108'}, row_filter=None, schema_name='migrate_qw4qe', sql_path=None, storage_credential_name=None, storage_location='dbfs:/tmp/ucx_test_N144', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:37 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.CSV: 'CSV'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qw4qe.ucx_te9ik', metastore_id=None, name='ucx_te9ik', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070108'}, row_filter=None, schema_name='migrate_qw4qe', sql_path=None, storage_credential_name=None, storage_location='dbfs:/mnt/TEST_MOUNT_NAME/a/b/Vul0', table_constraints=None, table_id=None, table_type=<TableType.EXTERNAL: 'EXTERNAL'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
07:37 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qw4qe.ucx_tlskj', metastore_id=None, name='ucx_tlskj', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070108'}, row_filter=None, schema_name='migrate_qw4qe', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=<TableType.VIEW: 'VIEW'>, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_qw4qe.ucx_tisrk', view_dependencies=None)
07:37 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while table TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qw4qe.ucx_tlskj', metastore_id=None, name='ucx_tlskj', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070108'}, row_filter=None, schema_name='migrate_qw4qe', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=<TableType.VIEW: 'VIEW'>, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_qw4qe.ucx_tisrk', view_dependencies=None) teardown: [WRONG_COMMAND_FOR_OBJECT_TYPE] The operation DROP TABLE requires a EXTERNAL or MANAGED. But hive_metastore.migrate_qw4qe.ucx_tlskj is a VIEW. Use DROP VIEW instead. SQLSTATE: 42809
07:37 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qw4qe.ucx_tvirs', metastore_id=None, name='ucx_tvirs', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070108'}, row_filter=None, schema_name='migrate_qw4qe', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=<TableType.VIEW: 'VIEW'>, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_qw4qe.ucx_tlskj', view_dependencies=None)
07:37 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while table TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=None, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.migrate_qw4qe.ucx_tvirs', metastore_id=None, name='ucx_tvirs', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070108'}, row_filter=None, schema_name='migrate_qw4qe', sql_path=None, storage_credential_name=None, storage_location=None, table_constraints=None, table_id=None, table_type=<TableType.VIEW: 'VIEW'>, updated_at=None, updated_by=None, view_definition='SELECT * FROM hive_metastore.migrate_qw4qe.ucx_tlskj', view_dependencies=None) teardown: [WRONG_COMMAND_FOR_OBJECT_TYPE] The operation DROP TABLE requires a EXTERNAL or MANAGED. But hive_metastore.migrate_qw4qe.ucx_tvirs is a VIEW. Use DROP VIEW instead. SQLSTATE: 42809
07:37 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 3 schema fixtures
07:37 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.migrate_qw4qe', metastore_id=None, name='migrate_qw4qe', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:37 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='ucx_cwwto', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='ucx_cwwto.migrate_qw4qe', metastore_id=None, name='migrate_qw4qe', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
07:37 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_s9bvq', metastore_id=None, name='ucx_s9bvq', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
[gw6] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python

Running from nightly #108

@nfx nfx assigned nfx and unassigned asnare Jul 10, 2024
nfx added a commit that referenced this issue Jul 10, 2024
@nfx nfx closed this as completed in #2129 Jul 10, 2024
@nfx nfx closed this as completed in f52a018 Jul 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants