Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tidy up whitespace #1923

Merged
merged 7 commits into from
Jul 5, 2024
Merged

Tidy up whitespace #1923

merged 7 commits into from
Jul 5, 2024

Conversation

asnare
Copy link
Contributor

@asnare asnare commented Jun 20, 2024

Changes

This PR follows on from #1778 by cleaning up files that use tabs, have trailing whitespace, or are missing EOL at the end of files. By fixing these in one go other PRs become less noisy. (Sample files used in tests have been left alone and are not affected by this PR.)

@asnare asnare self-assigned this Jun 20, 2024
@asnare asnare requested review from a team and FastLee June 20, 2024 09:13
@asnare asnare temporarily deployed to account-admin June 20, 2024 09:13 — with GitHub Actions Inactive
Copy link

codecov bot commented Jun 20, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 89.59%. Comparing base (69acd4a) to head (6af0fdf).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #1923   +/-   ##
=======================================
  Coverage   89.59%   89.59%           
=======================================
  Files          95       95           
  Lines       12347    12347           
  Branches     2166     2166           
=======================================
  Hits        11062    11062           
  Misses        864      864           
  Partials      421      421           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link

github-actions bot commented Jun 20, 2024

❌ 197/198 passed, 2 flaky, 1 failed, 23 skipped, 4h34m38s total

❌ test_running_real_validate_groups_permissions_job: TimeoutError: Timed out after 0:05:00 (6m48.022s)
TimeoutError: Timed out after 0:05:00
[gw1] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python
08:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace user fixture: User(active=True, display_name='[email protected]', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='[email protected]')], entitlements=[], external_id=None, groups=[], id='1216970546986544', name=Name(family_name=None, given_name='[email protected]'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='[email protected]')
08:06 INFO [databricks.labs.ucx.mixins.fixtures] Workspace group ucx_GisUC: https://DATABRICKS_HOST#setting/accounts/groups/211537985122044
08:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace group fixture: Group(display_name='ucx_GisUC', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='211537985122044', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/1216970546986544', type=None, value='1216970546986544')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
08:06 INFO [databricks.labs.ucx.mixins.fixtures] Account group ucx_GisUC: https://accounts.CLOUD_ENVdatabricks.net/users/groups/726939818273819/members
08:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added account group fixture: Group(display_name='ucx_GisUC', entitlements=[], external_id=None, groups=[], id='726939818273819', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/1216970546986544', type=None, value='1216970546986544')], meta=None, roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
08:06 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_snhlg: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_snhlg
08:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_snhlg', metastore_id=None, name='ucx_snhlg', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
08:06 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.ucx_snhlg.ucx_tzpi2: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_snhlg/ucx_tzpi2
08:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.DELTA: 'DELTA'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.ucx_snhlg.ucx_tzpi2', metastore_id=None, name='ucx_tzpi2', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070209'}, row_filter=None, schema_name='ucx_snhlg', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/ucx_snhlg/ucx_tzpi2', table_constraints=None, table_id=None, table_type=<TableType.MANAGED: 'MANAGED'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
08:06 INFO [databricks.labs.ucx.mixins.fixtures] Query Created ucx_query_QsosT: https://DATABRICKS_HOST/sql/editor/a4925302-ad5b-406c-8015-e7070a4d56ee
08:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added query fixture: Query(can_edit=None, created_at='2024-07-02T08:06:34Z', data_source_id=None, description='TEST QUERY FOR UCX', id='a4925302-ad5b-406c-8015-e7070a4d56ee', is_archived=False, is_draft=False, is_favorite=False, is_safe=True, last_modified_by=User(email='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', id=481119220561874, name='labs-account-admin-identity'), last_modified_by_id=None, latest_query_data_id=None, name='ucx_query_QsosT', options=QueryOptions(catalog=None, moved_to_trash_at=None, parameters=[], schema=None), parent='folders/2080761702917835', permission_tier=None, query='SELECT * FROM ucx_snhlg.ucx_tzpi2', query_hash=None, run_as_role=<RunAsRole.OWNER: 'owner'>, tags=['original_query_tag'], updated_at='2024-07-02T08:06:34Z', user=User(email='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', id=481119220561874, name='labs-account-admin-identity'), user_id=481119220561874, visualizations=[Visualization(created_at='2024-07-02T08:06:33Z', description='', id='6dfe2c2b-1244-4262-85ad-fbde805665a1', name='Results', options={'version': 2}, query=None, type='TABLE', updated_at='2024-07-02T08:06:33Z')])
08:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added query permissions fixture: a4925302-ad5b-406c-8015-e7070a4d56ee [user_name 0a330eb5-dd51-4d97-b6e4-c474356b1d5d CAN_MANAGE, group_name admins CAN_MANAGE] -> [group_name ucx_GisUC CAN_EDIT]
08:06 INFO [databricks.labs.ucx.mixins.fixtures] Cluster policy: https://DATABRICKS_HOST#setting/clusters/cluster-policies/view/00090F0E9ABE179A
08:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster policy fixture: CreatePolicyResponse(policy_id='00090F0E9ABE179A')
08:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster_policy permissions fixture: 00090F0E9ABE179A [group_name admins CAN_USE] -> [group_name ucx_GisUC CAN_USE]
08:06 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_svt6d: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_svt6d
08:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_svt6d', metastore_id=None, name='ucx_svt6d', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
08:06 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.ucx_svt6d.ucx_thh1o: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_svt6d/ucx_thh1o
08:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.DELTA: 'DELTA'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.ucx_svt6d.ucx_thh1o', metastore_id=None, name='ucx_thh1o', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070209'}, row_filter=None, schema_name='ucx_svt6d', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/ucx_svt6d/ucx_thh1o', table_constraints=None, table_id=None, table_type=<TableType.MANAGED: 'MANAGED'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
08:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added secret scope fixture: sdk-Izbe
08:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added secret scope acl fixture: ('sdk-Izbe', 'ucx_GisUC')
08:06 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_so69q: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_so69q
08:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_so69q', metastore_id=None, name='ucx_so69q', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
08:06 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.15id/config.yml) doesn't exist.
08:06 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration
08:06 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data.
08:06 INFO [databricks.labs.ucx.install] Fetching installations...
08:06 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy.
08:06 DEBUG [tests.integration.conftest] Waiting for clusters to start...
08:06 DEBUG [tests.integration.conftest] Waiting for clusters to start...
08:06 INFO [databricks.labs.ucx.install] Installing UCX v0.27.2+1920240702080653
08:06 INFO [databricks.labs.ucx.install] Creating ucx schemas...
08:07 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
08:07 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
08:07 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
08:07 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
08:07 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation
08:07 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
08:07 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
08:07 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter
08:07 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental
08:07 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
08:07 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
08:07 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
08:07 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental
08:07 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.15id/README for the next steps.
08:07 DEBUG [databricks.labs.ucx.workspace_access.manager] Crawling permissions
08:07 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_so69q.grants] fetching grants inventory
08:08 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_so69q.grants] crawling new batch for grants
08:08 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_so69q.tables] fetching tables inventory
08:08 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_so69q.tables] crawling new batch for tables
08:08 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.ucx_svt6d] listing tables
08:08 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.ucx_svt6d.ucx_thh1o] fetching table metadata
08:08 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_so69q.tables] found 1 new records for tables
08:08 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_so69q.udfs] fetching udfs inventory
08:08 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_so69q.udfs] crawling new batch for udfs
08:08 DEBUG [databricks.labs.ucx.hive_metastore.udfs] [hive_metastore.ucx_svt6d] listing udfs
08:08 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_so69q.udfs] found 0 new records for udfs
08:09 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_so69q.grants] found 5 new records for grants
08:09 INFO [databricks.labs.ucx.workspace_access.manager] Starting to crawl permissions. Total tasks: 5
08:09 INFO [databricks.labs.ucx.workspace_access.manager] Total crawled permissions: 5
08:09 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_so69q.permissions] found 5 new records for permissions
08:09 INFO [databricks.labs.ucx.workspace_access.manager] Successfully saved the items to inventory table
08:09 INFO [databricks.labs.ucx.workspace_access.manager] Saved 5 to hive_metastore.ucx_so69q.permissions
08:09 DEBUG [databricks.labs.ucx.installer.workflows] starting validate-groups-permissions job: https://DATABRICKS_HOST#job/662269344681005
08:11 INFO [databricks.labs.ucx.installer.workflows] ---------- REMOTE LOGS --------------
08:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace user fixture: User(active=True, display_name='[email protected]', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='[email protected]')], entitlements=[], external_id=None, groups=[], id='1216970546986544', name=Name(family_name=None, given_name='[email protected]'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='[email protected]')
08:06 INFO [databricks.labs.ucx.mixins.fixtures] Workspace group ucx_GisUC: https://DATABRICKS_HOST#setting/accounts/groups/211537985122044
08:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace group fixture: Group(display_name='ucx_GisUC', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='211537985122044', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/1216970546986544', type=None, value='1216970546986544')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
08:06 INFO [databricks.labs.ucx.mixins.fixtures] Account group ucx_GisUC: https://accounts.CLOUD_ENVdatabricks.net/users/groups/726939818273819/members
08:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added account group fixture: Group(display_name='ucx_GisUC', entitlements=[], external_id=None, groups=[], id='726939818273819', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/1216970546986544', type=None, value='1216970546986544')], meta=None, roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
08:06 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_snhlg: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_snhlg
08:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_snhlg', metastore_id=None, name='ucx_snhlg', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
08:06 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.ucx_snhlg.ucx_tzpi2: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_snhlg/ucx_tzpi2
08:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.DELTA: 'DELTA'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.ucx_snhlg.ucx_tzpi2', metastore_id=None, name='ucx_tzpi2', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070209'}, row_filter=None, schema_name='ucx_snhlg', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/ucx_snhlg/ucx_tzpi2', table_constraints=None, table_id=None, table_type=<TableType.MANAGED: 'MANAGED'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
08:06 INFO [databricks.labs.ucx.mixins.fixtures] Query Created ucx_query_QsosT: https://DATABRICKS_HOST/sql/editor/a4925302-ad5b-406c-8015-e7070a4d56ee
08:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added query fixture: Query(can_edit=None, created_at='2024-07-02T08:06:34Z', data_source_id=None, description='TEST QUERY FOR UCX', id='a4925302-ad5b-406c-8015-e7070a4d56ee', is_archived=False, is_draft=False, is_favorite=False, is_safe=True, last_modified_by=User(email='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', id=481119220561874, name='labs-account-admin-identity'), last_modified_by_id=None, latest_query_data_id=None, name='ucx_query_QsosT', options=QueryOptions(catalog=None, moved_to_trash_at=None, parameters=[], schema=None), parent='folders/2080761702917835', permission_tier=None, query='SELECT * FROM ucx_snhlg.ucx_tzpi2', query_hash=None, run_as_role=<RunAsRole.OWNER: 'owner'>, tags=['original_query_tag'], updated_at='2024-07-02T08:06:34Z', user=User(email='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', id=481119220561874, name='labs-account-admin-identity'), user_id=481119220561874, visualizations=[Visualization(created_at='2024-07-02T08:06:33Z', description='', id='6dfe2c2b-1244-4262-85ad-fbde805665a1', name='Results', options={'version': 2}, query=None, type='TABLE', updated_at='2024-07-02T08:06:33Z')])
08:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added query permissions fixture: a4925302-ad5b-406c-8015-e7070a4d56ee [user_name 0a330eb5-dd51-4d97-b6e4-c474356b1d5d CAN_MANAGE, group_name admins CAN_MANAGE] -> [group_name ucx_GisUC CAN_EDIT]
08:06 INFO [databricks.labs.ucx.mixins.fixtures] Cluster policy: https://DATABRICKS_HOST#setting/clusters/cluster-policies/view/00090F0E9ABE179A
08:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster policy fixture: CreatePolicyResponse(policy_id='00090F0E9ABE179A')
08:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster_policy permissions fixture: 00090F0E9ABE179A [group_name admins CAN_USE] -> [group_name ucx_GisUC CAN_USE]
08:06 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_svt6d: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_svt6d
08:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_svt6d', metastore_id=None, name='ucx_svt6d', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
08:06 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.ucx_svt6d.ucx_thh1o: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_svt6d/ucx_thh1o
08:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.DELTA: 'DELTA'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.ucx_svt6d.ucx_thh1o', metastore_id=None, name='ucx_thh1o', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070209'}, row_filter=None, schema_name='ucx_svt6d', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/ucx_svt6d/ucx_thh1o', table_constraints=None, table_id=None, table_type=<TableType.MANAGED: 'MANAGED'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
08:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added secret scope fixture: sdk-Izbe
08:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added secret scope acl fixture: ('sdk-Izbe', 'ucx_GisUC')
08:06 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_so69q: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_so69q
08:06 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_so69q', metastore_id=None, name='ucx_so69q', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
08:06 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.15id/config.yml) doesn't exist.
08:06 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration
08:06 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data.
08:06 INFO [databricks.labs.ucx.install] Fetching installations...
08:06 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy.
08:06 DEBUG [tests.integration.conftest] Waiting for clusters to start...
08:06 DEBUG [tests.integration.conftest] Waiting for clusters to start...
08:06 INFO [databricks.labs.ucx.install] Installing UCX v0.27.2+1920240702080653
08:06 INFO [databricks.labs.ucx.install] Creating ucx schemas...
08:07 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
08:07 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
08:07 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
08:07 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
08:07 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation
08:07 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
08:07 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
08:07 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter
08:07 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental
08:07 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
08:07 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
08:07 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
08:07 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental
08:07 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.15id/README for the next steps.
08:07 DEBUG [databricks.labs.ucx.workspace_access.manager] Crawling permissions
08:07 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_so69q.grants] fetching grants inventory
08:08 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_so69q.grants] crawling new batch for grants
08:08 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_so69q.tables] fetching tables inventory
08:08 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_so69q.tables] crawling new batch for tables
08:08 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.ucx_svt6d] listing tables
08:08 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.ucx_svt6d.ucx_thh1o] fetching table metadata
08:08 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_so69q.tables] found 1 new records for tables
08:08 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_so69q.udfs] fetching udfs inventory
08:08 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_so69q.udfs] crawling new batch for udfs
08:08 DEBUG [databricks.labs.ucx.hive_metastore.udfs] [hive_metastore.ucx_svt6d] listing udfs
08:08 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_so69q.udfs] found 0 new records for udfs
08:09 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_so69q.grants] found 5 new records for grants
08:09 INFO [databricks.labs.ucx.workspace_access.manager] Starting to crawl permissions. Total tasks: 5
08:09 INFO [databricks.labs.ucx.workspace_access.manager] Total crawled permissions: 5
08:09 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_so69q.permissions] found 5 new records for permissions
08:09 INFO [databricks.labs.ucx.workspace_access.manager] Successfully saved the items to inventory table
08:09 INFO [databricks.labs.ucx.workspace_access.manager] Saved 5 to hive_metastore.ucx_so69q.permissions
08:09 DEBUG [databricks.labs.ucx.installer.workflows] starting validate-groups-permissions job: https://DATABRICKS_HOST#job/662269344681005
08:11 INFO [databricks.labs.ucx.installer.workflows] ---------- REMOTE LOGS --------------
08:11 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 secret scope acl fixtures
08:11 DEBUG [databricks.labs.ucx.mixins.fixtures] removing secret scope acl fixture: ('sdk-Izbe', 'ucx_GisUC')
08:11 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while secret scope acl ('sdk-Izbe', 'ucx_GisUC') teardown: User or Group ucx_GisUC does not exist.
08:11 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 secret scope fixtures
08:11 DEBUG [databricks.labs.ucx.mixins.fixtures] removing secret scope fixture: sdk-Izbe
08:11 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 cluster_policy permissions fixtures
08:11 DEBUG [databricks.labs.ucx.mixins.fixtures] removing cluster_policy permissions fixture: 00090F0E9ABE179A [group_name admins CAN_USE] -> [group_name ucx_GisUC CAN_USE]
08:11 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 cluster policy fixtures
08:11 DEBUG [databricks.labs.ucx.mixins.fixtures] removing cluster policy fixture: CreatePolicyResponse(policy_id='00090F0E9ABE179A')
08:11 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while cluster policy CreatePolicyResponse(policy_id='00090F0E9ABE179A') teardown: Can't find a cluster policy with id: 00090F0E9ABE179A.
08:11 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 query permissions fixtures
08:11 DEBUG [databricks.labs.ucx.mixins.fixtures] removing query permissions fixture: a4925302-ad5b-406c-8015-e7070a4d56ee [user_name 0a330eb5-dd51-4d97-b6e4-c474356b1d5d CAN_MANAGE, group_name admins CAN_MANAGE] -> [group_name ucx_GisUC CAN_EDIT]
08:11 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 query fixtures
08:11 DEBUG [databricks.labs.ucx.mixins.fixtures] removing query fixture: Query(can_edit=None, created_at='2024-07-02T08:06:34Z', data_source_id=None, description='TEST QUERY FOR UCX', id='a4925302-ad5b-406c-8015-e7070a4d56ee', is_archived=False, is_draft=False, is_favorite=False, is_safe=True, last_modified_by=User(email='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', id=481119220561874, name='labs-account-admin-identity'), last_modified_by_id=None, latest_query_data_id=None, name='ucx_query_QsosT', options=QueryOptions(catalog=None, moved_to_trash_at=None, parameters=[], schema=None), parent='folders/2080761702917835', permission_tier=None, query='SELECT * FROM ucx_snhlg.ucx_tzpi2', query_hash=None, run_as_role=<RunAsRole.OWNER: 'owner'>, tags=['original_query_tag'], updated_at='2024-07-02T08:06:34Z', user=User(email='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', id=481119220561874, name='labs-account-admin-identity'), user_id=481119220561874, visualizations=[Visualization(created_at='2024-07-02T08:06:33Z', description='', id='6dfe2c2b-1244-4262-85ad-fbde805665a1', name='Results', options={'version': 2}, query=None, type='TABLE', updated_at='2024-07-02T08:06:33Z')])
08:11 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while query Query(can_edit=None, created_at='2024-07-02T08:06:34Z', data_source_id=None, description='TEST QUERY FOR UCX', id='a4925302-ad5b-406c-8015-e7070a4d56ee', is_archived=False, is_draft=False, is_favorite=False, is_safe=True, last_modified_by=User(email='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', id=481119220561874, name='labs-account-admin-identity'), last_modified_by_id=None, latest_query_data_id=None, name='ucx_query_QsosT', options=QueryOptions(catalog=None, moved_to_trash_at=None, parameters=[], schema=None), parent='folders/2080761702917835', permission_tier=None, query='SELECT * FROM ucx_snhlg.ucx_tzpi2', query_hash=None, run_as_role=<RunAsRole.OWNER: 'owner'>, tags=['original_query_tag'], updated_at='2024-07-02T08:06:34Z', user=User(email='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', id=481119220561874, name='labs-account-admin-identity'), user_id=481119220561874, visualizations=[Visualization(created_at='2024-07-02T08:06:33Z', description='', id='6dfe2c2b-1244-4262-85ad-fbde805665a1', name='Results', options={'version': 2}, query=None, type='TABLE', updated_at='2024-07-02T08:06:33Z')]) teardown: Asset a4925302-ad5b-406c-8015-e7070a4d56ee is already in trash
08:11 INFO [databricks.labs.ucx.install] Deleting UCX v0.27.2+1920240702080653 from https://DATABRICKS_HOST
08:11 ERROR [databricks.labs.ucx.install] Check if /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.15id is present
08:11 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 workspace user fixtures
08:11 DEBUG [databricks.labs.ucx.mixins.fixtures] removing workspace user fixture: User(active=True, display_name='[email protected]', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='[email protected]')], entitlements=[], external_id=None, groups=[], id='1216970546986544', name=Name(family_name=None, given_name='[email protected]'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='[email protected]')
08:12 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while workspace user User(active=True, display_name='[email protected]', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='[email protected]')], entitlements=[], external_id=None, groups=[], id='1216970546986544', name=Name(family_name=None, given_name='[email protected]'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='[email protected]') teardown: None User with id 1216970546986544 not found.
08:12 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 account group fixtures
08:12 DEBUG [databricks.labs.ucx.mixins.fixtures] removing account group fixture: Group(display_name='ucx_GisUC', entitlements=[], external_id=None, groups=[], id='726939818273819', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/1216970546986544', type=None, value='1216970546986544')], meta=None, roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
08:12 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 workspace group fixtures
08:12 DEBUG [databricks.labs.ucx.mixins.fixtures] removing workspace group fixture: Group(display_name='ucx_GisUC', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='211537985122044', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/1216970546986544', type=None, value='1216970546986544')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
08:12 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while workspace group Group(display_name='ucx_GisUC', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='211537985122044', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/1216970546986544', type=None, value='1216970546986544')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>]) teardown: None Group with id 211537985122044 not found.
08:12 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures
08:12 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 2 table fixtures
08:12 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.DELTA: 'DELTA'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.ucx_snhlg.ucx_tzpi2', metastore_id=None, name='ucx_tzpi2', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070209'}, row_filter=None, schema_name='ucx_snhlg', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/ucx_snhlg/ucx_tzpi2', table_constraints=None, table_id=None, table_type=<TableType.MANAGED: 'MANAGED'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
08:12 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.DELTA: 'DELTA'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.ucx_svt6d.ucx_thh1o', metastore_id=None, name='ucx_thh1o', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024070209'}, row_filter=None, schema_name='ucx_svt6d', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/ucx_svt6d/ucx_thh1o', table_constraints=None, table_id=None, table_type=<TableType.MANAGED: 'MANAGED'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
08:12 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 3 schema fixtures
08:12 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_snhlg', metastore_id=None, name='ucx_snhlg', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
08:13 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_svt6d', metastore_id=None, name='ucx_svt6d', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
08:13 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_so69q', metastore_id=None, name='ucx_so69q', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
[gw1] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python

Flaky tests:

  • 🤪 test_run_external_locations_using_access_connector (28.215s)
  • 🤪 test_fix_dashboard (3m58.81s)

Running from acceptance #4045

Copy link
Member

@JCZuurmond JCZuurmond left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this the effect of changing a linter setting somewhere? The Changelog changes are a bit tricky, they will come. back

@asnare
Copy link
Contributor Author

asnare commented Jun 25, 2024

Is this the effect of changing a linter setting somewhere?

It's a side-effect of the editor settings in #1778. Historically we have quite a few files with trailing whitespace and missing the EOL at EOF. These aren't enforced by linting (which only covers python code), but with the new editor settings in place any time you modify a file trailing whitespace elsewhere in the file will be stripped. This was intended as a fix-them-all-for-now PR to avoid spurious updates on other PRs, particularly in Markdown.

The Changelog changes are a bit tricky, they will come back

Agreed; I'm happy to drop those given that the file is generated.

@asnare asnare requested a review from JCZuurmond June 27, 2024 13:02
LICENSE Outdated Show resolved Hide resolved
@asnare asnare requested a review from nfx July 3, 2024 08:09
Copy link
Collaborator

@nfx nfx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@nfx nfx disabled auto-merge July 5, 2024 07:34
@nfx nfx merged commit 6198a28 into main Jul 5, 2024
3 of 4 checks passed
@nfx nfx deleted the whitespace branch July 5, 2024 07:34
@nfx nfx added the internal this pull request won't appear in release notes label Jul 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
internal this pull request won't appear in release notes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants