Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Reuse code between runtime_ctx, installation_ctx and other integration testing contexts #3085

Closed
wants to merge 2 commits into from
Closed

Reuse code between runtime_ctx, installation_ctx and other integration testing contexts #3085

wants to merge 2 commits into from

Conversation

nfx
Copy link
Collaborator

@nfx nfx commented Oct 25, 2024

No description provided.

@nfx nfx added the internal this pull request won't appear in release notes label Oct 25, 2024
@nfx nfx requested a review from a team as a code owner October 25, 2024 17:03
@nfx nfx added the bug/test-infra issues related to testing infrastructure label Oct 25, 2024
@github-actions GitHub Actions
Copy link

❌ 125/129 passed, 4 failed, 26 skipped, 4h5m10s total

❌ test_migration_job_ext_hms[regular]: databricks.sdk.errors.platform.NotFound: [SCHEMA_NOT_FOUND] The schema `dummy_stt8k` cannot be found. Verify the spelling and correctness of the schema and catalog. (6m39.507s) 
databricks.sdk.errors.platform.NotFound: [SCHEMA_NOT_FOUND] The schema `dummy_stt8k` cannot be found. Verify the spelling and correctness of the schema and catalog.
If you did not qualify the name with a catalog, verify the current_schema() output, or qualify the name with the correct catalog.
To tolerate the error on drop use DROP SCHEMA IF EXISTS. SQLSTATE: 42704
18:10 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.S4KX/config.yml) doesn't exist.
18:10 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration
18:10 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data.
18:10 INFO [databricks.labs.ucx.install] Fetching installations...
18:11 WARNING [databricks.labs.ucx.install] Existing installation at /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.S4KX is corrupted. Skipping...
18:11 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy.
18:11 DEBUG [tests.integration.contexts.installation] Waiting for clusters to start...
18:11 DEBUG [tests.integration.contexts.installation] Waiting for clusters to start...
18:10 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.S4KX/config.yml) doesn't exist.
18:10 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration
18:10 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data.
18:10 INFO [databricks.labs.ucx.install] Fetching installations...
18:11 WARNING [databricks.labs.ucx.install] Existing installation at /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.S4KX is corrupted. Skipping...
18:11 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy.
18:11 DEBUG [tests.integration.contexts.installation] Waiting for clusters to start...
18:11 DEBUG [tests.integration.contexts.installation] Waiting for clusters to start...
18:12 INFO [databricks.labs.ucx.install] Deleting UCX v0.47.1+2820241025181205 from https://DATABRICKS_HOST
18:12 INFO [databricks.labs.ucx.install] Deleting inventory database dummy_stt8k
18:12 INFO [databricks.labs.ucx.install] Deleting cluster policy
18:12 INFO [databricks.labs.ucx.install] Deleting secret scope
18:12 INFO [databricks.labs.ucx.install] UnInstalling UCX complete
[gw2] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python
❌ test_create_catalog_schema_with_principal_acl_CLOUD_ENV: AssertionError: assert PrivilegeAssignment(principal='[email protected]', privileges=[]) in [] (2m34.123s) 
AssertionError: assert PrivilegeAssignment(principal='[email protected]', privileges=[<Privilege.USE_SCHEMA: 'USE_SCHEMA'>]) in []
 +  where [] = PermissionsList(privilege_assignments=[]).privilege_assignments
[gw9] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python
18:10 WARNING [databricks.labs.ucx.hive_metastore.catalog_schema] Skipping already existing catalog: dummy_cawgo
18:10 WARNING [databricks.labs.ucx.hive_metastore.catalog_schema] Skipping already existing schema: dummy_cawgo.dummy_ssma5
18:10 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sax5g.grants] fetching grants inventory
18:10 DEBUG [databricks.labs.ucx.framework.crawlers] Inventory table not found
Traceback (most recent call last):
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/framework/crawlers.py", line 152, in _snapshot
    cached_results = list(fetcher())
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/hive_metastore/grants.py", line 236, in _try_fetch
    for row in self._fetch(f"SELECT * FROM {escape_sql_identifier(self.full_name)}"):
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 331, in fetch_all
    execute_response = self.execute(
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 255, in execute
    self._raise_if_needed(status)
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 465, in _raise_if_needed
    raise NotFound(error_message)
databricks.sdk.errors.platform.NotFound: [TABLE_OR_VIEW_NOT_FOUND] The table or view `hive_metastore`.`dummy_sax5g`.`grants` cannot be found. Verify the spelling and correctness of the schema and catalog.
If you did not qualify the name with a schema, verify the current_schema() output, or qualify the name with the correct schema and catalog.
To tolerate the error on drop use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS. SQLSTATE: 42P01; line 1 pos 14
18:10 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sax5g.grants] crawling new set of snapshot data for grants
18:10 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sax5g.tables] fetching tables inventory
18:10 DEBUG [databricks.labs.ucx.framework.crawlers] Inventory table not found
Traceback (most recent call last):
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/framework/crawlers.py", line 152, in _snapshot
    cached_results = list(fetcher())
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/hive_metastore/tables.py", line 411, in _try_fetch
    for row in self._fetch(f"SELECT * FROM {escape_sql_identifier(self.full_name)}"):
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 331, in fetch_all
    execute_response = self.execute(
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 255, in execute
    self._raise_if_needed(status)
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 465, in _raise_if_needed
    raise NotFound(error_message)
databricks.sdk.errors.platform.NotFound: [TABLE_OR_VIEW_NOT_FOUND] The table or view `hive_metastore`.`dummy_sax5g`.`tables` cannot be found. Verify the spelling and correctness of the schema and catalog.
If you did not qualify the name with a schema, verify the current_schema() output, or qualify the name with the correct schema and catalog.
To tolerate the error on drop use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS. SQLSTATE: 42P01; line 1 pos 14
18:10 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sax5g.tables] crawling new set of snapshot data for tables
18:10 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.dummy_ssma5] listing tables and views
18:10 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.dummy_ssma5.dummy_tkjbr] fetching table metadata
18:10 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sax5g.tables] found 1 new records for tables
18:10 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sax5g.udfs] fetching udfs inventory
18:10 DEBUG [databricks.labs.ucx.framework.crawlers] Inventory table not found
Traceback (most recent call last):
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/framework/crawlers.py", line 152, in _snapshot
    cached_results = list(fetcher())
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/hive_metastore/udfs.py", line 63, in _try_fetch
    for row in self._fetch(f"SELECT * FROM {escape_sql_identifier(self.full_name)}"):
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 331, in fetch_all
    execute_response = self.execute(
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 255, in execute
    self._raise_if_needed(status)
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 465, in _raise_if_needed
    raise NotFound(error_message)
databricks.sdk.errors.platform.NotFound: [TABLE_OR_VIEW_NOT_FOUND] The table or view `hive_metastore`.`dummy_sax5g`.`udfs` cannot be found. Verify the spelling and correctness of the schema and catalog.
If you did not qualify the name with a schema, verify the current_schema() output, or qualify the name with the correct schema and catalog.
To tolerate the error on drop use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS. SQLSTATE: 42P01; line 1 pos 14
18:10 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sax5g.udfs] crawling new set of snapshot data for udfs
18:10 DEBUG [databricks.labs.ucx.hive_metastore.udfs] [hive_metastore.dummy_ssma5] listing udfs
18:10 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sax5g.udfs] found 0 new records for udfs
18:11 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sax5g.grants] found 2 new records for grants
18:11 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sax5g.tables] fetching tables inventory
18:11 INFO [databricks.labs.ucx.hive_metastore.grants] No interactive cluster found with spn configured
18:11 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sax5g.groups] fetching groups inventory
18:11 DEBUG [databricks.labs.ucx.framework.crawlers] Inventory table not found
Traceback (most recent call last):
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/framework/crawlers.py", line 152, in _snapshot
    cached_results = list(fetcher())
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/workspace_access/groups.py", line 629, in _try_fetch
    for row in self._sql_backend.fetch(f"SELECT * FROM {escape_sql_identifier(self.full_name)}"):
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 331, in fetch_all
    execute_response = self.execute(
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 255, in execute
    self._raise_if_needed(status)
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 465, in _raise_if_needed
    raise NotFound(error_message)
databricks.sdk.errors.platform.NotFound: [TABLE_OR_VIEW_NOT_FOUND] The table or view `hive_metastore`.`dummy_sax5g`.`groups` cannot be found. Verify the spelling and correctness of the schema and catalog.
If you did not qualify the name with a schema, verify the current_schema() output, or qualify the name with the correct schema and catalog.
To tolerate the error on drop use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS. SQLSTATE: 42P01; line 1 pos 14
18:11 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sax5g.groups] crawling new set of snapshot data for groups
18:11 INFO [databricks.labs.ucx.workspace_access.groups] Listing workspace groups (resource_type=WorkspaceGroup) with id,displayName,meta,externalId,members,roles,entitlements ...
18:11 INFO [databricks.labs.ucx.workspace_access.groups] Found 1 WorkspaceGroup
18:11 INFO [databricks.labs.ucx.workspace_access.groups] Listing account groups with id,displayName,externalId...
18:11 INFO [databricks.labs.ucx.workspace_access.groups] Found 68 account groups
18:11 INFO [databricks.labs.ucx.workspace_access.groups] No group listing provided, all matching groups will be migrated
18:11 INFO [databricks.labs.ucx.workspace_access.groups] Couldn't find a matching account group for sdk-2Rwc-ra78a55a78 group using name matching
18:11 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sax5g.groups] found 0 new records for groups
18:11 DEBUG [databricks.labs.ucx.hive_metastore.grants] Migrating acls on dummy_cawgo.dummy_ssma5 using SQL query: ALTER DATABASE `dummy_cawgo`.`dummy_ssma5` OWNER TO `0a330eb5-dd51-4d97-b6e4-c474356b1d5d`
18:11 DEBUG [databricks.labs.ucx.hive_metastore.grants] Migrating acls on dummy_cawgo using SQL query: ALTER CATALOG `dummy_cawgo` OWNER TO `0a330eb5-dd51-4d97-b6e4-c474356b1d5d`
18:10 WARNING [databricks.labs.ucx.hive_metastore.catalog_schema] Skipping already existing catalog: dummy_cawgo
18:10 WARNING [databricks.labs.ucx.hive_metastore.catalog_schema] Skipping already existing schema: dummy_cawgo.dummy_ssma5
18:10 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sax5g.grants] fetching grants inventory
18:10 DEBUG [databricks.labs.ucx.framework.crawlers] Inventory table not found
Traceback (most recent call last):
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/framework/crawlers.py", line 152, in _snapshot
    cached_results = list(fetcher())
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/hive_metastore/grants.py", line 236, in _try_fetch
    for row in self._fetch(f"SELECT * FROM {escape_sql_identifier(self.full_name)}"):
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 331, in fetch_all
    execute_response = self.execute(
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 255, in execute
    self._raise_if_needed(status)
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 465, in _raise_if_needed
    raise NotFound(error_message)
databricks.sdk.errors.platform.NotFound: [TABLE_OR_VIEW_NOT_FOUND] The table or view `hive_metastore`.`dummy_sax5g`.`grants` cannot be found. Verify the spelling and correctness of the schema and catalog.
If you did not qualify the name with a schema, verify the current_schema() output, or qualify the name with the correct schema and catalog.
To tolerate the error on drop use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS. SQLSTATE: 42P01; line 1 pos 14
18:10 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sax5g.grants] crawling new set of snapshot data for grants
18:10 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sax5g.tables] fetching tables inventory
18:10 DEBUG [databricks.labs.ucx.framework.crawlers] Inventory table not found
Traceback (most recent call last):
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/framework/crawlers.py", line 152, in _snapshot
    cached_results = list(fetcher())
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/hive_metastore/tables.py", line 411, in _try_fetch
    for row in self._fetch(f"SELECT * FROM {escape_sql_identifier(self.full_name)}"):
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 331, in fetch_all
    execute_response = self.execute(
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 255, in execute
    self._raise_if_needed(status)
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 465, in _raise_if_needed
    raise NotFound(error_message)
databricks.sdk.errors.platform.NotFound: [TABLE_OR_VIEW_NOT_FOUND] The table or view `hive_metastore`.`dummy_sax5g`.`tables` cannot be found. Verify the spelling and correctness of the schema and catalog.
If you did not qualify the name with a schema, verify the current_schema() output, or qualify the name with the correct schema and catalog.
To tolerate the error on drop use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS. SQLSTATE: 42P01; line 1 pos 14
18:10 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sax5g.tables] crawling new set of snapshot data for tables
18:10 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.dummy_ssma5] listing tables and views
18:10 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.dummy_ssma5.dummy_tkjbr] fetching table metadata
18:10 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sax5g.tables] found 1 new records for tables
18:10 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sax5g.udfs] fetching udfs inventory
18:10 DEBUG [databricks.labs.ucx.framework.crawlers] Inventory table not found
Traceback (most recent call last):
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/framework/crawlers.py", line 152, in _snapshot
    cached_results = list(fetcher())
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/hive_metastore/udfs.py", line 63, in _try_fetch
    for row in self._fetch(f"SELECT * FROM {escape_sql_identifier(self.full_name)}"):
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 331, in fetch_all
    execute_response = self.execute(
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 255, in execute
    self._raise_if_needed(status)
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 465, in _raise_if_needed
    raise NotFound(error_message)
databricks.sdk.errors.platform.NotFound: [TABLE_OR_VIEW_NOT_FOUND] The table or view `hive_metastore`.`dummy_sax5g`.`udfs` cannot be found. Verify the spelling and correctness of the schema and catalog.
If you did not qualify the name with a schema, verify the current_schema() output, or qualify the name with the correct schema and catalog.
To tolerate the error on drop use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS. SQLSTATE: 42P01; line 1 pos 14
18:10 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sax5g.udfs] crawling new set of snapshot data for udfs
18:10 DEBUG [databricks.labs.ucx.hive_metastore.udfs] [hive_metastore.dummy_ssma5] listing udfs
18:10 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sax5g.udfs] found 0 new records for udfs
18:11 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sax5g.grants] found 2 new records for grants
18:11 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sax5g.tables] fetching tables inventory
18:11 INFO [databricks.labs.ucx.hive_metastore.grants] No interactive cluster found with spn configured
18:11 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sax5g.groups] fetching groups inventory
18:11 DEBUG [databricks.labs.ucx.framework.crawlers] Inventory table not found
Traceback (most recent call last):
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/framework/crawlers.py", line 152, in _snapshot
    cached_results = list(fetcher())
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/workspace_access/groups.py", line 629, in _try_fetch
    for row in self._sql_backend.fetch(f"SELECT * FROM {escape_sql_identifier(self.full_name)}"):
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 331, in fetch_all
    execute_response = self.execute(
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 255, in execute
    self._raise_if_needed(status)
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 465, in _raise_if_needed
    raise NotFound(error_message)
databricks.sdk.errors.platform.NotFound: [TABLE_OR_VIEW_NOT_FOUND] The table or view `hive_metastore`.`dummy_sax5g`.`groups` cannot be found. Verify the spelling and correctness of the schema and catalog.
If you did not qualify the name with a schema, verify the current_schema() output, or qualify the name with the correct schema and catalog.
To tolerate the error on drop use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS. SQLSTATE: 42P01; line 1 pos 14
18:11 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sax5g.groups] crawling new set of snapshot data for groups
18:11 INFO [databricks.labs.ucx.workspace_access.groups] Listing workspace groups (resource_type=WorkspaceGroup) with id,displayName,meta,externalId,members,roles,entitlements ...
18:11 INFO [databricks.labs.ucx.workspace_access.groups] Found 1 WorkspaceGroup
18:11 INFO [databricks.labs.ucx.workspace_access.groups] Listing account groups with id,displayName,externalId...
18:11 INFO [databricks.labs.ucx.workspace_access.groups] Found 68 account groups
18:11 INFO [databricks.labs.ucx.workspace_access.groups] No group listing provided, all matching groups will be migrated
18:11 INFO [databricks.labs.ucx.workspace_access.groups] Couldn't find a matching account group for sdk-2Rwc-ra78a55a78 group using name matching
18:11 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sax5g.groups] found 0 new records for groups
18:11 DEBUG [databricks.labs.ucx.hive_metastore.grants] Migrating acls on dummy_cawgo.dummy_ssma5 using SQL query: ALTER DATABASE `dummy_cawgo`.`dummy_ssma5` OWNER TO `0a330eb5-dd51-4d97-b6e4-c474356b1d5d`
18:11 DEBUG [databricks.labs.ucx.hive_metastore.grants] Migrating acls on dummy_cawgo using SQL query: ALTER CATALOG `dummy_cawgo` OWNER TO `0a330eb5-dd51-4d97-b6e4-c474356b1d5d`
[gw9] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python
❌ test_migrate_external_tables_with_spn_CLOUD_ENV: assert False (1m52.482s) 
assert False
[gw8] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python
18:11 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_svw62.tables] fetching tables inventory
18:11 DEBUG [databricks.labs.ucx.framework.crawlers] Inventory table not found
Traceback (most recent call last):
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/framework/crawlers.py", line 152, in _snapshot
    cached_results = list(fetcher())
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/hive_metastore/tables.py", line 411, in _try_fetch
    for row in self._fetch(f"SELECT * FROM {escape_sql_identifier(self.full_name)}"):
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 331, in fetch_all
    execute_response = self.execute(
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 255, in execute
    self._raise_if_needed(status)
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 465, in _raise_if_needed
    raise NotFound(error_message)
databricks.sdk.errors.platform.NotFound: [TABLE_OR_VIEW_NOT_FOUND] The table or view `hive_metastore`.`dummy_svw62`.`tables` cannot be found. Verify the spelling and correctness of the schema and catalog.
If you did not qualify the name with a schema, verify the current_schema() output, or qualify the name with the correct schema and catalog.
To tolerate the error on drop use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS. SQLSTATE: 42P01; line 1 pos 14
18:11 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_svw62.tables] crawling new set of snapshot data for tables
18:11 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.dummy_suxml] listing tables and views
18:11 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.dummy_suxml.dummy_tmhft] fetching table metadata
18:11 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_svw62.tables] found 1 new records for tables
18:11 DEBUG [databricks.labs.ucx.hive_metastore.table_migrate] Migrating external table hive_metastore.dummy_suxml.dummy_tmhft to using SQL query: SYNC TABLE `dummy_cdhby`.`dummy_suxml`.`dummy_tmhft` FROM `hive_metastore`.`dummy_suxml`.`dummy_tmhft`;
18:11 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_svw62.grants] fetching grants inventory
18:11 DEBUG [databricks.labs.ucx.framework.crawlers] Inventory table not found
Traceback (most recent call last):
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/framework/crawlers.py", line 152, in _snapshot
    cached_results = list(fetcher())
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/hive_metastore/grants.py", line 236, in _try_fetch
    for row in self._fetch(f"SELECT * FROM {escape_sql_identifier(self.full_name)}"):
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 331, in fetch_all
    execute_response = self.execute(
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 255, in execute
    self._raise_if_needed(status)
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 465, in _raise_if_needed
    raise NotFound(error_message)
databricks.sdk.errors.platform.NotFound: [TABLE_OR_VIEW_NOT_FOUND] The table or view `hive_metastore`.`dummy_svw62`.`grants` cannot be found. Verify the spelling and correctness of the schema and catalog.
If you did not qualify the name with a schema, verify the current_schema() output, or qualify the name with the correct schema and catalog.
To tolerate the error on drop use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS. SQLSTATE: 42P01; line 1 pos 14
18:11 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_svw62.grants] crawling new set of snapshot data for grants
18:11 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_svw62.tables] fetching tables inventory
18:11 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_svw62.udfs] fetching udfs inventory
18:11 DEBUG [databricks.labs.ucx.framework.crawlers] Inventory table not found
Traceback (most recent call last):
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/framework/crawlers.py", line 152, in _snapshot
    cached_results = list(fetcher())
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/hive_metastore/udfs.py", line 63, in _try_fetch
    for row in self._fetch(f"SELECT * FROM {escape_sql_identifier(self.full_name)}"):
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 331, in fetch_all
    execute_response = self.execute(
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 255, in execute
    self._raise_if_needed(status)
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 465, in _raise_if_needed
    raise NotFound(error_message)
databricks.sdk.errors.platform.NotFound: [TABLE_OR_VIEW_NOT_FOUND] The table or view `hive_metastore`.`dummy_svw62`.`udfs` cannot be found. Verify the spelling and correctness of the schema and catalog.
If you did not qualify the name with a schema, verify the current_schema() output, or qualify the name with the correct schema and catalog.
To tolerate the error on drop use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS. SQLSTATE: 42P01; line 1 pos 14
18:11 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_svw62.udfs] crawling new set of snapshot data for udfs
18:11 DEBUG [databricks.labs.ucx.hive_metastore.udfs] [hive_metastore.dummy_suxml] listing udfs
18:11 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_svw62.udfs] found 0 new records for udfs
18:12 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_svw62.grants] found 2 new records for grants
18:12 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_svw62.tables] fetching tables inventory
18:12 INFO [databricks.labs.ucx.hive_metastore.grants] No interactive cluster found with spn configured
18:12 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_svw62.groups] fetching groups inventory
18:12 DEBUG [databricks.labs.ucx.framework.crawlers] Inventory table not found
Traceback (most recent call last):
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/framework/crawlers.py", line 152, in _snapshot
    cached_results = list(fetcher())
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/workspace_access/groups.py", line 629, in _try_fetch
    for row in self._sql_backend.fetch(f"SELECT * FROM {escape_sql_identifier(self.full_name)}"):
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 331, in fetch_all
    execute_response = self.execute(
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 255, in execute
    self._raise_if_needed(status)
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 465, in _raise_if_needed
    raise NotFound(error_message)
databricks.sdk.errors.platform.NotFound: [TABLE_OR_VIEW_NOT_FOUND] The table or view `hive_metastore`.`dummy_svw62`.`groups` cannot be found. Verify the spelling and correctness of the schema and catalog.
If you did not qualify the name with a schema, verify the current_schema() output, or qualify the name with the correct schema and catalog.
To tolerate the error on drop use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS. SQLSTATE: 42P01; line 1 pos 14
18:12 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_svw62.groups] crawling new set of snapshot data for groups
18:12 INFO [databricks.labs.ucx.workspace_access.groups] Listing workspace groups (resource_type=WorkspaceGroup) with id,displayName,meta,externalId,members,roles,entitlements ...
18:12 INFO [databricks.labs.ucx.workspace_access.groups] Found 1 WorkspaceGroup
18:12 INFO [databricks.labs.ucx.workspace_access.groups] Listing account groups with id,displayName,externalId...
18:12 INFO [databricks.labs.ucx.workspace_access.groups] Found 68 account groups
18:12 INFO [databricks.labs.ucx.workspace_access.groups] No group listing provided, all matching groups will be migrated
18:12 INFO [databricks.labs.ucx.workspace_access.groups] Couldn't find a matching account group for sdk-2Rwc-ra78a55a78 group using name matching
18:12 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_svw62.groups] found 0 new records for groups
18:12 DEBUG [databricks.labs.ucx.hive_metastore.grants] Migrating acls on dummy_cdhby.dummy_suxml.dummy_tmhft using SQL query: ALTER TABLE `dummy_cdhby`.`dummy_suxml`.`dummy_tmhft` OWNER TO `0a330eb5-dd51-4d97-b6e4-c474356b1d5d`
18:11 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_svw62.tables] fetching tables inventory
18:11 DEBUG [databricks.labs.ucx.framework.crawlers] Inventory table not found
Traceback (most recent call last):
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/framework/crawlers.py", line 152, in _snapshot
    cached_results = list(fetcher())
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/hive_metastore/tables.py", line 411, in _try_fetch
    for row in self._fetch(f"SELECT * FROM {escape_sql_identifier(self.full_name)}"):
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 331, in fetch_all
    execute_response = self.execute(
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 255, in execute
    self._raise_if_needed(status)
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 465, in _raise_if_needed
    raise NotFound(error_message)
databricks.sdk.errors.platform.NotFound: [TABLE_OR_VIEW_NOT_FOUND] The table or view `hive_metastore`.`dummy_svw62`.`tables` cannot be found. Verify the spelling and correctness of the schema and catalog.
If you did not qualify the name with a schema, verify the current_schema() output, or qualify the name with the correct schema and catalog.
To tolerate the error on drop use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS. SQLSTATE: 42P01; line 1 pos 14
18:11 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_svw62.tables] crawling new set of snapshot data for tables
18:11 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.dummy_suxml] listing tables and views
18:11 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.dummy_suxml.dummy_tmhft] fetching table metadata
18:11 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_svw62.tables] found 1 new records for tables
18:11 DEBUG [databricks.labs.ucx.hive_metastore.table_migrate] Migrating external table hive_metastore.dummy_suxml.dummy_tmhft to using SQL query: SYNC TABLE `dummy_cdhby`.`dummy_suxml`.`dummy_tmhft` FROM `hive_metastore`.`dummy_suxml`.`dummy_tmhft`;
18:11 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_svw62.grants] fetching grants inventory
18:11 DEBUG [databricks.labs.ucx.framework.crawlers] Inventory table not found
Traceback (most recent call last):
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/framework/crawlers.py", line 152, in _snapshot
    cached_results = list(fetcher())
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/hive_metastore/grants.py", line 236, in _try_fetch
    for row in self._fetch(f"SELECT * FROM {escape_sql_identifier(self.full_name)}"):
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 331, in fetch_all
    execute_response = self.execute(
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 255, in execute
    self._raise_if_needed(status)
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 465, in _raise_if_needed
    raise NotFound(error_message)
databricks.sdk.errors.platform.NotFound: [TABLE_OR_VIEW_NOT_FOUND] The table or view `hive_metastore`.`dummy_svw62`.`grants` cannot be found. Verify the spelling and correctness of the schema and catalog.
If you did not qualify the name with a schema, verify the current_schema() output, or qualify the name with the correct schema and catalog.
To tolerate the error on drop use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS. SQLSTATE: 42P01; line 1 pos 14
18:11 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_svw62.grants] crawling new set of snapshot data for grants
18:11 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_svw62.tables] fetching tables inventory
18:11 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_svw62.udfs] fetching udfs inventory
18:11 DEBUG [databricks.labs.ucx.framework.crawlers] Inventory table not found
Traceback (most recent call last):
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/framework/crawlers.py", line 152, in _snapshot
    cached_results = list(fetcher())
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/hive_metastore/udfs.py", line 63, in _try_fetch
    for row in self._fetch(f"SELECT * FROM {escape_sql_identifier(self.full_name)}"):
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 331, in fetch_all
    execute_response = self.execute(
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 255, in execute
    self._raise_if_needed(status)
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 465, in _raise_if_needed
    raise NotFound(error_message)
databricks.sdk.errors.platform.NotFound: [TABLE_OR_VIEW_NOT_FOUND] The table or view `hive_metastore`.`dummy_svw62`.`udfs` cannot be found. Verify the spelling and correctness of the schema and catalog.
If you did not qualify the name with a schema, verify the current_schema() output, or qualify the name with the correct schema and catalog.
To tolerate the error on drop use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS. SQLSTATE: 42P01; line 1 pos 14
18:11 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_svw62.udfs] crawling new set of snapshot data for udfs
18:11 DEBUG [databricks.labs.ucx.hive_metastore.udfs] [hive_metastore.dummy_suxml] listing udfs
18:11 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_svw62.udfs] found 0 new records for udfs
18:12 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_svw62.grants] found 2 new records for grants
18:12 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_svw62.tables] fetching tables inventory
18:12 INFO [databricks.labs.ucx.hive_metastore.grants] No interactive cluster found with spn configured
18:12 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_svw62.groups] fetching groups inventory
18:12 DEBUG [databricks.labs.ucx.framework.crawlers] Inventory table not found
Traceback (most recent call last):
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/framework/crawlers.py", line 152, in _snapshot
    cached_results = list(fetcher())
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/workspace_access/groups.py", line 629, in _try_fetch
    for row in self._sql_backend.fetch(f"SELECT * FROM {escape_sql_identifier(self.full_name)}"):
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 331, in fetch_all
    execute_response = self.execute(
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 255, in execute
    self._raise_if_needed(status)
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 465, in _raise_if_needed
    raise NotFound(error_message)
databricks.sdk.errors.platform.NotFound: [TABLE_OR_VIEW_NOT_FOUND] The table or view `hive_metastore`.`dummy_svw62`.`groups` cannot be found. Verify the spelling and correctness of the schema and catalog.
If you did not qualify the name with a schema, verify the current_schema() output, or qualify the name with the correct schema and catalog.
To tolerate the error on drop use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS. SQLSTATE: 42P01; line 1 pos 14
18:12 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_svw62.groups] crawling new set of snapshot data for groups
18:12 INFO [databricks.labs.ucx.workspace_access.groups] Listing workspace groups (resource_type=WorkspaceGroup) with id,displayName,meta,externalId,members,roles,entitlements ...
18:12 INFO [databricks.labs.ucx.workspace_access.groups] Found 1 WorkspaceGroup
18:12 INFO [databricks.labs.ucx.workspace_access.groups] Listing account groups with id,displayName,externalId...
18:12 INFO [databricks.labs.ucx.workspace_access.groups] Found 68 account groups
18:12 INFO [databricks.labs.ucx.workspace_access.groups] No group listing provided, all matching groups will be migrated
18:12 INFO [databricks.labs.ucx.workspace_access.groups] Couldn't find a matching account group for sdk-2Rwc-ra78a55a78 group using name matching
18:12 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_svw62.groups] found 0 new records for groups
18:12 DEBUG [databricks.labs.ucx.hive_metastore.grants] Migrating acls on dummy_cdhby.dummy_suxml.dummy_tmhft using SQL query: ALTER TABLE `dummy_cdhby`.`dummy_suxml`.`dummy_tmhft` OWNER TO `0a330eb5-dd51-4d97-b6e4-c474356b1d5d`
[gw8] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python
❌ test_migrate_external_tables_with_principal_acl_CLOUD_ENV: assert False (2m39.899s) 
assert False
[gw5] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python
18:14 WARNING [databricks.labs.ucx.hive_metastore.table_migration_status] Catalog ucx_hwasxebfioslz5i1 no longer exists. Skipping checking its migration status.
18:14 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_s4h9i.tables] fetching tables inventory
18:14 DEBUG [databricks.labs.ucx.framework.crawlers] Inventory table not found
Traceback (most recent call last):
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/framework/crawlers.py", line 152, in _snapshot
    cached_results = list(fetcher())
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/hive_metastore/tables.py", line 411, in _try_fetch
    for row in self._fetch(f"SELECT * FROM {escape_sql_identifier(self.full_name)}"):
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 331, in fetch_all
    execute_response = self.execute(
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 255, in execute
    self._raise_if_needed(status)
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 465, in _raise_if_needed
    raise NotFound(error_message)
databricks.sdk.errors.platform.NotFound: [TABLE_OR_VIEW_NOT_FOUND] The table or view `hive_metastore`.`dummy_s4h9i`.`tables` cannot be found. Verify the spelling and correctness of the schema and catalog.
If you did not qualify the name with a schema, verify the current_schema() output, or qualify the name with the correct schema and catalog.
To tolerate the error on drop use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS. SQLSTATE: 42P01; line 1 pos 14
18:14 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_s4h9i.tables] crawling new set of snapshot data for tables
18:14 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.dummy_s9rfg] listing tables and views
18:14 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.dummy_s9rfg.dummy_tmsai] fetching table metadata
18:15 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_s4h9i.tables] found 1 new records for tables
18:15 DEBUG [databricks.labs.ucx.hive_metastore.table_migrate] Migrating external table hive_metastore.dummy_s9rfg.dummy_tmsai to using SQL query: SYNC TABLE `dummy_cnjqk`.`dummy_s9rfg`.`dummy_tmsai` FROM `hive_metastore`.`dummy_s9rfg`.`dummy_tmsai`;
18:15 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_s4h9i.grants] fetching grants inventory
18:15 DEBUG [databricks.labs.ucx.framework.crawlers] Inventory table not found
Traceback (most recent call last):
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/framework/crawlers.py", line 152, in _snapshot
    cached_results = list(fetcher())
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/hive_metastore/grants.py", line 236, in _try_fetch
    for row in self._fetch(f"SELECT * FROM {escape_sql_identifier(self.full_name)}"):
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 331, in fetch_all
    execute_response = self.execute(
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 255, in execute
    self._raise_if_needed(status)
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 465, in _raise_if_needed
    raise NotFound(error_message)
databricks.sdk.errors.platform.NotFound: [TABLE_OR_VIEW_NOT_FOUND] The table or view `hive_metastore`.`dummy_s4h9i`.`grants` cannot be found. Verify the spelling and correctness of the schema and catalog.
If you did not qualify the name with a schema, verify the current_schema() output, or qualify the name with the correct schema and catalog.
To tolerate the error on drop use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS. SQLSTATE: 42P01; line 1 pos 14
18:15 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_s4h9i.grants] crawling new set of snapshot data for grants
18:15 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_s4h9i.tables] fetching tables inventory
18:15 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_s4h9i.udfs] fetching udfs inventory
18:15 DEBUG [databricks.labs.ucx.framework.crawlers] Inventory table not found
Traceback (most recent call last):
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/framework/crawlers.py", line 152, in _snapshot
    cached_results = list(fetcher())
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/hive_metastore/udfs.py", line 63, in _try_fetch
    for row in self._fetch(f"SELECT * FROM {escape_sql_identifier(self.full_name)}"):
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 331, in fetch_all
    execute_response = self.execute(
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 255, in execute
    self._raise_if_needed(status)
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 465, in _raise_if_needed
    raise NotFound(error_message)
databricks.sdk.errors.platform.NotFound: [TABLE_OR_VIEW_NOT_FOUND] The table or view `hive_metastore`.`dummy_s4h9i`.`udfs` cannot be found. Verify the spelling and correctness of the schema and catalog.
If you did not qualify the name with a schema, verify the current_schema() output, or qualify the name with the correct schema and catalog.
To tolerate the error on drop use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS. SQLSTATE: 42P01; line 1 pos 14
18:15 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_s4h9i.udfs] crawling new set of snapshot data for udfs
18:15 DEBUG [databricks.labs.ucx.hive_metastore.udfs] [hive_metastore.dummy_s9rfg] listing udfs
18:15 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_s4h9i.udfs] found 0 new records for udfs
18:16 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_s4h9i.grants] found 2 new records for grants
18:16 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_s4h9i.tables] fetching tables inventory
18:16 INFO [databricks.labs.ucx.hive_metastore.grants] No interactive cluster found with spn configured
18:16 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_s4h9i.groups] fetching groups inventory
18:16 DEBUG [databricks.labs.ucx.framework.crawlers] Inventory table not found
Traceback (most recent call last):
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/framework/crawlers.py", line 152, in _snapshot
    cached_results = list(fetcher())
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/workspace_access/groups.py", line 629, in _try_fetch
    for row in self._sql_backend.fetch(f"SELECT * FROM {escape_sql_identifier(self.full_name)}"):
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 331, in fetch_all
    execute_response = self.execute(
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 255, in execute
    self._raise_if_needed(status)
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 465, in _raise_if_needed
    raise NotFound(error_message)
databricks.sdk.errors.platform.NotFound: [TABLE_OR_VIEW_NOT_FOUND] The table or view `hive_metastore`.`dummy_s4h9i`.`groups` cannot be found. Verify the spelling and correctness of the schema and catalog.
If you did not qualify the name with a schema, verify the current_schema() output, or qualify the name with the correct schema and catalog.
To tolerate the error on drop use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS. SQLSTATE: 42P01; line 1 pos 14
18:16 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_s4h9i.groups] crawling new set of snapshot data for groups
18:16 INFO [databricks.labs.ucx.workspace_access.groups] Listing workspace groups (resource_type=WorkspaceGroup) with id,displayName,meta,externalId,members,roles,entitlements ...
18:16 INFO [databricks.labs.ucx.workspace_access.groups] Found 6 WorkspaceGroup
18:16 INFO [databricks.labs.ucx.workspace_access.groups] Listing account groups with id,displayName,externalId...
18:16 INFO [databricks.labs.ucx.workspace_access.groups] Found 70 account groups
18:16 INFO [databricks.labs.ucx.workspace_access.groups] No group listing provided, all matching groups will be migrated
18:16 INFO [databricks.labs.ucx.workspace_access.groups] Couldn't find a matching account group for sdk-0KaJ-ra78a55a78 group using name matching
18:16 INFO [databricks.labs.ucx.workspace_access.groups] Couldn't find a matching account group for sdk-FZrW-ra78a55a78 group using name matching
18:16 INFO [databricks.labs.ucx.workspace_access.groups] Couldn't find a matching account group for sdk-lNz1-ra78a55a78 group using name matching
18:16 INFO [databricks.labs.ucx.workspace_access.groups] Couldn't find a matching account group for sdk-5WVr-ra78a55a78 group using name matching
18:16 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_s4h9i.groups] found 2 new records for groups
18:16 DEBUG [databricks.labs.ucx.hive_metastore.grants] Migrating acls on dummy_cnjqk.dummy_s9rfg.dummy_tmsai using SQL query: ALTER TABLE `dummy_cnjqk`.`dummy_s9rfg`.`dummy_tmsai` OWNER TO `0a330eb5-dd51-4d97-b6e4-c474356b1d5d`
18:14 WARNING [databricks.labs.ucx.hive_metastore.table_migration_status] Catalog ucx_hwasxebfioslz5i1 no longer exists. Skipping checking its migration status.
18:14 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_s4h9i.tables] fetching tables inventory
18:14 DEBUG [databricks.labs.ucx.framework.crawlers] Inventory table not found
Traceback (most recent call last):
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/framework/crawlers.py", line 152, in _snapshot
    cached_results = list(fetcher())
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/hive_metastore/tables.py", line 411, in _try_fetch
    for row in self._fetch(f"SELECT * FROM {escape_sql_identifier(self.full_name)}"):
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 331, in fetch_all
    execute_response = self.execute(
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 255, in execute
    self._raise_if_needed(status)
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 465, in _raise_if_needed
    raise NotFound(error_message)
databricks.sdk.errors.platform.NotFound: [TABLE_OR_VIEW_NOT_FOUND] The table or view `hive_metastore`.`dummy_s4h9i`.`tables` cannot be found. Verify the spelling and correctness of the schema and catalog.
If you did not qualify the name with a schema, verify the current_schema() output, or qualify the name with the correct schema and catalog.
To tolerate the error on drop use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS. SQLSTATE: 42P01; line 1 pos 14
18:14 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_s4h9i.tables] crawling new set of snapshot data for tables
18:14 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.dummy_s9rfg] listing tables and views
18:14 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.dummy_s9rfg.dummy_tmsai] fetching table metadata
18:15 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_s4h9i.tables] found 1 new records for tables
18:15 DEBUG [databricks.labs.ucx.hive_metastore.table_migrate] Migrating external table hive_metastore.dummy_s9rfg.dummy_tmsai to using SQL query: SYNC TABLE `dummy_cnjqk`.`dummy_s9rfg`.`dummy_tmsai` FROM `hive_metastore`.`dummy_s9rfg`.`dummy_tmsai`;
18:15 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_s4h9i.grants] fetching grants inventory
18:15 DEBUG [databricks.labs.ucx.framework.crawlers] Inventory table not found
Traceback (most recent call last):
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/framework/crawlers.py", line 152, in _snapshot
    cached_results = list(fetcher())
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/hive_metastore/grants.py", line 236, in _try_fetch
    for row in self._fetch(f"SELECT * FROM {escape_sql_identifier(self.full_name)}"):
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 331, in fetch_all
    execute_response = self.execute(
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 255, in execute
    self._raise_if_needed(status)
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 465, in _raise_if_needed
    raise NotFound(error_message)
databricks.sdk.errors.platform.NotFound: [TABLE_OR_VIEW_NOT_FOUND] The table or view `hive_metastore`.`dummy_s4h9i`.`grants` cannot be found. Verify the spelling and correctness of the schema and catalog.
If you did not qualify the name with a schema, verify the current_schema() output, or qualify the name with the correct schema and catalog.
To tolerate the error on drop use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS. SQLSTATE: 42P01; line 1 pos 14
18:15 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_s4h9i.grants] crawling new set of snapshot data for grants
18:15 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_s4h9i.tables] fetching tables inventory
18:15 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_s4h9i.udfs] fetching udfs inventory
18:15 DEBUG [databricks.labs.ucx.framework.crawlers] Inventory table not found
Traceback (most recent call last):
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/framework/crawlers.py", line 152, in _snapshot
    cached_results = list(fetcher())
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/hive_metastore/udfs.py", line 63, in _try_fetch
    for row in self._fetch(f"SELECT * FROM {escape_sql_identifier(self.full_name)}"):
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 331, in fetch_all
    execute_response = self.execute(
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 255, in execute
    self._raise_if_needed(status)
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 465, in _raise_if_needed
    raise NotFound(error_message)
databricks.sdk.errors.platform.NotFound: [TABLE_OR_VIEW_NOT_FOUND] The table or view `hive_metastore`.`dummy_s4h9i`.`udfs` cannot be found. Verify the spelling and correctness of the schema and catalog.
If you did not qualify the name with a schema, verify the current_schema() output, or qualify the name with the correct schema and catalog.
To tolerate the error on drop use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS. SQLSTATE: 42P01; line 1 pos 14
18:15 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_s4h9i.udfs] crawling new set of snapshot data for udfs
18:15 DEBUG [databricks.labs.ucx.hive_metastore.udfs] [hive_metastore.dummy_s9rfg] listing udfs
18:15 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_s4h9i.udfs] found 0 new records for udfs
18:16 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_s4h9i.grants] found 2 new records for grants
18:16 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_s4h9i.tables] fetching tables inventory
18:16 INFO [databricks.labs.ucx.hive_metastore.grants] No interactive cluster found with spn configured
18:16 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_s4h9i.groups] fetching groups inventory
18:16 DEBUG [databricks.labs.ucx.framework.crawlers] Inventory table not found
Traceback (most recent call last):
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/framework/crawlers.py", line 152, in _snapshot
    cached_results = list(fetcher())
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/workspace_access/groups.py", line 629, in _try_fetch
    for row in self._sql_backend.fetch(f"SELECT * FROM {escape_sql_identifier(self.full_name)}"):
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 331, in fetch_all
    execute_response = self.execute(
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 255, in execute
    self._raise_if_needed(status)
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 465, in _raise_if_needed
    raise NotFound(error_message)
databricks.sdk.errors.platform.NotFound: [TABLE_OR_VIEW_NOT_FOUND] The table or view `hive_metastore`.`dummy_s4h9i`.`groups` cannot be found. Verify the spelling and correctness of the schema and catalog.
If you did not qualify the name with a schema, verify the current_schema() output, or qualify the name with the correct schema and catalog.
To tolerate the error on drop use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS. SQLSTATE: 42P01; line 1 pos 14
18:16 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_s4h9i.groups] crawling new set of snapshot data for groups
18:16 INFO [databricks.labs.ucx.workspace_access.groups] Listing workspace groups (resource_type=WorkspaceGroup) with id,displayName,meta,externalId,members,roles,entitlements ...
18:16 INFO [databricks.labs.ucx.workspace_access.groups] Found 6 WorkspaceGroup
18:16 INFO [databricks.labs.ucx.workspace_access.groups] Listing account groups with id,displayName,externalId...
18:16 INFO [databricks.labs.ucx.workspace_access.groups] Found 70 account groups
18:16 INFO [databricks.labs.ucx.workspace_access.groups] No group listing provided, all matching groups will be migrated
18:16 INFO [databricks.labs.ucx.workspace_access.groups] Couldn't find a matching account group for sdk-0KaJ-ra78a55a78 group using name matching
18:16 INFO [databricks.labs.ucx.workspace_access.groups] Couldn't find a matching account group for sdk-FZrW-ra78a55a78 group using name matching
18:16 INFO [databricks.labs.ucx.workspace_access.groups] Couldn't find a matching account group for sdk-lNz1-ra78a55a78 group using name matching
18:16 INFO [databricks.labs.ucx.workspace_access.groups] Couldn't find a matching account group for sdk-5WVr-ra78a55a78 group using name matching
18:16 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_s4h9i.groups] found 2 new records for groups
18:16 DEBUG [databricks.labs.ucx.hive_metastore.grants] Migrating acls on dummy_cnjqk.dummy_s9rfg.dummy_tmsai using SQL query: ALTER TABLE `dummy_cnjqk`.`dummy_s9rfg`.`dummy_tmsai` OWNER TO `0a330eb5-dd51-4d97-b6e4-c474356b1d5d`
[gw5] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

Running from acceptance #7088

asnare
asnare reviewed Nov 5, 2024
View reviewed changes
src/databricks/labs/ucx/hive_metastore/locations.py
def __init__(
self,
ws: WorkspaceClient,
sql_backend: SqlBackend,
schema: str,
tables_crawler: TablesCrawler,
mounts_crawler: 'MountsCrawler',
mounts_crawler: CrawlerBase['Mount'],
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
mounts_crawler: CrawlerBase['Mount'],
mounts_crawler: CrawlerBase[Mount],

@JCZuurmond JCZuurmond mentioned this pull request Dec 6, 2024
Open
1 task
@JCZuurmond
Copy link
Member

Created issue to pick in planning, closing this PR for now to clean up, can reopen later: See #3085

@JCZuurmond JCZuurmond closed this Dec 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@asnare asnare asnare left review comments

Assignees
No one assigned
Labels
bug/test-infra issues related to testing infrastructure internal this pull request won't appear in release notes
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@nfx @JCZuurmond @asnare