Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG]: FEATURE_DISABLED Error in crawl_permissions task. #1617

Closed
1 task done
RobMandersBJSS opened this issue May 3, 2024 · 0 comments · Fixed by #1618
Closed
1 task done

[BUG]: FEATURE_DISABLED Error in crawl_permissions task. #1617

RobMandersBJSS opened this issue May 3, 2024 · 0 comments · Fixed by #1618
Labels
needs-triage

Comments

@RobMandersBJSS
Copy link

Is there an existing issue for this?

  • I have searched the existing issues

Current Behavior

I was advised to re-raise this issue, previously raised on the new closed #1228, by David Poulet as it does not appear this issue has been resolved.

A potential fix for this issue was merged in PR #1275 and released with version 0.22.0.

Following a fresh installation of UCX version 0.22.0 on our Databricks environment, we re-ran the assessment pipeline and the same error occorred when running the crawl_permissions task:

image

Our logs and stack trace have been provided for you. Please let me know if you need any additional information.

Expected Behavior

If model serving is not enabled within your region, the FEATURE_DISABLED error will be ignored and and the crawl_permissions task will succeed.

Steps To Reproduce

  1. Install UCX v0.22.0 via Powershell.
  2. On an environment in which Model Serving is not enables, run the assessment workflow.

Cloud

Azure

Operating System

Windows

Version

latest via Databricks CLI

Relevant log output

09:02:24 INFO [databricks.labs.ucx] {MainThread} UCX v0.22.0 After job finishes, see debug logs at /Workspace/Applications/ucx/logs/assessment/run-227626125051970-0/crawl_permissions.log
09:02:24 DEBUG [databricks.sdk] {MainThread} GET /api/2.0/preview/scim/v2/Groups?attributes=id,displayName,meta,roles,entitlements&startIndex=1&count=100
< 200 OK
< {
<   "Resources": [
<     {
<       "displayName": "admins",
<       "entitlements": [
<         {
<           "value": "**REDACTED**"
<         },
<         {
<           "value": "**REDACTED**"
<         },
<         {
<           "value": "**REDACTED**"
<         },
<         {
<           "value": "**REDACTED**"
<         }
<       ],
<       "id": "4463504595441",
<       "meta": {
<         "resourceType": "WorkspaceGroup"
<       }
<     },
<     "... (4 additional elements)"
<   ],
<   "itemsPerPage": 5,
<   "schemas": [
<     "urn:ietf:params:scim:api:messages:2.0:ListResponse"
<   ],
<   "startIndex": 1,
<   "totalResults": 5
< }
09:02:24 DEBUG [databricks.sdk] {MainThread} GET /api/2.0/preview/scim/v2/Groups?attributes=id,displayName,meta,roles,entitlements&startIndex=6&count=100
< 200 OK
< {
<   "itemsPerPage": 0,
<   "schemas": [
<     "urn:ietf:params:scim:api:messages:2.0:ListResponse"
<   ],
<   "startIndex": 6,
<   "totalResults": 5
< }
09:02:24 INFO [databricks.labs.ucx.workspace_access.manager] {MainThread} Cleaning up inventory table hive_metastore.ucx.permissions
09:02:24 DEBUG [databricks.labs.lsql.backends] {MainThread} [spark][execute] DROP TABLE IF EXISTS hive_metastore.ucx.permissions
09:02:26 INFO [databricks.labs.ucx.workspace_access.manager] {MainThread} Inventory table cleanup complete
09:02:26 DEBUG [databricks.labs.ucx.workspace_access.manager] {MainThread} Crawling permissions
09:02:26 DEBUG [databricks.sdk] {MainThread} GET /api/2.0/clusters/list
< 200 OK
< {
<   "clusters": [
<     {
<       "autoscale": {
<         "max_workers": 2,
<         "min_workers": 1,
<         "target_workers": 1
<       },
<       "autotermination_minutes": 0,
<       "azure_attributes": {
<         "availability": "ON_DEMAND_AZURE",
<         "first_on_demand": 1,
<         "spot_bid_max_price": -1.0
<       },
<       "cluster_cores": 16.0,
<       "cluster_id": "0429-085859-6gtds98z",
<       "cluster_log_conf": {
<         "dbfs": {
<           "destination": "dbfs:/logs/dlt/website"
<         }
<       },
<       "cluster_log_status": {
<         "last_attempted": 1714381227692
<       },
<       "cluster_memory_mb": 32768,
<       "cluster_name": "dlt-execution-270664c8-5913-406f-a44b-f7794ef884e4",
<       "cluster_source": "PIPELINE",
<       "creator_user_name": "87507d41-ffd1-4f56-8ddd-81edb0f12563",
<       "default_tags": {
<         "ClusterId": "0429-085859-6gtds98z",
<         "ClusterName": "dlt-execution-270664c8-5913-406f-a44b-f7794ef884e4",
<         "Creator": "87507d41-ffd1-4f56-8ddd-81edb0f12563",
<         "Vendor": "Databricks",
<         "component": "Data Processing",
<         "last_run": "2023-10-20T18:17:57.2717803Z",
<         "purpose": "Data Analysis",
<         "source": "ARM"
<       },
<       "disk_spec": {},
<       "driver": {
<         "host_private_ip": "10.40.192.12",
<         "instance_id": "940f0d187ee14625afd33009b1d3c70a",
<         "node_attributes": {
<           "is_spot": false
<         },
<         "node_id": "5b618ff437f04c3db471d19c486d77bf",
<         "private_ip": "10.40.208.11",
<         "public_dns": "172.167.154.186",
<         "start_timestamp": 1714381140273
<       },
<       "driver_healthy": true,
<       "driver_instance_source": {
<         "node_type_id": "Standard_F8s"
<       },
<       "driver_node_type_id": "Standard_F8s",
<       "effective_spark_version": "dlt:14.1-delta-pipelines-dlt-release-2024.15-rc1-commit-9b76dca-image-bc65e26",
<       "enable_elastic_disk": true,
<       "enable_local_disk_encryption": false,
<       "executors": [
<         {
<           "host_private_ip": "10.40.192.11",
<           "instance_id": "af9761e0f02c40d2a0661f37060be6e5",
<           "node_attributes": {
<             "is_spot": false
<           },
<           "node_id": "319fddc3f97f4b7ea5fc144028f75d63",
<           "private_ip": "10.40.208.12",
<           "public_dns": "172.167.120.91",
<           "start_timestamp": 1714381140221
<         }
<       ],
<       "init_scripts_safe_mode": false,
<       "instance_source": {
<         "node_type_id": "Standard_F8s"
<       },
<       "jdbc_port": 10000,
<       "last_activity_time": 1714381223870,
<       "last_restarted_time": 1714381197953,
<       "last_state_loss_time": 0,
<       "node_type_id": "Standard_F8s",
<       "runtime_engine": "STANDARD",
<       "spark_conf": {
<         "pipelines.advancedAutoscaling.enabled": "true",
<         "pipelines.applyChanges.enableGCFix": "true",
<         "pipelines.applyChanges.viewMetrics.enabled": "true",
<         "pipelines.autoscaling.maxNumExecutors": "2",
<         "pipelines.autoscaling.minNumExecutors": "1",
<         "pipelines.catalog": "",
<         "pipelines.changeDataFeed.enabled": "true",
<         "pipelines.channel": "CURRENT",
<         "pipelines.cloud": "Azure",
<         "pipelines.clusterLabelsV2Enabled": "true",
<         "pipelines.clusterSpecChecksum": "C2D33C87E7601CC92D829ABED2198ADE",
<         "pipelines.dataPlaneStructuredLogging.enabled": "true",
<         "pipelines.dbrVersion": "14.1",
<         "pipelines.decomposition.enabled": "true",
<         "pipelines.dltDebugger.enableNodeStatusReporter": "true",
<         "pipelines.dltDebugger.enabled": "true",
<         "pipelines.dltDebugger.nodeStatusEventLoggingEnabled": "false",
<         "pipelines.enableCyclicDependencyDetection": "true",
<         "pipelines.enableLiveSchemaUsageValidation": "true",
<         "pipelines.enabled": "true",
<         "pipelines.enzyme.classic.action": "Block",
<         "pipelines.enzyme.decomposeAggExpr.enabled": "true",
<         "pipelines.enzyme.enabled": "true",
<         "pipelines.enzyme.fileActionCountThreshold": "2000000000",
<         "pipelines.enzyme.genericAggregationTechnique.enabled": "true",
<         "pipelines.eventLevelDataFrameCollectUsage": "warn",
<         "pipelines.eventLevelDataFrameSaveAsTableUsage": "warn",
<         "pipelines.eventLevelDataFrameSaveUsage": "warn",
<         "pipelines.eventLevelDataStreamWriterStartUsage": "warn",
<         "pipelines.eventLevelPivotUsage": "warn",
<         "pipelines.events.enableSourceCodePosition": "true",
<         "pipelines.id": "270664c8-5913-406f-a44b-f7794ef884e4",
<         "pipelines.longRunningStageDetector.enabled": "true",
<         "pipelines.maxDatasetsPerPipeline": "1000",
<         "pipelines.metrics.clusterResources.enabled": "true",
<         "pipelines.metrics.clusterUtilization.enabled": "true",
<         "pipelines.metrics.flowProgressAdditionalBacklogMetricsEnabled": "true",
<         "pipelines.metrics.flowProgressBacklog.enabled": "true",
<         "pipelines.metrics.flowTimeReporter.enabled": "true",
<         "pipelines.schemaStorageLocations.enabled": "true",
<         "pipelines.target": "website_afcd63b9060611efa01b47bcfb7c44c7",
<         "pipelines.topologicalOrderForStreamingFlowsUponFullRefresh.enabled": "true",
<         "pipelines.updateSnapshot.enabled": "true",
<         "spark.databricks.acceptSoakingFeatures": "false",
<         "spark.databricks.cloudFiles.rootSchemaLocation": "abfss://[email protected]/website/afcd63b9060611efa01b47bcfb7c44c7/... (10 more bytes)",
<         "spark.databricks.cloudFiles.sqlApi.enabled": "true",
<         "spark.databricks.delta.deletionVectors.allowAutoEnable": "false",
<         "spark.databricks.delta.optimizeWrite.enabled": "true",
<         "spark.databricks.delta.optimizeWrite.useAQE": "false",
<         "spark.databricks.delta.schema.autoMerge.enabled": "true",
<         "spark.databricks.delta.updateAndMergeCastingFollowsAnsiEnabledFlag": "true",
<         "spark.databricks.driver.hardKillUnresponsive.enabled": "false",
<         "spark.databricks.driver.ipykernel.commChannelEnabled": "true",
<         "spark.databricks.driver.ipykernel.commChannelPort": "15152",
<         "spark.databricks.driverNfs.enabled": "true",
<         "spark.databricks.preemption.enabled": "false",
<         "spark.databricks.pyspark.passPythonUserToClonedSession": "false",
<         "spark.databricks.sql.MaterializedViewReadAllowInDBR": "true",
<         "spark.databricks.sql.functions.fromJson.inference.enabled": "true",
<         "spark.debugger.enabled": "true",
<         "spark.debugger.eventListener": "org.apache.spark.debugger.DLTDebuggerEventReporter",
<         "spark.decommission.enabled": "true",
<         "spark.hadoop.fs.azure.account.auth.type.dlrwpdauksdev001.dfs.core.windows.net": "OAuth",
<         "spark.hadoop.fs.azure.account.oauth.provider.type.dlrwpdauksdev001.dfs.core.windows.net": "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider",
<         "spark.hadoop.fs.azure.account.oauth2.client.endpoint.dlrwpdauksdev001.dfs.core.windows.net": "{{secrets/dataplatform/datalake-authority-uri}}",
<         "spark.hadoop.fs.azure.account.oauth2.client.id.dlrwpdauksdev001.dfs.core.windows.net": "{{secrets/dataplatform/datalake-client-id}}",
<         "spark.hadoop.fs.azure.account.oauth2.client.secret.dlrwpdauksdev001.dfs.core.windows.net": "{{secrets/dataplatform/datalake-client-secret}}",
<         "spark.scheduler.listenerbus.eventqueue.databricks.capacity": "1000000",
<         "spark.sql.streaming.stopTimeout": "60s",
<         "spark.storage.decommission.enabled": "true",
<         "spark.storage.decommission.rddBlocks.enabled": "true",
<         "spark.storage.decommission.shuffleBlocks.enabled": "true",
<         "spark.storage.decommission.shuffleBlocks.refreshLocationsEnabled": "true"
<       },
<       "spark_context_id": 6147402479108993279,
<       "spark_version": "dlt:14.1-delta-pipelines-dlt-release-2024.15-rc1-commit-9b76dca-image-bc65e26",
<       "start_time": 1714381139066,
<       "state": "RUNNING",
<       "state_message": ""
<     },
<     "... (34 additional elements)"
<   ]
< }
09:02:26 INFO [databricks.labs.ucx.workspace_access.generic] {MainThread} Listed clusters in 0:00:00.096570
09:02:26 DEBUG [databricks.sdk] {MainThread} GET /api/2.0/policies/clusters/list
< 200 OK
< {
<   "policies": [
<     {
<       "created_at_timestamp": 1702628476000,
<       "definition": "{\"node_type_id\": {\"type\": \"fixed\", \"value\": \"Standard_E16_v3\", \"hidden\": true} }",
<       "description": "Limits the cluster node size to Standard_F4",
<       "is_default": false,
<       "name": "actionable_insights",
<       "policy_id": "000016672541A23B"
<     },
<     "... (9 additional elements)"
<   ],
<   "total_count": 10
< }
09:02:26 INFO [databricks.labs.ucx.workspace_access.generic] {MainThread} Listed cluster-policies in 0:00:00.068760
09:02:26 DEBUG [databricks.sdk] {MainThread} GET /api/2.0/instance-pools/list
< 200 OK
< {}
09:02:26 INFO [databricks.labs.ucx.workspace_access.generic] {MainThread} Listed instance-pools in 0:00:00.044191
09:02:26 DEBUG [databricks.sdk] {MainThread} GET /api/2.0/sql/warehouses
< 200 OK
< {
<   "warehouses": [
<     {
<       "auto_resume": true,
<       "auto_stop_mins": 60,
<       "channel": {},
<       "cluster_size": "2X-Small",
<       "creator_id": 5673461684272674,
<       "creator_name": "87507d41-ffd1-4f56-8ddd-81edb0f12563",
<       "enable_photon": true,
<       "enable_serverless_compute": false,
<       "health": {
<         "status": "HEALTHY"
<       },
<       "id": "625bc13752520b8b",
<       "jdbc_url": "jdbc:spark://**HOSTNAME REDACTED**:443/default;transportMode=http;ssl=1;Aut... (54 more bytes)",
<       "max_num_clusters": 1,
<       "min_num_clusters": 1,
<       "name": "Starter Warehouse",
<       "num_active_sessions": 0,
<       "num_clusters": 1,
<       "odbc_params": {
<         "hostname": "**HOSTNAME REDACTED**",
<         "path": "/sql/1.0/warehouses/625bc13752520b8b",
<         "port": 443,
<         "protocol": "https"
<       },
<       "size": "XXSMALL",
<       "spot_instance_policy": "COST_OPTIMIZED",
<       "state": "RUNNING",
<       "tags": {},
<       "warehouse_type": "PRO"
<     }
<   ]
< }
09:02:26 INFO [databricks.labs.ucx.workspace_access.generic] {MainThread} Listed sql/warehouses in 0:00:00.036004
09:02:26 DEBUG [databricks.sdk] {MainThread} GET /api/2.1/jobs/list
< 200 OK
< {
<   "has_more": false,
<   "jobs": [
<     {
<       "created_time": 1714380209513,
<       "creator_user_name": "**EMAIL REDACTED**",
<       "job_id": 602737855593523,
<       "settings": {
<         "email_notifications": {
<           "on_failure": [
<             "**EMAIL REDACTED**"
<           ],
<           "on_success": [
<             "**EMAIL REDACTED**"
<           ]
<         },
<         "format": "MULTI_TASK",
<         "max_concurrent_runs": 1,
<         "name": "[UCX] assessment",
<         "tags": {
<           "version": "v0.22.0"
<         },
<         "timeout_seconds": 0
<       }
<     },
<     "... (14 additional elements)"
<   ]
< }
09:02:26 INFO [databricks.labs.ucx.workspace_access.generic] {MainThread} Listed jobs in 0:00:00.114157
09:02:26 DEBUG [databricks.sdk] {MainThread} GET /api/2.0/pipelines
< 200 OK
< {
<   "statuses": [
<     {
<       "creator_user_name": "87507d41-ffd1-4f56-8ddd-81edb0f12563",
<       "name": "survey_trend_analysis",
<       "pipeline_id": "0155011d-53b0-41ad-bf0e-3bd9985f3cd2",
<       "run_as_user_name": "87507d41-ffd1-4f56-8ddd-81edb0f12563",
<       "state": "IDLE"
<     },
<     "... (18 additional elements)"
<   ]
< }
09:02:26 INFO [databricks.labs.ucx.workspace_access.generic] {MainThread} Listed pipelines in 0:00:00.085585
09:02:26 DEBUG [databricks.sdk] {MainThread} GET /api/2.0/serving-endpoints
< 404 Not Found
< {
<   "error_code": "FEATURE_DISABLED",
<   "message": "Model serving is not enabled for your shard. Please contact your organization admin or Databrick... (10 more bytes)"
< }
09:02:26 ERROR [databricks.labs.ucx] {MainThread} Execute `databricks workspace export //Applications/ucx/logs/assessment/run-227626125051970-0/crawl_permissions.log` locally to troubleshoot with more details. Model serving is not enabled for your shard. Please contact your organization admin or Databricks support.
09:02:26 DEBUG [databricks] {MainThread} Task crash details
Traceback (most recent call last):
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/databricks/labs/ucx/runtime.py", line 90, in trigger
    current_task(ctx)
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/databricks/labs/ucx/assessment/workflows.py", line 172, in crawl_permissions
    permission_manager.inventorize_permissions()
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/databricks/labs/ucx/workspace_access/manager.py", line 30, in inventorize_permissions
    crawler_tasks = list(self._get_crawler_tasks())
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/databricks/labs/ucx/workspace_access/manager.py", line 163, in _get_crawler_tasks
    yield from support.get_crawler_tasks()
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/databricks/labs/ucx/workspace_access/generic.py", line 86, in get_crawler_tasks
    for info in listing:
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/databricks/labs/ucx/workspace_access/generic.py", line 58, in __iter__
    for item in self._func():
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/databricks/sdk/service/serving.py", line 2602, in list
    json = self._api.do('GET', '/api/2.0/serving-endpoints', headers=headers)
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/databricks/sdk/core.py", line 130, in do
    response = retryable(self._perform)(method,
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/databricks/sdk/retries.py", line 54, in wrapper
    raise err
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/databricks/sdk/retries.py", line 33, in wrapper
    return func(*args, **kwargs)
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/databricks/sdk/core.py", line 238, in _perform
    raise self._make_nicer_error(response=response, **payload) from None
databricks.sdk.errors.platform.NotFound: Model serving is not enabled for your shard. Please contact your organization admin or Databricks support.
@nkvuong nkvuong mentioned this issue May 3, 2024
Merged
2 tasks
nfx pushed a commit that referenced this issue May 3, 2024
@nkvuong
Addressed issue with disabled features in certain regions (#1618)
9ca2d82 
## Changes
- Added error handling when listing serving endpoints, as it will raise
NotFound error if the feature is disabled.

### Linked issues
<!-- DOC: Link issue with a keyword: close, closes, closed, fix, fixes,
fixed, resolve, resolves, resolved. See
https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword
-->

Resolves #1617


### Tests
<!-- How is this tested? Please see the checklist below and also
describe any other relevant tests -->

- [x] manually tested
- [x] added unit tests
@nfx nfx closed this as completed in #1618 May 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
needs-triage
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Addressed issue with disabled features in certain regions databrickslabs/ucx
1 participant
@RobMandersBJSS