google.api_core.exceptions.InvalidArgument: 400 The requested routine projects/c2c-dwh-dev/datasets/jsons_rep_data_warehouse/routines/mask_name is not found or an invalid masking routine #13313

nyck33 · 2024-11-30T10:57:33Z

Determine this is the right repository

I determined this is the correct repository in which to report this bug.

Summary of the issue

Context
trying to set data policy to map policy tags to UDFs in my BigQuery Dataset but get this error:

(bigquery) nyck33@lenovo-gtx1650:/mnt/d/c2c/sdp-masking-nov28-2024/policy_tags_udfs/data_policies$ p
ython map_datapolicy_to_udf.py
Traceback (most recent call last):
  File "/mnt/d/c2c/sdp-masking-nov28-2024/policy_tags_udfs/data_policies/map_datapolicy_to_udf.py", line 69, in <module>
    response = client.create_data_policy(
  File "/home/nyck33/miniconda3/envs/bigquery/lib/python3.9/site-packages/google/cloud/bigquery_datapolicies_v1/services/data_policy_service/client.py", line 769, in create_data_policy
    response = rpc(
  File "/home/nyck33/miniconda3/envs/bigquery/lib/python3.9/site-packages/google/api_core/gapic_v1/method.py", line 131, in __call__
    return wrapped_func(*args, **kwargs)
  File "/home/nyck33/miniconda3/envs/bigquery/lib/python3.9/site-packages/google/api_core/retry/retry_unary.py", line 293, in retry_wrapped_func
    return retry_target(
  File "/home/nyck33/miniconda3/envs/bigquery/lib/python3.9/site-packages/google/api_core/retry/retry_unary.py", line 153, in retry_target
    _retry_error_helper(
  File "/home/nyck33/miniconda3/envs/bigquery/lib/python3.9/site-packages/google/api_core/retry/retry_base.py", line 212, in _retry_error_helper
    raise final_exc from source_exc
  File "/home/nyck33/miniconda3/envs/bigquery/lib/python3.9/site-packages/google/api_core/retry/retry_unary.py", line 144, in retry_target

Expected Behavior:
this code to map my policy tag on columns on my table to the JavaScript UDF I have stored in my dataset:

from google.cloud import bigquery_datapolicies_v1
import json
import logging
import os
from google.cloud import bigquery
from google.cloud.bigquery.dataset import AccessEntry
from google.oauth2 import service_account

from dotenv import load_dotenv

# Configure logging
logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s - %(levelname)s - %(message)s"
)

# Load environment variables from a .env file
load_dotenv()

# Retrieve environment variables
SERVICE_ACCOUNT_JSON = os.getenv("GCP_C2C_SERVICE_ACCOUNT_JSON")
PROJECT_ID = os.getenv("GCP_C2C_PROJECT_ID")
DATASET_ID = "jsons_rep_data_warehouse"
#MASKED_READER_EMAIL = os.getenv("MASKEDREADER_EMAIL")
#FINEGRAINEDREADER_EMAIL = os.getenv("FINEGRAINEDREADER_EMAIL")

credentials = service_account.Credentials.from_service_account_file(SERVICE_ACCOUNT_JSON)
# Initialize the Data Policy client
client = bigquery_datapolicies_v1.DataPolicyServiceClient(credentials=credentials)

# Define the project and location
project_id = PROJECT_ID
location = 'us'  # e.g., 'us-central1'

# Specify the policy tag and UDF
policy_tag = 'projects/c2c-dwh-dev/locations/us/taxonomies/2093754720879615579/policyTags/2621461886793509723'
udf_resource_name = 'projects/c2c-dwh-dev/datasets/jsons_rep_data_warehouse/routines/mask_name'

# Define the data policy
'''
data_policy = bigquery_datapolicies_v1.DataPolicy(
    policy_tag=policy_tag,
    data_masking_policy=bigquery_datapolicies_v1.DataMaskingPolicy(
        predefined_expression=bigquery_datapolicies_v1.DataMaskingPolicy.PredefinedExpression.CUSTOMER_SUPPLIED, # does not exist
        routine=udf_resource_name
    )
)
'''
# Define the data masking policy with the custom routine
data_masking_policy = bigquery_datapolicies_v1.DataMaskingPolicy(
    routine=udf_resource_name
)



# Create the data policy with a valid data_policy_id
parent = f'projects/{project_id}/locations/{location}'
data_policy_id = 'mask_jp_name_fields'  # Replace with a valid identifier

# Define the data policy with a unique name
data_policy = bigquery_datapolicies_v1.DataPolicy(
    name=f'projects/{project_id}/locations/{location}/dataPolicies/{data_policy_id}',
    data_policy_id=data_policy_id,
    policy_tag=policy_tag,
    data_masking_policy=data_masking_policy,
    data_policy_type=bigquery_datapolicies_v1.DataPolicy.DataPolicyType.DATA_MASKING_POLICY,

)
response = client.create_data_policy(
    parent=parent,
    data_policy=data_policy
)


print(f'Data Policy created: {response.name}')

Actual Behavior:
I get that error so are JS UDFs not allowed? It says invalid masking routine but it works if I invoke it in the Console.

API client name and version

google-cloud-bigquery-datapolicies 0.6.10

Reproduction steps: code

from google.cloud import bigquery_datapolicies_v1
import json
import logging
import os
from google.cloud import bigquery
from google.cloud.bigquery.dataset import AccessEntry
from google.oauth2 import service_account

from dotenv import load_dotenv

# Configure logging
logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s - %(levelname)s - %(message)s"
)

# Load environment variables from a .env file
load_dotenv()

# Retrieve environment variables
SERVICE_ACCOUNT_JSON = os.getenv("GCP_C2C_SERVICE_ACCOUNT_JSON")
PROJECT_ID = os.getenv("GCP_C2C_PROJECT_ID")
DATASET_ID = "jsons_rep_data_warehouse"
#MASKED_READER_EMAIL = os.getenv("MASKEDREADER_EMAIL")
#FINEGRAINEDREADER_EMAIL = os.getenv("FINEGRAINEDREADER_EMAIL")

credentials = service_account.Credentials.from_service_account_file(SERVICE_ACCOUNT_JSON)
# Initialize the Data Policy client
client = bigquery_datapolicies_v1.DataPolicyServiceClient(credentials=credentials)

# Define the project and location
project_id = PROJECT_ID
location = 'us'  # e.g., 'us-central1'

# Specify the policy tag and UDF
policy_tag = 'projects/c2c-dwh-dev/locations/us/taxonomies/2093754720879615579/policyTags/2621461886793509723'
udf_resource_name = 'projects/c2c-dwh-dev/datasets/jsons_rep_data_warehouse/routines/mask_name'

# Define the data policy
'''
data_policy = bigquery_datapolicies_v1.DataPolicy(
    policy_tag=policy_tag,
    data_masking_policy=bigquery_datapolicies_v1.DataMaskingPolicy(
        predefined_expression=bigquery_datapolicies_v1.DataMaskingPolicy.PredefinedExpression.CUSTOMER_SUPPLIED, # does not exist
        routine=udf_resource_name
    )
)
'''
# Define the data masking policy with the custom routine
data_masking_policy = bigquery_datapolicies_v1.DataMaskingPolicy(
    routine=udf_resource_name
)



# Create the data policy with a valid data_policy_id
parent = f'projects/{project_id}/locations/{location}'
data_policy_id = 'mask_jp_name_fields'  # Replace with a valid identifier

# Define the data policy with a unique name
data_policy = bigquery_datapolicies_v1.DataPolicy(
    name=f'projects/{project_id}/locations/{location}/dataPolicies/{data_policy_id}',
    data_policy_id=data_policy_id,
    policy_tag=policy_tag,
    data_masking_policy=data_masking_policy,
    data_policy_type=bigquery_datapolicies_v1.DataPolicy.DataPolicyType.DATA_MASKING_POLICY,

)
response = client.create_data_policy(
    parent=parent,
    data_policy=data_policy
)


print(f'Data Policy created: {response.name}')```

### Reproduction steps: supporting files

_No response_

### Reproduction steps: actual results

_No response_

### Reproduction steps: expected results

_No response_

### OS & version + platform

Ubuntu 24.04 WSL

### Python environment

Python 3.9.20

### Python dependencies

onda list nyck33@lenovo-gtx1650:/mnt/d/c2c/sdp-masking-nov28-2024/policy_tags_udfs/data_policies$

packages in environment at /home/nyck33/miniconda3/envs/bigquery:


### Additional context

_No response_

The text was updated successfully, but these errors were encountered:

chalmerlowe · 2024-12-02T17:01:09Z

@nyck33
Thanks for submitting a question.
The error (as shown in the title) says that there are two possible reasons for this error:
google.api_core.exceptions.InvalidArgument: 400 The requested routine projects/c2c-dwh-dev/datasets/jsons_rep_data_warehouse/routines/mask_name

is not found
or an invalid masking routine

If the routine works in the Console but does not appear to work when invoked via the client, I would start first with considering whether the path to the routine as used in the client might have a subtle error. Are we sure about the path to the routine and the routine's name?

'projects/c2c-dwh-dev/datasets/jsons_rep_data_warehouse/routines/mask_name'

nyck33 added triage me I really want to be triaged. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. labels Nov 30, 2024

chalmerlowe self-assigned this Dec 2, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

google.api_core.exceptions.InvalidArgument: 400 The requested routine projects/c2c-dwh-dev/datasets/jsons_rep_data_warehouse/routines/mask_name is not found or an invalid masking routine #13313

google.api_core.exceptions.InvalidArgument: 400 The requested routine projects/c2c-dwh-dev/datasets/jsons_rep_data_warehouse/routines/mask_name is not found or an invalid masking routine #13313

nyck33 commented Nov 30, 2024

chalmerlowe commented Dec 2, 2024

google.api_core.exceptions.InvalidArgument: 400 The requested routine projects/c2c-dwh-dev/datasets/jsons_rep_data_warehouse/routines/mask_name is not found or an invalid masking routine #13313

google.api_core.exceptions.InvalidArgument: 400 The requested routine projects/c2c-dwh-dev/datasets/jsons_rep_data_warehouse/routines/mask_name is not found or an invalid masking routine #13313

Comments

nyck33 commented Nov 30, 2024

Determine this is the right repository

Summary of the issue

API client name and version

Reproduction steps: code

packages in environment at /home/nyck33/miniconda3/envs/bigquery:

Name Version Build Channel

chalmerlowe commented Dec 2, 2024