Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] Crash when dbt runs show grants statement due to column name capitalization mismatch #1086

Closed
2 tasks done
lamalex opened this issue Aug 3, 2024 · 7 comments · May be fixed by #1087
Closed
2 tasks done

[Bug] Crash when dbt runs show grants statement due to column name capitalization mismatch #1086

lamalex opened this issue Aug 3, 2024 · 7 comments · May be fixed by #1087
Labels
bug Something isn't working grants Issues related to dbt's grants functionality

Comments

@lamalex
Copy link

lamalex commented Aug 3, 2024

Is this a new bug in dbt-spark?

  • I believe this is a new bug in dbt-spark
  • I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

My DBT build is failing (in certain environments, have not been able to pinpoint what the difference is) due to column name capitalization mismatch from a show grants statement.

In my dbt-spark library i've added a print on the grants table before columns are accessed by name, and I am seeing lower cased column names instead of upper-case (which does not match when I run this statement in databricks sql console!)

| column     | data_type |
| ---------- | --------- |
| principal  | Text      |
| actiontype | Text      |
| objecttype | Text      |
| objectkey  | Text      |

if I change the column names in the adapter my models build successfully.
This is occurring on my workstation, as well as in our Dagster instance, but is not occurring in a Databricks workflow job (which are all using dbt-core 1.8.4, and dbt-databricks 1.8.3(or 4), and dbt-spark 1.8.0).

Expected Behavior

not crash

Steps To Reproduce

run dbt build --profile databricks --select tag:elementary --log-level debug

Relevant log output

15:35:00  Using databricks connection "model.elementary.metadata"
15:35:00  On model.elementary.metadata: /* {"app": "dbt", "dbt_version": "1.8.4", "dbt_databricks_version": "1.8.4", "databricks_sql_connector_version": "3.1.2", "profile_name": "databricks", "target_name": "dev", "node_id": "model.elementary.metadata"} */


    show grants on `launi-databricks_catalog_nonprod`.`zzz_launi_alauni`.`metadata`


15:35:00  Databricks adapter: Cursor(session-id=01ef51ad-d40d-1dc7-be88-71c780641be1, command-id=Unknown) - Created cursor
15:35:01  SQL status: OK in 0.4000000059604645 seconds
15:35:01  Databricks adapter: Cursor(session-id=01ef51ad-d40d-1dc7-be88-71c780641be1, command-id=01ef51ad-f259-1668-889b-9ea8e4fa88ba) - Closing cursor
| column     | data_type |
| ---------- | --------- |
| principal  | Text      |
| actiontype | Text      |
| objecttype | Text      |
| objectkey  | Text      |

15:35:01  Databricks adapter: DatabricksDBTConnection(id=5338871184, session-id=01ef51ad-d40d-1dc7-be88-71c780641be1, name=model.elementary.metadata, idle-time=4.0531158447265625e-06s, acquire-count=0, language=sql, thread-identifier=(60914, 12901707776), compute-name=) - Released connection
15:35:01  Unhandled error while executing target/run/elementary/models/edr/system/metadata.sql
'Principal'
15:35:01  Traceback (most recent call last):
  File "/Users/alauni/Code/launi/launi-dbt/.venv/lib/python3.12/site-packages/dbt/task/base.py", line 368, in safe_run
    result = self.compile_and_execute(manifest, ctx)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alauni/Code/launi/launi-dbt/.venv/lib/python3.12/site-packages/dbt/task/base.py", line 314, in compile_and_execute
    result = self.run(ctx.node, manifest)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alauni/Code/launi/launi-dbt/.venv/lib/python3.12/site-packages/dbt/task/base.py", line 415, in run
    return self.execute(compiled_node, manifest)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alauni/Code/launi/launi-dbt/.venv/lib/python3.12/site-packages/dbt/task/run.py", line 298, in execute
    result = MacroGenerator(
             ^^^^^^^^^^^^^^^
  File "/Users/alauni/Code/launi/launi-dbt/.venv/lib/python3.12/site-packages/dbt/clients/jinja.py", line 84, in __call__
    return self.call_macro(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alauni/Code/launi/launi-dbt/.venv/lib/python3.12/site-packages/dbt_common/clients/jinja.py", line 298, in call_macro
    return macro(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alauni/Code/launi/launi-dbt/.venv/lib/python3.12/site-packages/jinja2/runtime.py", line 768, in __call__
    return self._invoke(arguments, autoescape)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alauni/Code/launi/launi-dbt/.venv/lib/python3.12/site-packages/jinja2/runtime.py", line 782, in _invoke
    rv = self._func(*arguments)
         ^^^^^^^^^^^^^^^^^^^^^^
  File "<template>", line 75, in macro
  File "/Users/alauni/Code/launi/launi-dbt/.venv/lib/python3.12/site-packages/jinja2/sandbox.py", line 394, in call
    return __context.call(__obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alauni/Code/launi/launi-dbt/.venv/lib/python3.12/site-packages/jinja2/runtime.py", line 303, in call
    return __obj(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alauni/Code/launi/launi-dbt/.venv/lib/python3.12/site-packages/dbt/clients/jinja.py", line 84, in __call__
    return self.call_macro(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alauni/Code/launi/launi-dbt/.venv/lib/python3.12/site-packages/dbt_common/clients/jinja.py", line 298, in call_macro
    return macro(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alauni/Code/launi/launi-dbt/.venv/lib/python3.12/site-packages/jinja2/runtime.py", line 768, in __call__
    return self._invoke(arguments, autoescape)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alauni/Code/launi/launi-dbt/.venv/lib/python3.12/site-packages/jinja2/runtime.py", line 782, in _invoke
    rv = self._func(*arguments)
         ^^^^^^^^^^^^^^^^^^^^^^
  File "<template>", line 25, in macro
  File "/Users/alauni/Code/launi/launi-dbt/.venv/lib/python3.12/site-packages/jinja2/sandbox.py", line 394, in call
    return __context.call(__obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alauni/Code/launi/launi-dbt/.venv/lib/python3.12/site-packages/jinja2/runtime.py", line 303, in call
    return __obj(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alauni/Code/launi/launi-dbt/.venv/lib/python3.12/site-packages/dbt/clients/jinja.py", line 84, in __call__
    return self.call_macro(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alauni/Code/launi/launi-dbt/.venv/lib/python3.12/site-packages/dbt_common/clients/jinja.py", line 298, in call_macro
    return macro(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alauni/Code/launi/launi-dbt/.venv/lib/python3.12/site-packages/jinja2/runtime.py", line 768, in __call__
    return self._invoke(arguments, autoescape)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alauni/Code/launi/launi-dbt/.venv/lib/python3.12/site-packages/jinja2/runtime.py", line 782, in _invoke
    rv = self._func(*arguments)
         ^^^^^^^^^^^^^^^^^^^^^^
  File "<template>", line 48, in macro
  File "/Users/alauni/Code/launi/launi-dbt/.venv/lib/python3.12/site-packages/jinja2/sandbox.py", line 394, in call
    return __context.call(__obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alauni/Code/launi/launi-dbt/.venv/lib/python3.12/site-packages/jinja2/runtime.py", line 303, in call
    return __obj(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alauni/Code/launi/launi-dbt/.venv/lib/python3.12/site-packages/dbt/adapters/databricks/utils.py", line 78, in wrapper
    return func(*new_args, **new_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alauni/Code/launi/launi-dbt/.venv/lib/python3.12/site-packages/dbt/adapters/spark/impl.py", line 493, in standardize_grants_dict
    grantee = row["Principal"]
              ~~~^^^^^^^^^^^^^
  File "/Users/alauni/Code/launi/launi-dbt/.venv/lib/python3.12/site-packages/agate/mapped_sequence.py", line 88, in __getitem__
    return self.dict()[key]
           ~~~~~~~~~~~^^^^^
KeyError: 'Principal'


### Environment

```markdown
- OS: MacOS 14.6
- Python: 3.12.3
- dbt-core: 1.8.4
- dbt-spark: 1.8.0

Additional Context

No response
@lamalex lamalex added bug Something isn't working triage labels Aug 3, 2024
lamalex added a commit to lamalex/dbt-spark that referenced this issue Aug 3, 2024
@lamalex
Update standardize_grants_dict to handle potential case mismatches …
c1bae97 
…in grants table response

Match case to column names returned rather than hardcoding.
Fixes: dbt-labs#1086
@lamalex lamalex mentioned this issue Aug 3, 2024
Open
4 tasks
lamalex added a commit to lamalex/dbt-spark that referenced this issue Aug 3, 2024
@lamalex
Update standardize_grants_dict to handle potential case mismatches …
281fd60 
…in grants table response

Match case to column names returned rather than hardcoding.
Fixes: dbt-labs#1086
@amychen1776
Copy link
Contributor

@lamalex Can I confirm with you that you're using dbt-databricks and dbt-spark for your Databricks workflows? Is there a reason why you aren't just using dbt-databricks?

@lamalex
Copy link
Author

lamalex commented Aug 28, 2024

@amychen1776 i am using dbt-databricks, but dbt-spark is a dependency of dbt-databricks

@amychen1776
Copy link
Contributor

@lamalex You're very correct here. However you're currently using dbt-databricks and the issue you're raising is in comparison to the Databricks interface. It would make the most sense to start with the Databricks team - since I can't transfer issues to repositories outside of my org, would you please open up an issue on the Databricks repo for them to take a look?

@lamalex
Copy link
Author

lamalex commented Aug 31, 2024

@amychen1776 are you saying that although the code for this bug exists in dbt-spark, it's the dbt-databricks team who should be responsible for the fix?

@lamalex lamalex mentioned this issue Aug 31, 2024
Open
2 tasks
@amychen1776
Copy link
Contributor

Yes - they can also open up issues and prs for this adapter as needed but for initial triage, it should fall to the dbt-databricks team first. Thank you for opening up the issue!

@amychen1776 amychen1776 closed this as not planned Won't fix, can't repro, duplicate, stale Sep 5, 2024
@amychen1776 amychen1776 removed the triage label Sep 5, 2024
@lamalex
Copy link
Author

lamalex commented Sep 5, 2024

@amychen1776 Is there something wrong with the PR i opened? I mean I’m not asking anyone else to fix it!

@amychen1776
Copy link
Contributor

amychen1776 commented Sep 18, 2024 •
edited
Loading

@lamalex not at all! It's more I want to make sure Databricks is aware of this

@amychen1776 amychen1776 added the grants Issues related to dbt's grants functionality label Sep 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working grants Issues related to dbt's grants functionality
Projects
None yet
Milestone
No milestone
2 participants
@lamalex @amychen1776