-
Notifications
You must be signed in to change notification settings - Fork 495
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add quote_identifiers
parameter to unpivot
to handle case-sensitive column names
#792
Add quote_identifiers
parameter to unpivot
to handle case-sensitive column names
#792
Conversation
876fba7
to
32ccf3d
Compare
605dcaa
to
d2de2de
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @error418! The changes to the macro look good; I have a few nitpicks around the tests though ⛏️
d2de2de
to
9f0ff03
Compare
97a00ed
to
bcb1b1f
Compare
bcb1b1f
to
03467be
Compare
rebased to current main and resolved merge conflicts in |
Anything missing to get this merged? |
This PR has been marked as Stale because it has been open with no activity as of late. If you would like the PR to remain open, please comment on the PR or else it will be closed in 7 days. |
ping |
hey @error418 @joellabes, facing a similar issue and need to have this merged. can you merge this asap please? |
@Shadowsong27 @Ishankoradia @Abhishek-N at the moment I am waiting for a review by @joellabes (or any maintainer of @Ishankoradia thank you for testing the changes for BigQuery Should there be any remaining issues with this PR I am happy to provide the necessary fixes. |
bump |
What's missing here? I need this fix to be deployed. Thanks. |
What's missing here? I need this fix to be deployed. Thanks. |
@joellabes could you please provide some information about the status of this pull request? This is stalling for a while now without any feedback from the maintainers, which sadly becomes a bit frustrating. Thanks |
Hi all, This is our highest priority PR to review right now, and I'll be diving deeper into it this week. After taking an initial look, the heart of the implementation looks good. The main thing I'd like to follow-up on prior to approving + merging is the testing:
So I'd like to:
|
@error418 I think I figured out the root cause of the Redshift test failures in CI. As you noted, the default behavior in Redshift is to lowercase all column names even if a table is created with quoted mixed-case column names. It looks like the solution is to set |
quote_identifiers
parameter to unpivot
to handle case-sensitive column names
@@ -12,11 +12,11 @@ Arguments: | |||
value_name: Destination table column name for the pivoted values | |||
#} | |||
|
|||
{% macro unpivot(relation=none, cast_to='varchar', exclude=none, remove=none, field_name='field_name', value_name='value') -%} | |||
{{ return(adapter.dispatch('unpivot', 'dbt_utils')(relation, cast_to, exclude, remove, field_name, value_name)) }} | |||
{% macro unpivot(relation=none, cast_to='varchar', exclude=none, remove=none, field_name='field_name', value_name='value', quote_identifiers=False) -%} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In an ideal world, I would rather have the default be quote_identifiers=True
so that it matches with pivot
. But agree with your choice here to default it to False
instead so we don't have any breaking behavior change.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We're ready to rock @error418 🎸
The tests in Redshift are passing after setting enable_case_sensitive_identifier
to true
in the cluster’s parameter group via the Amazon Redshift console.
resolves #216
Description & motivation
Added
quote_identifiers
parameter tounpivot
to handle case-sensitive column names.This is like #135 was for
pivot
.Checklist
star()
source)limit_zero()
macro in place of the literal string:limit 0
dbt.type_*
macros instead of explicit datatypes (e.g.dbt.type_timestamp()
instead ofTIMESTAMP