Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

When using --empty, adapter.get_columns_in_relation() returns an error #896

Open
1 of 5 tasks
zeev-finaloop opened this issue May 23, 2024 · 4 comments
Open
1 of 5 tasks
Labels
bug Something isn't working triage

Comments

@zeev-finaloop
Copy link

Describe the bug

The adapter.get_columns_in_relation and dbt_utils.get_filtered_columns_in_relation on Snowflake are basically running the query describe table YOUR_TABLE. When running dbt build/run with the --empty flag, the generated query is describe table (select * from YOUR_TABLE where false limit 0). The latter is a broken syntax query and throws an error.

Steps to reproduce

Just build any macro with:
{%- set relation_column_names = adapter.get_columns_in_relation(your_model) -%}
Run dbt build --empty -d

Expected results

List of columns

Actual results

You will get something like:
001003 (42000): SQL compilation error: syntax error line 1 at position 15 unexpected '('.

Screenshots and log output

System information

The contents of your packages.yml file:

Which database are you using dbt with?

  • postgres
  • redshift
  • bigquery
  • snowflake
  • other (specify: ____________)

The output of dbt --version:

Core:
  - installed: 1.8.1
  - latest:    1.8.1 - Up to date!

Plugins:
  - snowflake: 1.8.2 - Up to date!

Additional context

Are you interested in contributing the fix?

@zeev-finaloop zeev-finaloop added bug Something isn't working triage labels May 23, 2024
@dataders
Copy link
Contributor

thanks for flagging @zeev-finaloop! we're actually aware of the issue with adapter.get_columns_in_relation() and --empty -- we're tracking it in dbt-labs/dbt-snowflake#1033 (somewhat related: dbt-labs/dbt-adapters#213).

@dataders dataders changed the title When using --empty adapter.get_columns_in_relation return an error When using --empty, adapter.get_columns_in_relation() returns an error May 23, 2024
@dataders
Copy link
Contributor

I'm tempted to close this as the problem and solution exists outside of the dbt-utils package, but I'll leave it open for now so that users encountering the same issue may discover it here

@kokorin
Copy link

kokorin commented Jun 3, 2024

I found this DBT Core issue: CT-1919 Create get_column_schema_from_query macro and it looks get_column_schema_from_query macro can help with --empty.

@kokorin
Copy link

kokorin commented Jun 3, 2024

Here is a workaround which makes union_releations to work with --empty:

{%- macro union_relations(relations, column_override=none, include=[], exclude=[], source_column_name='_dbt_source_relation', where=none) -%}
    {% if invocation_args_dict.get('empty', False) %}
        {% set actual_relations = [] %}
        {% for relation in relations %}
            {% set database, schema, identifier = relation.render().split('.') %}
            {%- set actual_relation = adapter.get_relation(database, schema, identifier) -%}
            {% do actual_relations.append(actual_relation) %}      
        {% endfor %}
        {% set relations = actual_relations %}
        {% set where = 'false limit 0'%}
    {% endif %}
    {{ return(adapter.dispatch('union_relations', 'dbt_utils')(relations, column_override, include, exclude, source_column_name, where)) }}
{% endmacro %}

The keys here are relation.render() which converts relation to full table name and set where = 'false limit 0' which applies exact same filter to relations.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working triage
Projects
None yet
Development

No branches or pull requests

3 participants