Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CT-997] [Feature] Support dbt.var + dbt.env_var in Python models #5617

Open
3 tasks done
jtcohen6 opened this issue Aug 5, 2022 · 3 comments · May be fixed by #10932 or dbt-labs/dbt-adapters#339
Open
3 tasks done

[CT-997] [Feature] Support dbt.var + dbt.env_var in Python models #5617

jtcohen6 opened this issue Aug 5, 2022 · 3 comments · May be fixed by #10932 or dbt-labs/dbt-adapters#339
Labels
enhancement New feature or request python_models stale Issues that have gone stale

Comments

@jtcohen6
Copy link
Contributor

jtcohen6 commented Aug 5, 2022

Is this your first time submitting a feature request?

  • I have read the expectations for open source contributors
  • I have searched the existing issues, and I could not find an existing issue for this feature
  • I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

Describe the feature

Copying from thread in #beta-feedback-python-models:

A quick thought, why not expose a dbt.var function (and similar for env var) rather than overloading config? If you apply the same limitations as for config function (ie. Only literal args) should be pretty straightforward to statically analyse which vars need to be injected?

It's true! The mechanism for dbt.var and dbt.env_var could be just the same as dbt.config.get. We detect the literal arguments passed in:

if func_name in ["dbt.ref", "dbt.source", "dbt.config", "dbt.config.get"]:
# drop the dot-dbt prefix
func_name = func_name.split(".")[-1]
args, kwargs = self._get_call_literals(node)
self.dbt_function_calls.append((func_name, args, kwargs))

if func == "get":
context["config"](utilized=args)
continue

And then provide those values at "compile" time (which we're currently doing via Jinja... gross), as new methods (var + env_var) on the dbtObj class:

{%- for key in model.config.utilized -%}
{# TODO: weird type testing with enum, would be much easier to write this logic in Python! #}
{%- if key == 'language' -%}
{%- set value = 'python' -%}
{%- endif -%}
{%- set value = model.config[key] -%}
{%- do config_dict.update({key: value}) -%}
{%- endfor -%}

class dbtObj:
def __init__(self, load_df_function) -> None:
self.source = lambda x: source(x, dbt_load_df_function=load_df_function)
self.ref = lambda x: ref(x, dbt_load_df_function=load_df_function)
self.config = config
self.this = this()
self.is_incremental = {{ is_incremental() }}

Describe alternatives you've considered

Not doing this, and continuing to ask users (as we are currently) to pass those vars + env_vars in as configs in .yml files:

version: 2

models:
  - name: my_python_model
    config:
      materialized: table
      target_name: "{{ target.name }}"
      specific_var: "{{ var('SPECIFIC_VAR') }}"
      specific_env_var: "{{ env_var('SPECIFIC_ENV_VAR') }}"

Who will this benefit?

Usability and ergonomics for people developing and configuring Python models, especially ones with more advanced conditional logic or env-specific behavior

Are you interested in contributing this feature?

No response

Anything else?

To be clear, the environment variable values would need to be the ones evaluated during dbt parsing. (They are NOT the environment variables set in the runtime environment, e.g. Snowpark.)

That's slightly different from how environment variables work in SQL models today, since the {{ env_var() }} Jinja is re-rendered at compile/execute runtime.

But I think it probably should work this way across the board, where all inputs (including env vars) are resolved during parsing and pre-execution.

@github-actions github-actions bot changed the title [Feature] Support dbt.var + dbt.env_var in Python models [CT-997] [Feature] Support dbt.var + dbt.env_var in Python models Aug 5, 2022
@github-actions
Copy link
Contributor

github-actions bot commented Feb 2, 2023

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

@github-actions github-actions bot added the stale Issues that have gone stale label Feb 2, 2023
@lostmygithubaccount lostmygithubaccount removed the stale Issues that have gone stale label Feb 2, 2023
Copy link
Contributor

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

@github-actions github-actions bot added the stale Issues that have gone stale label Feb 23, 2024
@devmessias
Copy link
Contributor

I believe this relates to #10914

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request python_models stale Issues that have gone stale
Projects
None yet
3 participants