Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Invoking dbt as a module #2013

Closed
gouline opened this issue Dec 18, 2019 · 13 comments
Closed

Invoking dbt as a module #2013

gouline opened this issue Dec 18, 2019 · 13 comments
Labels
enhancement New feature or request good_first_issue Straightforward + self-contained changes, good for new contributors! stale Issues that have gone stale

Comments

@gouline
Copy link

gouline commented Dec 18, 2019

Describe the feature

It would be nice to have a straightforward way of running dbt as a Python module.

To give some context, I use my own build tool https://github.com/gouline/molot that's written in Python and provides basic support for targets, dependencies and arguments. Something half way between Make and Gradle that helps with CI configuration. Things that can be done in Python (e.g. boto3, snowflake-connector-python) are done natively, everything else it just calls shell commands in a subprocess like a Makefile would.

So I'd like a way to invoke multiple dbt commands (e.g. run and docs generate) in one target by just importing dbt.main and calling a function multiple times. The way things are now, main() calls sys.exit(), which exists my outer build as well, and handle_and_check() requires raw command-line arguments passed manually as a list. Ideally, there should be a function of the form dbt.main.execute(command='run', profiles_dir=PROFILES_DIR, ...) that would execute a command and throw any exceptions for the caller to handle.

Describe alternatives you've considered

My current workaround is just invoking dbt command as a subprocess, but it seems like a backward way of doing it, considering that both are Python applications.

Additional context

Closest thing I could find on the issue tracker was #1488, but that sounds more complex than what I'm proposing.

Who will this benefit?

This would benefit anyone who orchestrates their builds with Python scripts.

@gouline gouline added enhancement New feature or request triage labels Dec 18, 2019
@drewbanin drewbanin removed the triage label Dec 18, 2019
@drewbanin
Copy link
Contributor

Hey @gouline - kind of funny we didn't already have an issue for this one :)

We've kicked around the idea of providing a public API for dbt in Python a couple of times now. I'm happy for us to add a public method like dbt.main.execute(command='run', profiles_dir=PROFILES_DIR, ...) which implements logic similar to handle_and_check.

I'd still like to provide a rich Python-based interface for running dbt projects in the future. I think that would entail model selection, configuration, execution, etc, etc. In this case though, I think a top-level method like execute (or similar) gets us moving in the right direction.

Thanks for raising this!

@drewbanin drewbanin added the good_first_issue Straightforward + self-contained changes, good for new contributors! label Dec 20, 2019
@Dandandan
Copy link

Hello I like to help out on this issue.
Any pointers on how this should be implemented?

Should it just pass on the args to the parser?

@picousse
Copy link

This one is pretty old.
Is there any progress on this?

Is there something I/we could help with?

@mik-laj
Copy link
Contributor

mik-laj commented Oct 12, 2021

sys.exit() raises SystemExit exception, so we can use a try-except statement to handle this situation.
Here is example;
https://github.com/apache/airflow/blob/8505d2f0a4524313e3eff7a4f16b9a9439c7a79f/airflow/cli/commands/config_command.py#L40-L44
https://github.com/apache/airflow/blob/8505d2f0a4524313e3eff7a4f16b9a9439c7a79f/tests/cli/commands/test_config_command.py#L60-L80

To catch stdout/stderr, we can use contextlib.redirect_stdout/contextlib.redirect_stderr decorator.

with contextlib.redirect_stdout(io.StringIO()) as temp_stdout:
    __import__('dbt.main').main(argv)

@ismailsimsek
Copy link

+1 Is there any progress on this?

@WillemSFV
Copy link

+1 Is there any progress on this?

I suspect there won't be much progress on this now that DBT Cloud's a thing.

@github-actions
Copy link
Contributor

github-actions bot commented Aug 7, 2022

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days.

@github-actions github-actions bot added the stale Issues that have gone stale label Aug 7, 2022
@github-actions
Copy link
Contributor

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest; add a comment to notify the maintainers.

@ismailsimsek
Copy link

Hello I like to help out on this issue. Any pointers on how this should be implemented?

Should it just pass on the args to the parser?

+1

@jtcohen6
Copy link
Contributor

This is a fairly old issue. We are finally making progress in this direction, providing a programmatic API into dbt Core: #5527

The ability to invoke dbt as a module (instead of CLI script) isn't explicitly in scope for that initiative, but the big idea— providing a more sensible "main" method as entry-point to Core execution—certainly is.

@manugarri
Copy link

any updates on this? Seems like it would be easy to implement given how all of dbt-core is python already.

@jtcohen6
Copy link
Contributor

@joellabes
Copy link
Contributor

This issue has better google results than the actual feature's docs, so I'm going to leave a pointer here: https://docs.getdbt.com/reference/programmatic-invocations

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request good_first_issue Straightforward + self-contained changes, good for new contributors! stale Issues that have gone stale
Projects
None yet
Development

No branches or pull requests

10 participants