Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] query-comments aren't appended when a Manifest is passed to dbtRunner #9546

Closed
2 tasks done
racheldaniel opened this issue Feb 9, 2024 · 2 comments · Fixed by #9556
Closed
2 tasks done

[Bug] query-comments aren't appended when a Manifest is passed to dbtRunner #9546

racheldaniel opened this issue Feb 9, 2024 · 2 comments · Fixed by #9556
Assignees
Labels
backport 1.7.latest bug Something isn't working High Severity bug with significant impact that should be resolved in a reasonable timeframe Impact: Exp python_api Issues related to dbtRunner Python entry point

Comments

@racheldaniel
Copy link
Contributor

Is this a new bug in dbt-core?

  • I believe this is a new bug in dbt-core
  • I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

The IDE uses the dbtRunner as an entry point for dbt invocations. One common flow is to get the manifest obj returned after calling a parse as a part of the dbtRunnerResult, then pass it back to the runner obj for subsequent commands until the user makes a change to their project. We have observed that passing a Manifest object to the dbtRunner results in the removal of expected query-comments. We confirmed the comments are in the sql as expected as long as dbtRunner is instantiated without a cached manifest.

Expected Behavior

If a query-comment is defined, sql returned from dbt invocations could contain those comments regardless of how the dbtRunner is invoked

Steps To Reproduce

Here are reliable repro steps from a python repl (this is using my locally installed dbt 1.7.7).

  1. Add the following to a local dbt project's dbt_project.yml:
query-comment:
  comment: "comment {{ dbt_version }}"
  append: true
  1. Start a python repl on a dbt project in a venv with dbt installed and import the dbtRunner
(dbt-cloud-env) racheldaniel:jaffle_shop/ (master✗) $ python3
[12:40:18] Python 3.8.10 (default, Nov 29 2023, 10:42:27)
>>> from dbt.cli.main import dbtRunner
  1. Create an instance without a manifest and run a build
>>> dbt = dbtRunner()
>>> dbt.invoke(["--log-format=json", "--log-level=debug", "build", "--select", "stg_customers"])
  1. Search for your comment in the logs, it should be present (log truncated):
{"data": {"conn_name": "model.jaffle_shop.stg_customers", "node_info": {"materialized": "view", ... "sql": "create or replace  ... nselect * from renamed\n  )\n/* comment 1.7.7 */;", "name": "SQLQuery", "pid": 34081, "thread": "Thread-1", "ts": "2024-02-08T18:43:37.207812Z"}}
  1. Run a parse and store the result:
>>> result = dbt.invoke(["--log-format=json", "--log-level=debug", "parse"])
  1. Reinstantiate the dbtRunner using the manifest in that result, then call the same build:
>>> dbt = dbtRunner(manifest=result.result)
>>> dbt.invoke(["--log-format=json", "--log-level=debug", "build", "--select", "stg_customers"])
  1. Observe the query comment is now missing (log truncated):
{"data": {"conn_name": "model.jaffle_shop.stg_customers", "node_info": {"materialized": "view", ... "sql": "create or replace   view DEVELOPMENT.dbt_dev_rdaniel_tests.stg_customers\n  \n   as (\n    with source as (\n    select * from DEVELOPMENT.dbt_dev_rdaniel_tests.raw_customers\n\n),\n\nrenamed as (\n\n    select\n        id as customer_id,\n        first_name,\n        last_name\n\n    from source\n\n)\n\nselect * from renamed\n  );"}...}

Relevant log output

No response

Environment

- OS: Sonoma 14.1
- Python: 3.8.10
- dbt: 1.7.7 (observed behavior in 1.6.0+, did not test with dbtRunner in 1.5 as IDE uses rpc for <=1.5)

Which database adapter are you using with dbt?

snowflake

Additional Context

Experience team ticket with historical context -- https://dbtlabs.atlassian.net/browse/XP-2671

@racheldaniel racheldaniel added bug Something isn't working triage labels Feb 9, 2024
@dbeatty10 dbeatty10 added the python_api Issues related to dbtRunner Python entry point label Feb 9, 2024
@dbeatty10
Copy link
Contributor

Thanks for such a nice write-up @racheldaniel -- it allowed me to reproduce the same thing that you are seeing.

@dbeatty10 dbeatty10 removed the triage label Feb 10, 2024
@graciegoheen graciegoheen added Impact: Exp backport 1.7.latest High Severity bug with significant impact that should be resolved in a reasonable timeframe labels Feb 12, 2024
@gshank gshank self-assigned this Feb 12, 2024
@davidharting
Copy link

Amazing, thank you @gshank!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 1.7.latest bug Something isn't working High Severity bug with significant impact that should be resolved in a reasonable timeframe Impact: Exp python_api Issues related to dbtRunner Python entry point
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants