Fix 'Unable to find the dbt executable: dbt' error

[tatiana]

Since Cosmos 1.2.2 users who used ExecutionMode.DBT_LS (directly or via ExecutionMode.AUTOMATIC) and set ExecutionConfig.dbt_executable_path (most, if not all, Astro CLI users), like:

execution_config = ExecutionConfig(
    dbt_executable_path = f"{os.environ['AIRFLOW_HOME']}/dbt_venv/bin/dbt",
)

Started facing the issue:

Broken DAG: [/usr/local/airflow/dags/my_example.py] Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/cosmos/dbt/graph.py", line 178, in load
    self.load_via_dbt_ls()
  File "/usr/local/lib/python3.11/site-packages/cosmos/dbt/graph.py", line 233, in load_via_dbt_ls
    raise CosmosLoadDbtException(f"Unable to find the dbt executable: {self.dbt_cmd}")
cosmos.dbt.graph.CosmosLoadDbtException: Unable to find the dbt executable: dbt

This issue was initially reported in the Airflow #airflow-astronomer Slack channel: https://apache-airflow.slack.com/archives/C03T0AVNA6A/p1699584315506629

The workaround to avoid this error in Cosmos 1.2.2 and 1.2.3 is to set the dbt_executable_path in the RenderConfig:

render_config=RenderConfig(dbt_executable_path = f"{os.environ['AIRFLOW_HOME']}/dbt_venv/bin/dbt",),

This PR solves the bug from Cosmos 1.2.4 onwards.

Closes: #633

[netlify]

👷 Deploy Preview for amazing-pothos-a3bca0 processing.

Name	Link
🔨 Latest commit	f6a9ff5
🔍 Latest deploy log	https://app.netlify.com/sites/amazing-pothos-a3bca0/deploys/654ec76bd39d0f0008bf5701

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (6e41471) 92.72% compared to head (f6a9ff5) 92.75%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #666      +/-   ##
==========================================
+ Coverage   92.72%   92.75%   +0.02%     
==========================================
  Files          54       54              
  Lines        2227     2235       +8     
==========================================
+ Hits         2065     2073       +8     
  Misses        162      162              

Files	Coverage Δ
cosmos/config.py	97.34% <100.00%> (+0.25%)	⬆️
cosmos/converter.py	97.01% <ø> (ø)
cosmos/dbt/graph.py	98.79% <100.00%> (-0.02%)	⬇️

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[MrBones757]

@tatiana as requested, some feedback on this item and more general commentary on dbt exe path discovery.

This looks like it will solve the issue of dbt command discovery at render time and make it more robust, and i had been looking to apply similar fixes in some of my previous PRs to make passing the dbt command path around more simple, but didn't due to scope.

For the render time issue this looks good.

I personally had an issue running on k8s at execution time (NOT Render Time) where the dbt command could not be found, and it boiled down to a documentation issue / lack of clarity for requirements on our docker image requirements for runtime.

In the cosmos docs, the example we provide uses a python image as the base and we install dbt packages into that, but have no documentation on how to use the official dbt docker images.
our python dockerfile: https://github.com/astronomer/cosmos-example/blob/main/Dockerfile.postgres_profile_docker_k8s

Looking at how these base python images are built (source: official library/python dockerhub images):
https://github.com/docker-library/python/blob/b34fc81375758e84dec797e9210a079b4889352e/3.10/bookworm/Dockerfile#L128

I noticed that they have a CMD set to python, and no ENTRYPOINT defined (thus using the default of /bin/sh -c).
Comparing to the official DBT images:
https://github.com/dbt-labs/dbt-core/blob/main/docker/Dockerfile#L55

We notice here that the ENTRYPOINT argument is set to "dbt".

Since our k8s manifests we generate pass the dbt command in as "args" to the pod spec, the end result we get when using the pre-built dbt docker images is a command that looks like "dbt dbt blah" which, is not a valid command. This occurs because the entrypoint args are prefixed to whatever we pass in.

For me to get the pre-baked dbt images to work, i had to add a line in my dockerfile "ENTRYPOINT []" to revert the official DBT images's commands back to the default entrypoint of ["/bin/sh", "-c"].

This, then allows the commands to run as expected at runtime.

While unrelated to this render time issue, since we're updating how we deal with the dbt pathing, it is probably a worthwhile docs update to make.

It would have to be something as simple as "if you want to build your own image and install dbt, make the dockerfile look like the example, else if you want to use the pre baked ones, you need to ensure the entrypoint removed (i.e []) to ensure the commands work in the current state.

The other option we have is to update our k8s / docker logic to add an additional arg when we generate pod specs and override the entrypoint / command to ensure we dont use the default:

https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.26/

in this case - args appends to the ENTRYPOINT, command would run it without depending on the entrypoint, in theory, allowing us to consume the default dbt images without having to update the ENTRYPOINT in custom builds.

[tatiana]

Thanks, @MrBones757 , for the feedback!

I've logged a task for us to improve the K8s docs: #668, to avoid other users running into the issues you faced. Thank you!