Run dbt deps before running the dbt ls command

[DanMawdsleyBA]

Description

Runs the dbt deps command before running the dbt ls command

Related Issue(s)

Closes #445

Breaking Change?

Checklist

• I have made corresponding changes to the documentation (if required)
• I have added tests that prove my fix is effective or that my feature works

[netlify]

👷 Deploy Preview for amazing-pothos-a3bca0 processing.

Name	Link
🔨 Latest commit	3103c30
🔍 Latest deploy log	https://app.netlify.com/sites/amazing-pothos-a3bca0/deploys/64f07a9f6fe79a00081d0261

[tatiana]

This looks great. Thanks a lot for your first contribution, @DanMawdsleyBA ! I'm hopeful this could be part of the upcoming 1.1 release.

I can see two areas for improvement:

1. A test that represents the original issue this PR solves, that breaks before the change and passes after this modification. Ideally, it would highlight what happens when a profile depends on a specific dbt adapter that is not installed and which error would be raised with dbt ls.
2. Do you think all users would be happy for, during DAG parsing time, to have dbt deps installed? What would happen if they had a read-only filesystem? A possibility to address this could be to have a new parameter added to RenderConfig, something like install_deps, set to True by default, but users could opt out (@jlaneve may have other ideas/suggestions)

Please do reach out if you'd like support writing the test.

[DanMawdsleyBA]

This looks great. Thanks a lot for your first contribution, @DanMawdsleyBA ! I'm hopeful this could be part of the upcoming 1.1 release.

I can see two areas for improvement:

1. A test that represents the original issue this PR solves, that breaks before the change and passes after this modification. Ideally, it would highlight what happens when a profile depends on a specific dbt adapter that is not installed and which error would be raised with dbt ls.
2. Do you think all users would be happy for, during DAG parsing time, to have dbt deps installed? What would happen if they had a read-only filesystem? A possibility to address this could be to have a new parameter added to RenderConfig, something like install_deps, set to True by default, but users could opt out (@jlaneve may have other ideas/suggestions)

Please do reach out if you'd like support writing the test.

@tatiana
I've added a dbt_deps into RenderConfig with a default value as True but can be set to False to not run dbt deps this only applies when loading via dbt ls . I've added a test to catch the failure by having a package file with dbt_utils and manually setting the dbt deps to False. I also improved the errors to catch this specific case. Let me know if you have any more thoughts. If you're happy if you could add it to a pre-release version I can test it on an actual airflow instance

[DanMawdsleyBA]

One thing I have realised is that this fix won't work on systems that have the read only issue as by default the packages are stored in dbt project path/ packages . There is a variable that can be set in the dbt_project.yml packages_install_path which may help however I don't think there is a way to specify that via a flag or env_var (Although happy to be wrong on that). It might be possible to do some kind of project.yml update to that variable and override the current but feels a janky solution. Not sure if @harels or @tatiana do you have any better ideas?
I'm starting to feel it might be easier to just copy the whole dbt project into a new directory for read only systems and have everything point to that new directory.

[tatiana]

One thing I have realised is that this fix won't work on systems that have the read only issue as by default the packages are stored in dbt project path/ packages . There is a variable that can be set in the dbt_project.yml packages_install_path which may help however I don't think there is a way to specify that via a flag or env_var (Although happy to be wrong on that). It might be possible to do some kind of project.yml update to that variable and override the current but feels a janky solution. Not sure if @harels or @tatiana do you have any better ideas? I'm starting to feel it might be easier to just copy the whole dbt project into a new directory for read only systems and have everything point to that new directory.

Hi @DanMawdsleyBA , that's a very important point.
I refactored a bit the logic in the dbt ls and hopefully it will be able to address the issue of running dbt deps on project directories that were originally read only: #488

[codecov]

Codecov Report

Patch coverage: 92.85% and project coverage change: -0.08% ⚠️

Comparison is base (8c9e768) 91.59% compared to head (3103c30) 91.51%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #481      +/-   ##
==========================================
- Coverage   91.59%   91.51%   -0.08%     
==========================================
  Files          50       50              
  Lines        1856     1874      +18     
==========================================
+ Hits         1700     1715      +15     
- Misses        156      159       +3     

Files Changed	Coverage Δ
cosmos/dbt/graph.py	98.67% <92.30%> (-1.33%)	⬇️
cosmos/config.py	92.85% <100.00%> (+0.08%)	⬆️
cosmos/converter.py	95.77% <100.00%> (-1.37%)	⬇️

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[tatiana]

@DanMawdsleyBA We managed to get all the tests to pass 🎉 Let's see after the rebase. Once this is passing, we'll merge this. It will be released as part of 1.1.

[tatiana]

Happy for these changes to be merged, I could test it locally.

It will help many Cosmos users with DAGs with no nodes due to dependencies not being installed. We'll improve the coverage in a follow up PR.