[CI] Reduce the amount taking to run tests in the CI from 5h to 11min

[tatiana]

Context

Closes: #1299

Some CI jobs took an outrageous amount of time to run.

One of our CI test pipeline jobs was taking over five hours to run:

https://github.com/astronomer/astronomer-cosmos/actions/runs/11505558596

More recently, this same job started taking over 6 hours to run and started timing out in the CI, making Cosmos' main branch red for both unit and integration tests for Airflow 2.6. The underlying reason is that it took a long time to resolve dependencies. This seems to have happened since October 29, as seen on the commit 84c5fbd to main.

Example: https://github.com/astronomer/astronomer-cosmos/actions/runs/11594745046/job/32330864858

About this change

This PR solves the original issue by changing where and how we install Airflow dependencies, simplifying the previous setup. We manage Airflow test dependencies in the pre-install-airflow.sh file, not in pyproject.toml, other sh, or the Github action. We are also being strict as we can dependent on the Airflow version. Where possible, we use constraints. Where different providers' dependencies conflict with previous versions of Airflow, we just ensure the expected version remains being used after the installation.

Example of a successful run:

https://github.com/astronomer/astronomer-cosmos/actions/runs/11685652312

Bonus

I realised with this change that users who use K8s and want to define different paths for their dbt projects in Airflow and K8s were facing an issue. This problem was evident when running the k8s example DAG. I've fixed the problem as part of this PR.

Follow-up actions

Since this has been taking a long time to solve and our main branch is still red, I commented out two tasks that were failing tests - and I've logged a follow-up issue for us to address this:

#1304

[netlify]

✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.

Name	Link
🔨 Latest commit	d471ec2
🔍 Latest deploy log	https://app.netlify.com/sites/sunny-pastelito-5ecb04/deploys/6728e685f83aae0008a14084

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 95.85%. Comparing base (342ce3a) to head (45aceec).
Report is 10 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1297      +/-   ##
==========================================
+ Coverage   95.73%   95.85%   +0.11%     
==========================================
  Files          67       67              
  Lines        3967     3976       +9     
==========================================
+ Hits         3798     3811      +13     
+ Misses        169      165       -4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.