Add dbt Core 1.11 to test matrix#2230
Conversation
✅ Deploy Preview for astronomer-cosmos canceled.
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2230 +/- ##
=======================================
Coverage 97.98% 97.98%
=======================================
Files 95 95
Lines 6190 6190
=======================================
Hits 6065 6065
Misses 125 125 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Pull request overview
This PR adds dbt-core 1.11 to the testing matrix to validate compatibility with the latest dbt version. The changes update both the project configuration and CI workflow to include dbt 1.11 in automated testing.
Key Changes:
- Added dbt 1.11 to the test matrix in project configuration
- Updated CI workflow to test against dbt 1.11 across different Python and Airflow versions
- Temporarily enabled CI runs on the support branch for validation
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| pyproject.toml | Added "1.11" to the dbt version matrix for comprehensive testing across all supported versions |
| .github/workflows/test.yml | Updated dbt-version parameters to "1.11" in multiple test jobs and added temporary branch trigger for CI validation |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
pankajkoti
left a comment
There was a problem hiding this comment.
Looks good. Inline suggestions to also preserve 1.10, along with 1.11, to run the Unit Integration and Kubernetes tests since it's still a supported version. WDYT?
Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>
Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>
Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>
|
Thanks for the quick review, @pankajkoti !
For unit tests, it does not make sense because they should not rely on For integration tests, it would be too expensive to run them against all supported versions of Python and Airflow (currently, the matrix spans 32 jobs). By having two dbt versions, we'd have 64 - if we were to add all dbt versions that didn't reach EOL. It would be 128). In most cases, Cosmos does not rely on specific features of dbt versions - except for two scenarios:
I'm happy for us to have a more extensive dbt version testing in the CI. That said, I don't believe the solution is to multiply the existing Py & Airflow matrix by also having multiple dbt versions. WDYT of logging a dedicated ticket for us to review and develop a better strategy? We could run a subset of integration tests with a varying matrix of AF & dbt, but pin the Python version. It feels this is outside of the scope of the current ticket. |
|
Sounds all good to me @tatiana 👍🏽 |
|
Logged: #2234 |
Features * Support cross-referencing models across dbt projects using dbt-loom by @pankajkoti in #2271 * Support use of YAML selectors when using ``LoadMode.DBT_MANIFEST`` by @YourRoyalLinus in #2261 * Introduce ``ExecutionMode.WATCHER_KUBERNETES`` to use the watcher with ``KubernetesPodOperator`` by @tatiana in #2207 * Add support for StarRocks profile mapping by @kurkim0661 in #2256 * Allow pushing URIs as XComs for Cosmos tasks by @corsettigyg in #2275 * Support defining custom callbacks alongside the ``WATCHER_KUBERNETES`` callback by @johnhoran in #2307 Enhancements * Refactor: remove duplicate ``_construct_dest_file_path`` by @jx2lee in #2077 * Leverage Airflow ``::group::`` to group logs associated with DAG parsing by @tatiana in #2235 * Refactor ``DbtConsumerWatcherSensor`` for reusability by @tatiana in #2245 * Restore plain text output when using ``ExecutionMode.WATCHER`` by @tiovader in #2241 Bug Fixes * Fix running empty models or ephemeral nodes in ``ExecutionMode.WATCHER`` by @tatiana in #2279 * Improve watcher producer task priority in scheduling and the UI by @tatiana in #2237 * Fix typos and formatting issues in documentation by @pankajkoti in #2259 * Allow watcher producer retries without erroring by @tatiana in #2283 * Fix ``TestBehavior.AFTER_ALL`` is missing project_name information when loading project using manifest file by @tuantran0910 in #2242 * Fix duplicate log lines in watcher subprocess execution and format timestamps by @pankajkoti in #2301 Docs * Add Watcher Kubernetes documentation by @tatiana in #2303 * Document newly added telemetry metrics in the privacy notice by @pankajkoti in #2249 * Add compatibility policy document by @pankajastro in #2251 * Improve watcher documentation related to dbt threads by @tatiana in #2273 * Fix link in watcher execution mode documentation by @jedcunningham in #2277 * Update Apache Airflow minimum compatibility policy by @tatiana in #2285 * Clarify Cosmos runtime support until "End of Basic Support" by @jedcunningham in #2286 * Update watcher docs by @tatiana in #2298 * Update watcher kubernetes documentation by @tatiana in #2306 Others * Add Airflow 3 DAG versioning tests for Cosmos by @michal-mrazek in #2177 * Add dbt Core 1.11 to the test matrix by @tatiana in #2230 * Add integration tests using InvocationMode.SUBPROCESS and validate output by @tatiana in #2287 * Fix main branch failing tests by @tatiana in #2296 * Update pre-commit hooks to the latest versions by @jedcunningham in #2289 * Pre-commit autoupdates by @pre-commit in #2222, #2264, #2274 and #2290 * Dependabot updates by @dependabot in #2218, #2219, #2220, #2280 and #2284 * Add Scarf metrics to understand Cosmos feature usage patterns - Add telemetry tracking for dbt docs plugin usage by @pankajkoti in #2240 - Add DAG run telemetry metrics for load mode, invocation, and render_config parameters by @pankajkoti in #2223 - Collect profile metrics for DAG runs by @pankajastro in #2228 - Compress telemetry metadata to reduce serialized DAG size by @pankajkoti in #2252 - Skip storing telemetry metadata when emission is disabled by @pankajkoti in #2278 - Hide telemetry metadata parameters from the Airflow trigger UI by @pankajkoti in #2247 closes: astronomer/oss-integrations-private#317 --------- Co-authored-by: Tatiana Al-Chueyr <tatiana.alchueyr@gmail.com>
There have been a few releases of dbt Core 1.11 since December 19th:

This PR aims to validate the latest dbt version against Cosmos tests to identify potential issues.
We temporarily enabled running tests against this branch to confirm that GH Actions Jobs were working. Example of CI job run:
https://github.com/astronomer/astronomer-cosmos/actions/runs/20570466441