Fix cache behaviour when DAG name contains "."#1908
Merged
Merged
Conversation
Cosmos stores the result of dbt ls as a Variable based on the DAG name, but if the name has a . in it, that Variable name will be invalid when working with certain secret managers, including Google cloud's. This is an example of error that is raised: ``` Invalid secret ID myhost-cosmos_cache__dag.moredag.yetmoredag Only ASCII alphabets (a-Z), numbers (0-9), dashes (-), and underscores (_) are allowed in the secret ID. ``` In this PR, we replace those names to use underscores instead of dot. Closes: #1875
✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.
|
tatiana
commented
Aug 6, 2025
Contributor
There was a problem hiding this comment.
Pull Request Overview
This PR fixes cache behavior when DAG names contain periods by replacing them with underscores to ensure compatibility with secret managers like Google Cloud that have strict naming requirements for secret IDs.
- Replaces periods in DAG names with triple underscores in cache identifiers
- Adds test coverage for DAG names containing periods
- Updates test hash expectations to reflect the new naming convention
Reviewed Changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| cosmos/cache.py | Modifies cache identifier generation to replace periods in DAG names with triple underscores |
| tests/test_cache.py | Adds test case for DAG names with periods and expected cache identifier format |
| tests/dbt/test_graph.py | Updates hash expectation to account for the new DAG name sanitization |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1908 +/- ##
=======================================
Coverage 98.06% 98.06%
=======================================
Files 86 86
Lines 5334 5334
=======================================
Hits 5231 5231
Misses 103 103 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Merged
tatiana
added a commit
that referenced
this pull request
Aug 8, 2025
Cosmos stores the result of dbt ls as a Variable based on the DAG name, but if the name has a . in it, that Variable name will be invalid when working with certain secret managers, including Google Cloud's. This is an example of an error that is raised: ``` Invalid secret ID myhost-cosmos_cache__dag.moredag.yetmoredag Only ASCII alphabets (a-Z), numbers (0-9), dashes (-), and underscores (_) are allowed in the secret ID. ``` In this PR, we replace those names to use underscores instead of ".". Closes: #1875 (cherry picked from commit 147fd10)
tatiana
added a commit
that referenced
this pull request
Aug 8, 2025
**Bug Fixes** * Fix task instance ``try_number`` attribute for Airflow 3 compatibility by @pankajkoti in #1781 * Fix rendered template override logic when ``should_store_compiled_sql=False`` to restore pre-refactor behaviour by @pankajkoti in #1777 * Fix ``ProfileConfig`` in GCP Cloud Run job execution mode by @ramonvermeulen in #1783 * Fix dbt Docs page height by @1cadumagalhaes in #1793 * Add support to base64 encoded pem in Snowflake profiles by @brunocmartins in #1801 * Allow to disable owner inheritance from dbt into airflow DAG owners by @CorsettiS in #1787 * Fix Kubernetes Pod Operator conversion of ``container_resources`` to ``resources`` by @johnhoran in #1821 * Fix ``dbt deps`` with project level variables by @AlexandrKhabarov in #1822 * Fix source freshness warnings in kubernetes execution mode by @Pawel-Drabczyk in #1859 * Fix: Harden DbtNode against null config/meta by @pankajkoti in #1877 * Fix cache behaviour when DAG name contains "." by @tatiana in #1908 **Documentation** * Fix ``contributing.rst`` docs by @tatiana in #1785 * Fix docs rendering in Airflow 3 Compatibility by @pankajastro in #1790 * Fix typo in ``selecting-excluding.rst`` by @msshroff in #1814 * Update testing behavior file with ``ExecutionMode.KUBERNETES`` by @LuigiCerone in #1813 * Add step to fork repo in contributing guide by @pankajastro in #1808 * Fix ``depends_on`` attribute by @benedikt-buchert in #1837 * Fix character name by @ThePsyjo in #1860 * Update suggested MWAA startup script by @jaklan in #1884 * Make implementation & docs consistent regarding ``use_dataset_airflow3_uri_standard`` by @Anti0ff in #1878 **Others** * Set retries to 0 in example DAGs by @pankajkoti in #1782 * Fix ``test_async_example_dag_without_setup_task`` tests by @pankajastro in #1788 * Fix test hash value for Darwin when using Py 3.12.10 by @tatiana in #1786 * Upgrade Python and Airflow used to run MyPy checks by @tatiana in #1796 * Assert example DAGs' ``DagRunState`` and fix issues by @pankajkoti and @tatiana in #1778 * Update the conflict matrix to include AF 2.10, 2.11 & 3.0 and dbt 1.9 & 1.10 by @tatiana in #1820 * Fix broken CI due to Pydantic conflicts by @tatiana in #1809 * Drop Python 3.8 Support by @pankajastro in #1852 * Add Airflow 2.11 to the test matrix by @tatiana in #1807 * Require Authorize for all jobs on pull requests from external contributors in CI by @pankajkoti in #1861 * Leverage Trusted Publisher Management when publishing PyPI package by @tatiana in #1862 * CI: Add back accidentally deleted python-version matrix for running unit tests by @pankajkoti in #1872 * Remove commented code and fix mypy failures by @pankajkoti in #1876 * Add Zizmor analysis GitHub action by @pankajkoti in #1870 * Catch FlushError on Datasets for Airflow 2.11 dags test by @pankajkoti in #1880 * Add deprecation warning for ``LoadMode.CUSTOM`` parser by @duongphannamhung in #1885 * CI: Add GitHub CodeQL analysis workflow (codeql.yml) by @pankajkoti in #1871 * Resolve 'credential persistence through GitHub Actions artifacts' warnings from Zizmor by @pankajkoti in #1890 * Resolve 'overly broad permissions' warnings from Zizmor by @pankajkoti in #1889 * Resolve Zizmor error alerts for unpinned image references; mark alert for pull_request_target ignored by @pankajkoti in #1888 * Fix broken CI ``tests.py3.11-2.8-1.9:test-integration-setup`` by @tatiana in #1902 * Add dbt-core 1.10 to test matrix by @tatiana in #1767 * Pin package dbt-databricks by @pankajastro in #1909 * Enable matrix test entry for dbt-1.9, python-3.9 and airflow-3.0 tests in CI by @pankajastro in #1900 * Pre-commit updates: #1779, #1795, #1800, #1857, #1863, #1869, #1892, #1901 * Dependabot updates: #1904 Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>
Merged
tatiana
added a commit
that referenced
this pull request
Aug 8, 2025
**Bug Fixes** * Fix task instance ``try_number`` attribute for Airflow 3 compatibility by @pankajkoti in #1781 * Fix rendered template override logic when ``should_store_compiled_sql=False`` to restore pre-refactor behaviour by @pankajkoti in #1777 * Fix ``ProfileConfig`` in GCP Cloud Run job execution mode by @ramonvermeulen in #1783 * Fix dbt Docs page height by @1cadumagalhaes in #1793 * Add support to base64 encoded pem in Snowflake profiles by @brunocmartins in #1801 * Allow to disable owner inheritance from dbt into airflow DAG owners by @CorsettiS in #1787 * Fix Kubernetes Pod Operator conversion of ``container_resources`` to ``resources`` by @johnhoran in #1821 * Fix ``dbt deps`` with project level variables by @AlexandrKhabarov in #1822 * Fix source freshness warnings in kubernetes execution mode by @Pawel-Drabczyk in #1859 * Fix: Harden DbtNode against null config/meta by @pankajkoti in #1877 * Fix cache behaviour when DAG name contains "." by @tatiana in #1908 **Documentation** * Fix ``contributing.rst`` docs by @tatiana in #1785 * Fix docs rendering in Airflow 3 Compatibility by @pankajastro in #1790 * Fix typo in ``selecting-excluding.rst`` by @msshroff in #1814 * Update testing behavior file with ``ExecutionMode.KUBERNETES`` by @LuigiCerone in #1813 * Add step to fork repo in contributing guide by @pankajastro in #1808 * Fix ``depends_on`` attribute by @benedikt-buchert in #1837 * Fix character name by @ThePsyjo in #1860 * Update suggested MWAA startup script by @jaklan in #1884 * Make implementation & docs consistent regarding ``use_dataset_airflow3_uri_standard`` by @Anti0ff in #1878 **Others** * Set retries to 0 in example DAGs by @pankajkoti in #1782 * Fix ``test_async_example_dag_without_setup_task`` tests by @pankajastro in #1788 * Fix test hash value for Darwin when using Py 3.12.10 by @tatiana in #1786 * Upgrade Python and Airflow used to run MyPy checks by @tatiana in #1796 * Assert example DAGs' ``DagRunState`` and fix issues by @pankajkoti and @tatiana in #1778 * Update the conflict matrix to include AF 2.10, 2.11 & 3.0 and dbt 1.9 & 1.10 by @tatiana in #1820 * Fix broken CI due to Pydantic conflicts by @tatiana in #1809 * Drop Python 3.8 Support by @pankajastro in #1852 * Add Airflow 2.11 to the test matrix by @tatiana in #1807 * Require Authorize for all jobs on pull requests from external contributors in CI by @pankajkoti in #1861 * Leverage Trusted Publisher Management when publishing PyPI package by @tatiana in #1862 * CI: Add back accidentally deleted python-version matrix for running unit tests by @pankajkoti in #1872 * Remove commented code and fix mypy failures by @pankajkoti in #1876 * Add Zizmor analysis GitHub action by @pankajkoti in #1870 * Catch FlushError on Datasets for Airflow 2.11 dags test by @pankajkoti in #1880 * Add deprecation warning for ``LoadMode.CUSTOM`` parser by @duongphannamhung in #1885 * CI: Add GitHub CodeQL analysis workflow (codeql.yml) by @pankajkoti in #1871 * Resolve 'credential persistence through GitHub Actions artifacts' warnings from Zizmor by @pankajkoti in #1890 * Resolve 'overly broad permissions' warnings from Zizmor by @pankajkoti in #1889 * Resolve Zizmor error alerts for unpinned image references; mark alert for pull_request_target ignored by @pankajkoti in #1888 * Fix broken CI ``tests.py3.11-2.8-1.9:test-integration-setup`` by @tatiana in #1902 * Add dbt-core 1.10 to test matrix by @tatiana in #1767 * Pin package dbt-databricks by @pankajastro in #1909 * Enable matrix test entry for dbt-1.9, python-3.9 and airflow-3.0 tests in CI by @pankajastro in #1900 * Pre-commit updates: #1779, #1795, #1800, #1857, #1863, #1869, #1892, #1901 * Dependabot updates: #1904 Co-authored-by: Tatiana Al-Chueyr <tatiana.alchueyr@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cosmos stores the result of dbt ls as a Variable based on the DAG name, but if the name has a . in it, that Variable name will be invalid when working with certain secret managers, including Google Cloud's.
This is an example of an error that is raised:
In this PR, we replace those names to use underscores instead of ".".
Closes: #1875