Add dbt project's hash to dag docs to support dag versioning in Airflow 3

[pankajkoti]

This PR leverages the existing method for generating a hash of the dbt project contents and appends that hash to the DAG documentation. This enables Airflow 3’s automatic DAG versioning feature to track changes in dbt project content.

Airflow 3 introduces automatic DAG versioning by detecting changes in a DAG’s structure or metadata. By embedding the dbt content hash into the doc_md field (a part of the serialized DAG), this change allows Airflow to:
• Detect when dbt project files are added, modified, or deleted
• Automatically generate a new DAG version in response to those changes

This ensures accurate and automatic versioning of DAGs tied to evolving dbt projects, without requiring manual intervention.

closes: #1868

[netlify]

✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.

Name	Link
🔨 Latest commit	c412d11
🔍 Latest deploy log	https://app.netlify.com/projects/sunny-pastelito-5ecb04/deploys/689348351476d20008e4e85e

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.06%. Comparing base (19861f6) to head (c412d11).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1907   +/-   ##
=======================================
  Coverage   98.06%   98.06%           
=======================================
  Files          85       86    +1     
  Lines        5313     5334   +21     
=======================================
+ Hits         5210     5231   +21     
  Misses        103      103           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[tatiana]

Looks great, @pankajkoti , thank you! Very excited to see Cosmos DAGs being versioned in AF3!

[Copilot]

Pull Request Overview

This PR adds support for Airflow 3's automatic DAG versioning by incorporating a hash of the dbt project contents into the DAG documentation. This allows Airflow to detect changes in dbt project files and automatically generate new DAG versions without manual intervention.

Key changes:

• Extracted the _create_folder_version_hash function from cache.py to a new versioning.py module for reusability
• Modified DbtToAirflowConverter to append the dbt project hash to DAG documentation
• Added comprehensive test coverage for hash appending scenarios and error handling

Reviewed Changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
cosmos/versioning.py	New module containing the extracted folder hash generation function
cosmos/converter.py	Added method to append dbt project hash to DAG documentation for versioning support
cosmos/cache.py	Removed _create_folder_version_hash function and imported it from versioning module
tests/test_versioning.py	New test file for the versioning module functionality
tests/test_converter.py	Added comprehensive tests for DAG documentation hash appending scenarios
tests/test_cache.py	Removed test that was moved to the versioning test file

[tatiana]

BTW: as we discussed, please, let's check on a long-term solution with @jedcunningham and other Airflow commiters - to see if we can have a more robust solution that does not rely on the Airflow DAG docs.

Last time we discussed this, #1488, there wasn't a better way - but there may be one moving forward 🤞

[pmuntianu]

Hi Team, this change is impacting parsing performance of cosmos because our IO is slow in k8s. And there is no way to turn this feature off while we are using airflow 2+. Do you have any suggestions?

[pankajkoti]

impacting parsing performance

hi @pmuntianu, I just logged an issue #2386 to allow disabling this work done here in the PR. We will try to prioritise this, but would like to check with you that, before we get a chance to take a stab at it, is there a chance you could contribute a PR for this?

[pmuntianu]

Hi @pankajkoti , thanks a ton. This is really pormpt response. I appreciate your dedication. For us, the solution is to not pass dag explicitly to the DbtTaskGroup and instead use with comprehension. Something like this:

        with dag:
            DbtTaskGroup(
                # by commenting line below we prevent airflow dag caching in _add_dbt_project_hash_to_dag_docs method
                # this is important because our io is slow in k8s. We might negatively impact caching as well.
                # dag=dag,
                group_id=f"{self.workflow_settings.name}_group",
                project_config=self._parse_project(),
...

Thanks a lot one more time for taking this work