Fix Variable not found issue in Airflow 3

[tatiana]

In Airflow 2.x, the following DAG used to work (when specifying other mandatory fields):

import json

from airflow import DAG
from airflow.operators.empty import EmptyOperator
from airflow.models import Variable

try:
    variable_value = Variable.get("some-variable", deserialize_json=True)
except (json.decoder.JSONDecodeError, KeyError):
    variable_value = {"task_id": "default_task_name"}

with DAG(dag_id="example_dag_variable_access") as dag:
    task_from_var = EmptyOperator(task_id=variable_value.get("task_id"))
    task_from_var

In Airflow 3 (commit dda86263e8a9c171d91729f14668028408568022), if we try to parse it using:

airflow dag-processor

It fails and logs:

================================================================================
DAG File Processing Stats

Bundle       File Path                       PID    Current Duration      # DAGs    # Errors  Last Duration    Last Run At
-----------  ------------------------------  -----  ------------------  --------  ----------  ---------------  -------------------
dags-folder  example_dag_variable_access.py                                    0           1  0.62s            2025-04-22T11:02:54
================================================================================
[2025-04-22T12:03:25.160+0100] {manager.py:523} INFO - Not time to refresh bundle dags-folder
[2025-04-22T12:03:25.338+0100] {_client.py:1025} INFO - HTTP Request: GET http://in-process.invalid./variables/some-variable "HTTP/1.1 404 Not Found"
2025-04-22 12:03:25 [warning  ] Server error                   [airflow.sdk.api.client] detail={'detail': {'reason': 'not_found', 'message': "Variable with key 'some-variable' not found"}}
2025-04-22 12:03:25 [error    ] Variable not found             [airflow.sdk.api.client] detail={'detail': {'reason': 'not_found', 'message': "Variable with key 'some-variable' not found"}} key=some-variable status_code=404

We're resolving this problem in Cosmos by capturing AirflowRuntimeError exceptions, when needed:

from airflow.sdk.exceptions import AirflowRuntimeError

apache/airflow#49543

[Copilot]

Pull Request Overview

This PR addresses the issue of Airflow variable not found errors in Airflow 3 by broadening the exception handling to include AirflowRuntimeError where appropriate.

• Updated exception handling to include AirflowRuntimeError via dynamic import and list extension.
• Modified the exception tuple in the try/except block to use the variable containing the list of exceptions.

Comments suppressed due to low confidence (1)

cosmos/dbt/graph.py:511

• [nitpick] Consider adding an inline comment explaining the rationale for dynamically constructing the exception tuple for clarity to future maintainers.

except tuple(airflow_variable_exceptions):

[netlify]

✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.

Name	Link
🔨 Latest commit	f84a800
🔍 Latest deploy log	https://app.netlify.com/sites/sunny-pastelito-5ecb04/deploys/68078f9c74b4ed000824d826

[cloudflare-workers-and-pages]

Deploying astronomer-cosmos with    Cloudflare Pages

Latest commit:	f84a800
Status:	✅  Deploy successful!
Preview URL:	https://63c58cb6.astronomer-cosmos.pages.dev
Branch Preview URL:	https://fix-variable-not-found-af3.astronomer-cosmos.pages.dev

View logs

[codecov]

Codecov Report

Attention: Patch coverage is 85.71429% with 1 line in your changes missing coverage. Please review.

Project coverage is 97.43%. Comparing base (156da56) to head (f84a800).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
cosmos/dbt/graph.py	85.71%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1684      +/-   ##
==========================================
- Coverage   97.44%   97.43%   -0.02%     
==========================================
  Files          80       80              
  Lines        5018     5024       +6     
==========================================
+ Hits         4890     4895       +5     
- Misses        128      129       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[tatiana]

The coverage issues will be solved once we merge #1646