Dag test backcompat#2036
Closed
kaxil wants to merge 64 commits into
Closed
Conversation
apache-airflow==3.1.0b2 depends on Python>=3.10,<3.14
…sts so it is compatible with AF 3.1
… change introduced in AF3.1
dbt has playing backnforth to changing the syntax to pass arguments to tests. Previously, dbt 1.10 and dbt Fusion were using the new syntax. In one of the latest dbt 1.10 releases, it seems arguments syntax stopped working. We're therefore removing tests that were using this
…Run for DAG ... because the dag is not serialized
______________________ test_example_dag[basic_cosmos_dag] ______________________
session = <sqlalchemy.orm.session.Session object at 0x7f510a21d610>
dag_id = 'basic_cosmos_dag'
@pytest.mark.skipif(
AIRFLOW_VERSION in PARTIALLY_SUPPORTED_AIRFLOW_VERSIONS,
reason="Airflow 2.9.0 and 2.9.1 have a breaking change in Dataset URIs, and Cosmos errors if `emit_datasets` is not False",
)
@pytest.mark.integration
@pytest.mark.parametrize("dag_id", get_dag_ids())
def test_example_dag(session, dag_id: str):
if dag_id in KUBERNETES_DAGS:
return
> run_dag(dag_id)
tests/test_example_dags.py:147:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_example_dags.py:135: in run_dag
test_utils.run_dag(dag)
tests/utils.py:31: in run_dag
return test_dag(dag=dag, conn_file_path=conn_file_path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tests/utils.py:61: in test_dag
dr = new_test_dag(dag)
^^^^^^^^^^^^^^^^^
tests/utils.py:46: in new_test_dag
dr = dag.test(logical_date=timezone.utcnow())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
../../../.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.11-3.1-1.10/lib/python3.11/site-packages/airflow/sdk/definitions/dag.py:1202: in test
dr: DagRun = get_or_create_dagrun(
../../../.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.11-3.1-1.10/lib/python3.11/site-packages/airflow/models/dagrun.py:2170: in get_or_create_dagrun
dr = dag.create_dagrun(
../../../.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.11-3.1-1.10/lib/python3.11/site-packages/airflow/utils/session.py:98: in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
../../../.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.11-3.1-1.10/lib/python3.11/site-packages/airflow/serialization/serialized_objects.py:3257: in create_dagrun
orm_dagrun = _create_orm_dagrun(
../../../.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.11-3.1-1.10/lib/python3.11/site-packages/airflow/utils/session.py:98: in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
@provide_session
def _create_orm_dagrun(
*,
dag: SerializedDAG,
run_id: str,
logical_date: datetime.datetime | None,
data_interval: DataInterval | None,
run_after: datetime.datetime,
start_date: datetime.datetime | None,
conf: Any,
state: DagRunState | None,
run_type: DagRunType,
creating_job_id: int | None,
backfill_id: NonNegativeInt | None,
triggered_by: DagRunTriggeredByType,
triggering_user_name: str | None = None,
session: Session = NEW_SESSION,
) -> DagRun:
bundle_version = None
if not dag.disable_bundle_versioning:
bundle_version = session.scalar(
select(DagModel.bundle_version).where(DagModel.dag_id == dag.dag_id),
)
dag_version = DagVersion.get_latest_version(dag.dag_id, session=session)
if not dag_version:
> raise AirflowException(f"Cannot create DagRun for DAG {dag.dag_id} because the dag is not serialized")
E airflow.exceptions.AirflowException: Cannot create DagRun for DAG basic_cosmos_dag because the dag is not serialized
…Run for DAG ... because the dag is not serialized
✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.
|
kaxil
force-pushed
the
dag-test-backcompat
branch
from
2bc4bd3 to
92a2428
Compare
kaxil
force-pushed
the
dag-test-backcompat
branch
from
c920390 to
f5e1ba5
Compare
Airflow 3.1+ requires DagVersion and DagModel records before calling dag.test(). Used `sync_bag_to_db()` to create these records, matching the approach used by Airflow's CLI `dag test` command.
kaxil
force-pushed
the
dag-test-backcompat
branch
from
f5e1ba5 to
dce0edf
Compare
2 tasks
tatiana
pushed a commit
that referenced
this pull request
Oct 17, 2025
Airflow 3.1+ requires DagVersion and DagModel records before calling `dag.test()`. [Example failure:](https://github.com/astronomer/astronomer-cosmos/actions/runs/18523664806/job/52789291341#step:7:6713) ``` FAILED tests/test_example_dags.py::test_example_dag[example_cosmos_cleanup_dag] - airflow.exceptions.AirflowException: Cannot create DagRun for DAG example_cosmos_cleanup_dag because the dag is not serialized FAILED tests/test_example_dags.py::test_example_dag[example_cosmos_dbt_build] - airflow.exceptions.AirflowException: Cannot create DagRun for DAG example_cosmos_dbt_build because the dag is not serialized FAILED tests/test_example_dags.py::test_async_example_dag_without_setup_task - airflow.exceptions.AirflowException: Cannot create DagRun for DAG simple_dag_async because the dag is not serialized ``` Used `sync_bag_to_db()` to create these records, matching the approach used by Airflow's CLI `dag test` command. [Example run](https://github.com/astronomer/astronomer-cosmos/actions/runs/18574691228/job/52957128514?pr=2036#step:7:1623) on a different branch (branched off of @tatiana 's [`af-31`](#1980): [`dag-test-backcompat`](#2036)) which runs `dag.test` with Airflow 3.1 . ``` tests/test_example_dags.py::test_example_dag[cosmos_manifest_example] PASSED [ 61%] tests/test_example_dags.py::test_example_dag[example_taskflow_project_config] PASSED [ 62%] tests/test_example_dags.py::test_example_dag[example_multiple_parents_test] PASSED [ 62%] tests/test_example_dags.py::test_example_dag[user_defined_profile] PASSED [ 63%] tests/test_example_dags.py::test_example_dag[customized_cosmos_dag] PASSED [ 64%] tests/test_example_dags.py::test_example_dag[basic_cosmos_dag] PASSED [ 65%] tests/test_example_dags.py::test_example_dag[cosmos_profile_mapping] PASSED [ 66%] tests/test_example_dags.py::test_example_dag[basic_cosmos_task_group] PASSED [ 66%] tests/test_example_dags.py::test_example_dag[example_duckdb_dag] PASSED [ 67%] tests/test_example_dags.py::test_example_dag[example_model_version] PASSED [ 68%] tests/test_example_dags.py::test_example_dag[cosmos_callback_dag] PASSED [ 69%] tests/test_example_dags.py::test_example_dag[example_operators] PASSED [ 70%] ```
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Related Issue(s)
Breaking Change?
Checklist