Release 1.11.1#2090
Merged
Merged
Conversation
#2076) Our CI started failing after fastapi [0.121.0 ](https://pypi.org/project/fastapi/0.121.0/) was released yesterday. It's observed that cadwyn(that airflow uses for its migrations) is [incompatible](zmievsa/cadwyn#305) with the latest the release of fastapi. Hence, putting an upper bound for fastapi version which can be removed once cadwyn becomes compatible. Succesful run: https://github.com/astronomer/astronomer-cosmos/actions/runs/19061976838 closes: #2079 --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
<!--pre-commit.ci start--> updates: - [github.com/astral-sh/ruff-pre-commit: v0.14.2 → v0.14.3](astral-sh/ruff-pre-commit@v0.14.2...v0.14.3) <!--pre-commit.ci end--> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Tatiana Al-Chueyr <tatiana.alchueyr@gmail.com>
…n Airflow >= 3.1.2 (#2073) This PR adds explicit Airflow version gating and resilient connection handling for the Airflow 3 FastAPI plugin, along with tests and documentation updates. We now support the plugin only on Airflow >= 3.1 (enforced via a runtime check), move FastAPI app initialisation into the plugin’s constructor to avoid import-time failures, and conditionally use the connection_env context manager only on Airflow 3.1.0–3.1.1 (direct connection access on 3.1.2+). Tests cover the version gate, plugin initialisation, conditional connection handling, and functional routes (skipped on Airflow < 3.1). The documentation is updated to clearly state the Airflow >= 3.1 requirement and configuration keys for dbt_docs_projects. closes: #2061
Airflow 3.0 and 3.1 are currently not thread-safe when running `xcom_push`, which leads to deadlocks. The problem description and troubleshooting are detailed in ticket #2057. Unfortunately, since this issue is not deterministic, I was unable to create an automated test to reproduce and prevent regressions. That said, given that the producer node hangs indefinitely until the task times out, and given that we were no longer able to reproduce the issue after the current changes, I firmly believe we should proceed with this change. Closes: #2057 Co-authored-by: Ash Berlin-Taylor <ash@astronomer.io>
When attempting to use `ExecutionMode.AIRFLOW_ASYNC` with `TaskGroup` in
Cosmos 1.11.0, it started failing due to our handling of XCom. The main
issue was that we were not setting the relative setup task ID in the
consumer tasks, and they were trying to consume the XCom from a task
that did not exist.
Example of DAG:
```
from airflow.models import DAG
try:
from airflow.providers.standard.operators.empty import EmptyOperator
except ImportError:
from airflow.operators.empty import EmptyOperator
from cosmos import DbtTaskGroup
profile_config = ProfileConfig(
profile_name="default",
target_name="dev",
profile_mapping=GoogleCloudServiceAccountDictProfileMapping(
conn_id="gcp_gs_conn", profile_args={"dataset": "release_17", "project": "astronomer-dag-authoring"}
),
)
with DAG(
dag_id="simple_dag_async_taskgroup",
schedule="@daily",
start_date=datetime(2023, 1, 1),
catchup=False,
):
pre_dbt = EmptyOperator(task_id="pre_dbt")
first_dbt_task_group = DbtTaskGroup(
group_id="first_dbt_task_group",
execution_config=ExecutionConfig(
execution_mode=ExecutionMode.AIRFLOW_ASYNC,
async_py_requirements=[f"dbt-bigquery=={DBT_ADAPTER_VERSION}"],
),
render_config=RenderConfig(select=["*customers*"], exclude=["path:seeds"]),
project_config=ProjectConfig(DBT_PROJECT_PATH),
profile_config=profile_config,
operator_args={
"location": "US",
"install_deps": True,
"full_refresh": True,
},
)
pre_dbt >> first_dbt_task_group
```
<img width="1624" height="1056" alt="Image"
src="https://github.com/user-attachments/assets/6e38464f-8fe6-4f6a-8ebc-762e8f3d65f4"
/>
Error:
```
Traceback (most recent call last):
File "/Users/tatiana.alchueyr/Code/astronomer-cosmos/venv-af31/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 920, in run
result = _execute_task(context=context, ti=ti, log=log)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/tatiana.alchueyr/Code/astronomer-cosmos/venv-af31/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 1307, in _execute_task
result = ctx.run(execute, context=context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/tatiana.alchueyr/Code/astronomer-cosmos/venv-af31/lib/python3.12/site-packages/airflow/sdk/bases/operator.py", line 416, in wrapper
return func(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/tatiana.alchueyr/Code/astronomer-cosmos/cosmos/operators/_asynchronous/bigquery.py", line 187, in execute
sql_query = self.get_sql_from_xcom(context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/tatiana.alchueyr/Code/astronomer-cosmos/cosmos/operators/_asynchronous/bigquery.py", line 141, in get_sql_from_xcom
compressed_b64_sql = base64.b64decode(compressed_b64_sql)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/base64.py", line 83, in b64decode
s = _bytes_from_decode_data(s)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/base64.py", line 45, in _bytes_from_decode_data
raise TypeError("argument should be a bytes-like object or ASCII "
TypeError: argument should be a bytes-like object or ASCII string, not 'NoneType'
```
Example of successful run after applying this PR:
<img width="1624" height="1056" alt="Screenshot 2025-11-07 at 14 17 44"
src="https://github.com/user-attachments/assets/9ae39c4d-0cbb-45c2-a895-4381bbcac815"
/>
Closes: #2054
The PR fixes a minor formatting issue in the documentation and addresses the following warning: ``` ./astronomer-cosmos/docs/getting_started/watcher-execution-mode.rst:272: WARNING: Inline literal start-string without end-string. [docutils] ``` Before changes <img width="1135" height="357" alt="image" src="https://github.com/user-attachments/assets/717fe3cf-62f1-4051-96aa-88e3d129a0f4" /> After changes <img width="1141" height="373" alt="image" src="https://github.com/user-attachments/assets/2110bb5f-2ba2-4b54-b285-04dea40030fa" /> (cherry picked from commit bfc2463)
…mode (#2083) This PR adds documentation explaining how to configure and optimize thread counts when using the `ExecutionMode.WATCHER` mode in Cosmos. It provides practical guidance on thread configuration and its performance impact. - Adds a note with best practices for configuring the `threads` parameter - Includes benchmark results showing performance improvements from 1 to 16 threads - Provides two configuration examples: via `profiles.yml` and via Cosmos `ProfileConfig` closes: #2081 (cherry picked from commit dd3b1b0)
) The PR adds the following changes to address #2102: - wrap the dbtRunner callback in DbtProducerWatcherOperator with a try/except so any failure during event handling is logged instead of silently hanging the airflow task - ensure `_handle_startup_event`/`_handle_node_finished` exceptions surface via `logger.exception` while letting execution continue - add a regression test covering the callback error path (`test_execute_callback_exception_is_logged`) closes: #2102 --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> (cherry picked from commit d002c25)
<!--pre-commit.ci start--> updates: - [github.com/asottile/pyupgrade: v3.21.0 → v3.21.1](asottile/pyupgrade@v3.21.0...v3.21.1) - [github.com/astral-sh/ruff-pre-commit: v0.14.3 → v0.14.4](astral-sh/ruff-pre-commit@v0.14.3...v0.14.4) - [github.com/psf/black-pre-commit-mirror: 25.9.0 → 25.11.0](psf/black-pre-commit-mirror@25.9.0...25.11.0) <!--pre-commit.ci end--> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> (cherry picked from commit 11d9661)
…nfig` (#2112) With the new ExecutionMode.WATCHER, the decision if we install dbt deps should be done in the same manner as LOCAL or VIRTUALENV execution modes. So I am changing the code to use the ProjectConfig settings for dbt deps to also work with the Watcher and now it will be in sync with the documentation. Closes: #2099 (cherry picked from commit ce87c97)
Further improve the `ExecutionMode.AIRFLOW_ASYNC`documentation by: - Encourage high performance configuration by default - Avoid information duplication (cherry picked from commit 80f1678)
pankajkoti
reviewed
Nov 12, 2025
Contributor
pankajkoti
left a comment
pankajkoti
left a comment
There was a problem hiding this comment.
Can/should we also cherry-pick #2114
Approving the PR in advance and believe you'd get a chance to consider reviewing my inline comments
pankajkoti
approved these changes
Nov 12, 2025
Contributor
|
Once released and merged, keeping this comment as a reminder to add the Changelog for 1.11.1 into |
…#2119) Fixes SQL not rendering in Airflow UI when using `DbtRunLocalOperator.partial` in Airflow 2. The problem was that when using dynamically mapped tasks in Airflow 2, the `_override_rtif method` was not filtering by map_index when deleting old RenderedTaskInstanceFields records. This meant it could delete records for other mapped task instances, not just the current one. I added a unittest to confirm the implementation takes into account the mapped tasks index, and I also validated by running `airflow standalone` with the example DAG shared by @hkc-8010 in the original bug report. The screenshots below show the SQL being rendered in all the tasks of the Dynamic Task Group, instead of only rendering for the last task: <img width="4064" height="2334" alt="Screenshot 2025-11-12 at 16 32 55" src="https://github.com/user-attachments/assets/75ce3a91-a6b5-4c2e-b37e-6369db7d15bc" /> <img width="4064" height="2334" alt="Screenshot 2025-11-12 at 16 34 33" src="https://github.com/user-attachments/assets/04b8e17a-c9f7-45e4-b8ac-d3a5e6ddb9a1" /> <img width="4064" height="2334" alt="Screenshot 2025-11-12 at 16 34 23" src="https://github.com/user-attachments/assets/21116ec7-d211-4afc-b0c3-8fc9ed69c28a" /> Closes: #2018 (cherry picked from commit 4994474)
Co-authored-by: Pankaj Koti <pankajkoti699@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bug fixes
ExecutionMode.WATCHERdeadlock in Airflow 3.0 & 3.1 by @tatiana in FixExecutionMode.WATCHERdeadlock in Airflow 3.0 & 3.1 #2087ExecutionMode.AIRFLOW_ASYNCTaskGroupXCom issue by @tatiana in FixExecutionMode.AIRFLOW_ASYNCTaskGroupXCom issue #2088ExecutionMode.WATCHERto useinstall_dbt_depsfromProjectConfigby @michal-mrazek in Fix ExecutionMode.WATCHER to use install_dbt_deps from ProjectConfig #2112Enhancements
Docs
ExecutionMode.AIRFLOW_ASYNCdocs by @tatiana in docs: ImproveAIRFLOW_ASYNCdocs #2103operator_argstooperator_kwargsby @jx2lee in docs: correct example YAML key from operator_args to operator_kwargs #2091Others
related: https://github.com/astronomer/oss-integrations-private/issues/272