Docs: clarify AIRFLOW_ASYNC dataset limits. by pankajkoti · Pull Request #2167 · astronomer/astronomer-cosmos

pankajkoti · 2025-11-26T15:44:52Z

Following up on PR #2143, add a note on the dataset event emission limitation for ExecutionMode.AIRFLOW_ASYNC in other relevant places of the documentation.

closes: #2142
related: #2143
related: #2141

Copilot

Pull request overview

This PR adds documentation about the limitation that ExecutionMode.AIRFLOW_ASYNC does not emit Dataset or DatasetAlias events, which was identified in issue #2141. The changes ensure this important limitation is documented in all relevant places where users might configure or use this execution mode.

Adds important note boxes to execution modes and scheduling documentation
Updates the emit_datasets parameter description in render config documentation

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
docs/getting_started/execution-modes.rst	Adds an important note box documenting the dataset emission limitation for AIRFLOW_ASYNC mode
docs/configuration/scheduling.rst	Adds an important note box about the dataset emission limitation in the data-aware scheduling section
docs/configuration/render-config.rst	Updates the `emit_datasets` parameter description to mention AIRFLOW_ASYNC does not emit datasets

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

codecov · 2025-11-26T16:21:13Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.81%. Comparing base (342d381) to head (7112bcd).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2167   +/-   ##
=======================================
  Coverage   97.81%   97.81%           
=======================================
  Files          93       93           
  Lines        6006     6006           
=======================================
  Hits         5875     5875           
  Misses        131      131

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

pankajastro

Looks good.

Looks like docs preview comment is missing here?

pankajkoti · 2025-11-27T09:43:21Z

Looks like docs preview comment is missing here?

Yes, looks like Netlify configuration is no longer working. I have asked for information in Astronomer Slack: https://astronomer.slack.com/archives/C015V2JFKT5/p1764236517168529

pankajkoti · 2025-11-27T14:06:45Z

Thanks @pankajastro for the review. I will wait for to see if there are inputs from @tatiana when she gets a chance to review before merging the PR

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Co-authored-by: Tatiana Al-Chueyr <tatiana.alchueyr@gmail.com>

Following up on PR astronomer#2143, add a note on the dataset event emission limitation for `ExecutionMode.AIRFLOW_ASYNC` in other relevant places of the documentation. closes: astronomer#2142 related: astronomer#2143 related: astronomer#2141 --------- Co-authored-by: Tatiana Al-Chueyr <tatiana.alchueyr@gmail.com>

@pankajastro

Breaking changes * Introduced in the PR #2080. The following functions are expected to be used internally only to Cosmos, so we hope these won't impact end-users, but we are documenting the changes just in case: - ``generate_task_or_group`` receives ``render_config`` instead of its individual configurations, such as ``test_behavior``, ``source_rendering_behavior`` and ``enable_owner_inheritance`` - ``create_task_metadata`` receives ``render_config`` instead of its individual configurations, such as ``test_behavior``, ``source_rendering_behavior`` and ``enable_owner_inheritance`` - ``create_task_metadata`` now expects the ``node_converters`` argument * Drop Python 3.9 support by @pankajastro in #2118 * Drop Airflow 2.4 support by @pankajastro in #2161 * Drop Airflow 2.5 support by @pankajastro in #2165 Features * Support applying ``node_converter`` at a task level instead of task group level by @anyapriya in #1759 * Allow overriding ``DbtProducerWatcherOperator`` parameters via ``ExecutionConfig.setup_operator_args`` by @pankajastro in #2133 * Use deferrable sensors by default in ``ExecutionMode.WATCHER`` by @pankajastro in #2084 * Support real-time consumer updates when using ``ExecutionMode.WATCHER`` and ``InvocationMode.SUBPROCESS`` by @pankajastro in #2152 * Update telemetry to v3 format with query parameters by @pankajkoti in #2192 * Add initial set of telemetry task listener metrics for Cosmos operators by @pankajkoti in #2195 Enhancements * Unify Airflow version handling into ``constants.py`` by @tatiana in #2089 * Refactor ``airflow/graph.py`` to simplify the code base by @tatiana in #2080 * Force watcher producer retries to zero by @pankajkoti in #2114 * Fail ``ExecutionMode.WATCHER`` consumer sensors immediately when the producer fails using Airflow context by @pankajkoti in #2126 * ``ExecutonMode.WATCHER``: fetch producer status asynchronously from the Airflow runtime so deferrable sensors fail immediately when the producer task fails by @pankajkoti in #2144 * Refactor ``ExecutionMode.WATCHER`` ``InvocationMode.SUBPROCESS`` log parser by @tatiana in #2183 * Replace map_index with is_mapped_task boolean in task telemetry metrics by @pankajkoti in #2210 * Collect cosmos profile metrics in task telemetry metrics by @pankajastro in #2198 * Remove unnecessary information from telemetry by @tatiana in #2211 Bug fixes * Clarify ``ExecutionMode.WATCHER`` deferrable failure messaging by @pankajkoti in #2124 * Remove empty test tasks when all tests are detached by @anyapriya in #2010 * Fix forwarding ``DbtProducerWatcherOperator`` ``dbt build`` flags by @michal-mrazek in #2127 * Add databricks oauth mock profile by @fjmacagno in #2164 * Register listeners in Airflow 3 plugin implementation by @pankajastro in #2187 * Fix resolution of ``packages-install-path`` when it uses ``env_var`` by @tatiana in #2194 * Fix ``template_fields`` in ``DbtConsumerWatcherSensor`` to include ``DbtRunLocalOperator`` template_fields`` by @tiovader and @emanuel-luis in #2209 * Emit asset events in ExecutionMode.AIRFLOW_ASYNC mode by @pankajastro in #2184 * Remove dag_run_id from telemetry tests by @tatiana in #2213 Docs * Document dataset-event limitation when using ``ExecutionMode.AIRFLOW_ASYNC`` by @varaprasadregani in #2143 * Expand ``ExecutionMode.KUBERNETES`` guidance by @tatiana in #2139 * Add docs for deferrable ``DbtConsumerWatcherSensor`` by @pankajastro in #2115 * Fix reStructuredText formatting by @dnskr in #2132 * Add docs for ``setup_operator_args`` param by @pankajastro in #2136 * Remove experimental flag for ``ExecutionMode.AIRFLOW_ASYNC`` by @pankajastro in #2153 * Clarify ``ExecutionMode.AIRFLOW_ASYNC`` dataset limits by @pankajkoti in #2167 * Update PRIVACY_NOTICE.rst by @tatiana in #2212 Others * Drop Python 3.9 support by @pankajastro in #2118 * Drop Airflow 2.4 support by @pankajastro in #2161 * Drop Airflow 2.5 support by @pankajastro in #2165 * Improve example DAG ``jaffle_shop_kubernetes.py`` by @tatiana in #2140 * Enable tests for Python 3.13 by @pankajastro in #2154 * Add Python 3.12 to CI integration tests matrix by @pankajastro in #2168 * Retry flaky Telemetry success test to stabilise CI by @pankajkoti in #2138 * Drop unused producer state xcom handling in ``ExecutionMode.WATCHER`` by @pankajkoti in #2145 * Remove unused Python3.9 uses from Github action CI by @pankajastro in #2117 * Run pre-commit on ``ExecutionMode.WATCHER`` modules by @pankajkoti in #2150 * Refactor: Use shared airflow version constant by @pankajkoti in #2157 * Pin ``pydantic<2.0`` for Airflow 2.6 compatibility by @pankajastro in #2172 * Remove duplicate ``dbt-duckdb`` dependency by @pankajastro in #2170 * Add targeted ``type: ignore`` for untyped decorators to fix ``mypy`` errors by @pankajastro in #2174 * Replace Legacy typing Aliases with Built-in Types for Python 3.10+ by @pankajastro in #2175 * Refactor to reuse ``load_method_from_module`` from ``_utils/importer.py`` by @pankajastro in #2176 * Remove try except block for cache import and unused python_version variable by @pankajastro in #2186 * Unpin Airflow to satisfy GitHub Security tab requirements by @pankajastro in #2171 * Update Python version for ``pyupgrade`` in ``pre-commit`` config by @pankajastro in #2190 * Add cooldown config in ``dependabot`` config by @pankajastro in #2189 * Adjust pre-commit so Python 3.10 or higher can be used by @tatiana in #2196 * Remove empty variables emission from telemetry metrics by @pankajkoti in #2197 * Reformat documented comments for historical URL formats by @pankajkoti in #2199 * Bump ``actions/checkout`` from ``5.0.0`` to ``5.0.1`` by @dependabot in #2135 * Bump ``actions/checkout`` to ``6.0.0`` in GitHub workflows by @dependabot in #2147 * Bump ``zizmorcore/zizmor-action`` from ``0.2.0`` to ``0.3.0`` by @dependabot in #2156 * Bump ``actions/checkout`` from ``5.0.1`` to ``6.0.0`` by @dependabot in #2155 * Bump ``actions/checkout`` from ``6.0.0`` to ``6.0.1`` by @dependabot in #2178 * Bump ``codecov/codecov-action`` from ``5.5.1`` to ``5.5.2`` by @dependabot in #2208 * pre-commit autoupdate by @pre-commit-ci[bot] in #2134, #2162, #2173, #2191, #2202 closes: astronomer/oss-integrations-private#275

pankajkoti temporarily deployed to internal November 26, 2025 15:44 — with GitHub Actions Inactive

pankajkoti force-pushed the docs/clarify-async-dataset branch from 9a06a0f to f227349 Compare November 26, 2025 15:45

pankajkoti temporarily deployed to internal November 26, 2025 15:45 — with GitHub Actions Inactive

pankajkoti marked this pull request as ready for review November 26, 2025 15:51

pankajkoti requested review from Copilot, pankajastro and tatiana November 26, 2025 15:51

Copilot AI reviewed Nov 26, 2025

View reviewed changes

Comment thread docs/configuration/scheduling.rst Outdated

Comment thread docs/configuration/render-config.rst Outdated

pankajkoti force-pushed the docs/clarify-async-dataset branch from f227349 to 89ca8cd Compare November 26, 2025 15:52

pankajkoti changed the title ~~Address review comments~~ Docs: clarify AIRFLOW_ASYNC dataset limits. Nov 26, 2025

pankajkoti temporarily deployed to internal November 26, 2025 15:53 — with GitHub Actions Inactive

pankajkoti force-pushed the docs/clarify-async-dataset branch from 89ca8cd to f249001 Compare November 26, 2025 15:55

pankajkoti temporarily deployed to internal November 26, 2025 15:56 — with GitHub Actions Inactive

pankajastro approved these changes Nov 26, 2025

View reviewed changes

tatiana reviewed Dec 1, 2025

View reviewed changes

Comment thread docs/configuration/render-config.rst Outdated

tatiana reviewed Dec 1, 2025

View reviewed changes

Comment thread docs/configuration/scheduling.rst Outdated

tatiana reviewed Dec 1, 2025

View reviewed changes

Comment thread docs/getting_started/execution-modes.rst Outdated

pankajkoti temporarily deployed to internal December 1, 2025 14:45 — with GitHub Actions Inactive

pankajkoti commented Dec 1, 2025

View reviewed changes

Comment thread docs/configuration/scheduling.rst Outdated

pankajkoti temporarily deployed to internal December 1, 2025 14:47 — with GitHub Actions Inactive

pankajkoti temporarily deployed to internal December 1, 2025 14:48 — with GitHub Actions Inactive

pankajkoti temporarily deployed to internal December 1, 2025 14:52 — with GitHub Actions Inactive

pankajkoti and others added 5 commits December 1, 2025 20:23

Docs: clarify AIRFLOW_ASYNC dataset limits

82ed604

Update docs/configuration/render-config.rst

91d52b2

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update docs/configuration/scheduling.rst

f9d38a9

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Apply suggestion from @tatiana

0b4ed10

Co-authored-by: Tatiana Al-Chueyr <tatiana.alchueyr@gmail.com>

Apply suggestion from @pankajkoti

3f30229

pankajkoti and others added 2 commits December 1, 2025 20:23

Apply suggestion from @tatiana

b758cfb

Co-authored-by: Tatiana Al-Chueyr <tatiana.alchueyr@gmail.com>

Fix backticks

76f9566

pankajkoti force-pushed the docs/clarify-async-dataset branch from 8a02339 to 76f9566 Compare December 1, 2025 14:53

pankajkoti temporarily deployed to internal December 1, 2025 14:53 — with GitHub Actions Inactive

pankajkoti merged commit e4d87e6 into main Dec 1, 2025
76 checks passed

pankajkoti deleted the docs/clarify-async-dataset branch December 1, 2025 14:54

pankajkoti mentioned this pull request Dec 9, 2025

Release 1.12.0 #2149

Merged

tatiana added this to the Cosmos 1.12.0 milestone Dec 18, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Docs: clarify AIRFLOW_ASYNC dataset limits.#2167

Docs: clarify AIRFLOW_ASYNC dataset limits.#2167
pankajkoti merged 7 commits into
mainfrom
docs/clarify-async-dataset

pankajkoti commented Nov 26, 2025 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Nov 26, 2025

Uh oh!

pankajastro left a comment •

edited

Loading

Uh oh!

pankajkoti commented Nov 27, 2025

Uh oh!

pankajkoti commented Nov 27, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

pankajkoti commented Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Nov 26, 2025

Codecov Report

Uh oh!

pankajastro left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pankajkoti commented Nov 27, 2025

Uh oh!

pankajkoti commented Nov 27, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

pankajkoti commented Nov 26, 2025 •

edited

Loading

pankajastro left a comment •

edited

Loading