Skip to content

Enable inlets and outlets using DBT Fusion on Airflow 3#2561

Merged
tatiana merged 6 commits into
astronomer:mainfrom
ichirotakami:main
Apr 17, 2026
Merged

Enable inlets and outlets using DBT Fusion on Airflow 3#2561
tatiana merged 6 commits into
astronomer:mainfrom
ichirotakami:main

Conversation

@ichirotakami
Copy link
Copy Markdown
Contributor

Description

This PR enables inlets and outlets on Airflow 3 using DBT Fusion when using DbtRunOperationLocalOperator, by letting DbtLocalArtifactProcessor() find --profiles-dir by passing dbt_command_line tocalculate_openlineage_events_completes as dbt-fusion does not write profiles_dir into run_results.json.

Please see #2560 for more details about the issue.

Related Issue(s)

closes: #2560

Breaking Change?

Checklist

  • I have made corresponding changes to the documentation (if required)
  • I have added tests that prove my fix is effective or that my feature works

Copilot AI review requested due to automatic review settings April 15, 2026 13:55
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes missing Airflow 3 inlets/outlets when running dbt-fusion via DbtRunOperationLocalOperator by ensuring OpenLineage’s DbtLocalArtifactProcessor can locate --profiles-dir (since dbt-fusion doesn’t write profiles_dir into run_results.json).

Changes:

  • Pass the full dbt CLI command (full_cmd) into calculate_openlineage_events_completes and forward it as dbt_command_line to DbtLocalArtifactProcessor.
  • Update the unit test that calls calculate_openlineage_events_completes to match the new method signature.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File Description
cosmos/operators/local.py Threads the dbt command line into the OpenLineage artifact processor so profiles discovery works with dbt-fusion on Airflow 3.
tests/operators/test_local.py Updates a unit test invocation to include the new dbt_command_line argument.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread cosmos/operators/local.py
Comment thread cosmos/operators/local.py Outdated
Comment thread tests/operators/test_local.py
@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 15, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.05%. Comparing base (fa3828b) to head (54f392c).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #2561   +/-   ##
=======================================
  Coverage   98.04%   98.05%           
=======================================
  Files         103      103           
  Lines        7589     7593    +4     
=======================================
+ Hits         7441     7445    +4     
  Misses        148      148           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copy link
Copy Markdown
Collaborator

@tatiana tatiana left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HI @ichirotakami , thank you very much for fixing this issue!

Please, could you address The feedback in the comments:
#2561 (comment)
#2561 (comment)

Copilot AI review requested due to automatic review settings April 16, 2026 13:42
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@ichirotakami
Copy link
Copy Markdown
Contributor Author

Hi @tatiana, thank you for your response. I have attempted to address the feedback with a new commit. Please let me know if you think I need to change something.

Copilot AI review requested due to automatic review settings April 17, 2026 08:59
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

cosmos/operators/local.py:727

  • The calculate_openlineage_events_completes docstring is now missing any mention of the new optional dbt_command_line parameter, and it still states it "Return[s] a list of RunEvents" even though the method returns None and mutates self.openlineage_events_completes. Please update the docstring to reflect the current behavior and document how dbt_command_line is used (e.g., to help OpenLineage locate --profiles-dir for dbt-fusion runs).
    def calculate_openlineage_events_completes(
        self,
        env: dict[str, str | os.PathLike[Any] | bytes],
        project_dir: Path,
        dbt_command_line: list[str] | None = None,
    ) -> None:
        """
        Use openlineage-integration-common to extract lineage events from the artifacts generated after running the dbt
        command. Relies on the following files:
        * profiles
        * {project_dir}/target/manifest.json
        * {project_dir}/target/run_results.json

        Return a list of RunEvents
        """

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copy link
Copy Markdown
Collaborator

@tatiana tatiana left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you very much, @ichirotakami , it looks very good - and the context you gave in the issue was also amazing, it really helped understanding the underlining issue.

@tatiana tatiana merged commit 6ba5403 into astronomer:main Apr 17, 2026
127 of 128 checks passed
@pankajkoti pankajkoti mentioned this pull request Apr 23, 2026
@tatiana tatiana added this to the Cosmos 1.14.1 milestone Apr 23, 2026
tatiana pushed a commit that referenced this pull request Apr 23, 2026
Bug Fixes

* Fix ``ExecutionMode.WATCHER`` producer retry behaviour by @tatiana in
#2559
* Prevent watcher producer skip propagating to downstream tasks via
gateway task by @johnhoran and @tatiana in #2597
* Keep watcher sensor polling when producer is still running by
@pankajkoti in #2592
* Fix circular import error in Cosmos plugin discovery under Astro
Runtime by @tatiana in #2538
* Fix ``CosmosRichLogger`` crash on ``None`` log message by @tatiana in
#2540
* Enable inlets and outlets using dbt Fusion on Airflow 3 by
@ichirotakami in #2561
* Fix incorrectly skipped source downstream tasks in
``ExecutionMode.WATCHER`` by @pankajastro in #2563
* Fix duplicate logs in ``dbt build`` when source freshness is enabled
by @pankajastro in #2564
* Warn and normalize when ``source_rendering_behavior=None`` is passed
by @pankajastro in #2570
* Gracefully handle ``Variable.set()`` failures on Astro Remote
Execution by @hkc-8010 in #2573
* Skip malformed YAML selectors instead of failing entirely by
@YourRoyalLinus in #2577

Docs

* Update watcher test behavior docs for Cosmos 1.14.0 by @tatiana in
#2549
* Add redirect for moved partial-parsing docs page by @tatiana in #2550
* Document ``ExecutionMode.WATCHER`` and ``depends_on_past`` limitation
by @tatiana in #2602
* Restore memory-optimised imports docs for Cosmos < 1.14.0 by
@pankajkoti in #2604

Others

* Speed up Airflow 3.1+ integration tests by caching
InProcessExecutionAPI by @pankajkoti in #2547
* Improve stability of cache hash unit tests by @tatiana in #2539
* Fix mypy 1.20.0 type check failures by @pankajkoti in #2546
* Fix CI failures caused by docs build memory exhaustion by @pankajkoti
in #2580
* Fix dbt Fusion broken integration tests by @tatiana in #2581
* Fix flaky ``cosmos_manifest_selectors_example`` DAG in CI by
@pankajkoti in #2593
* Reduce pre-commit autoupdate frequency PRs by @tatiana in #2544
* Bump ``reviewdog/action-actionlint`` from 1.71.0 to 1.72.0 by
@dependabot in #2542
* Skip watcher gateway test on Airflow 3.0 by @tatiana in #2607

closes: astronomer/oss-integrations-private#381
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Enable inlets and outlets using DBT Fusion on Airflow 3

3 participants