Skip to content

Add support for defining inlets#380

Merged
tatiana merged 9 commits into
astronomer:mainfrom
IvanSviridov:add_inlet_support
Mar 25, 2025
Merged

Add support for defining inlets#380
tatiana merged 9 commits into
astronomer:mainfrom
IvanSviridov:add_inlet_support

Conversation

@IvanSviridov

@IvanSviridov IvanSviridov commented Mar 10, 2025 •
edited by tatiana
Loading

Copy link
Copy Markdown
Contributor

DAG Factory 0.22 only supports defining outlets, but there is no built-in support for inlets. This limitation prevents proper integration with lineage tracking services such as DataHub, OpenLineage, and other metadata-driven tools.

Why is this needed?

  • Airflow 2.4.0+ introduced Dataset-based scheduling, where both inlets and outlets play a critical role in tracking data dependencies.
  • Many data lineage tools (e.g., DataHub, Marquez, OpenLineage) rely on both inlets and outlets to correctly infer dataset relationships.
  • Without inlets, lineage propagation in Airflow does not fully capture data source dependencies, which results in incomplete lineage graphs.

Closes: #379

@IvanSviridov IvanSviridov requested a review from a team as a code owner March 10, 2025 12:06
@codecov-commenter

codecov-commenter commented Mar 10, 2025 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 94.04%. Comparing base (61beb29) to head (6785ee8).
Report is 1 commits behind head on main.

Additional details and impacted files 
@@            Coverage Diff             @@
##             main     #380      +/-   ##
==========================================
+ Coverage   94.01%   94.04%   +0.02%     
==========================================
  Files          11       11              
  Lines         886      890       +4     
==========================================
+ Hits          833      837       +4     
  Misses         53       53

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@IvanSviridov IvanSviridov mentioned this pull request Mar 10, 2025
Closed
1 task
tatiana
tatiana reviewed Mar 10, 2025
View reviewed changes

@tatiana tatiana left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is looking great, @IvanSviridov , is there any chance you could add some tests and an example YAML? If you could extend our docs with this feature, it would also be amazing.

@IvanSviridov

IvanSviridov commented Mar 17, 2025

Copy link
Copy Markdown
Contributor Author

@tatiana, could you check it again please
and it is issue #379

@IvanSviridov IvanSviridov requested a review from tatiana March 17, 2025 12:59
tatiana
tatiana approved these changes Mar 25, 2025
View reviewed changes

@tatiana tatiana left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great, thanks for adding tests and updating the docs, alongside the feature itself, @IvanSviridov !

if possible, please, could you just remove the dags symbolic link? This feels like an unrelated change.

tatiana
tatiana reviewed Mar 25, 2025
View reviewed changes
Comment thread dags Outdated
@tatiana tatiana changed the title Added support for inlets in dag-factory Add support for defining inlets Mar 25, 2025
@tatiana tatiana merged commit 0f8aad3 into astronomer:main Mar 25, 2025
@tatiana tatiana added this to the DAG Factory 0.23.0 milestone Mar 25, 2025
@matveykortsev

matveykortsev commented Mar 25, 2025

Copy link
Copy Markdown
Contributor

@tatiana Hi! When you are planning to release this? Is it possible to release as minor release of 22x?

@pankajastro pankajastro mentioned this pull request Mar 28, 2025
Merged
@pankajastro

pankajastro commented Mar 28, 2025

Copy link
Copy Markdown
Contributor

Hi @matveykortsev, We have released a pre-release version, dag-factory 0.23.0a1, which is expected to include this PR.

@pankajastro pankajastro removed this from the DAG Factory 0.23.0 milestone Jul 8, 2025
pankajastro added a commit that referenced this pull request Jul 14, 2025
@pankajastro
Release 0.23.0 (#386)
1aa6ba0 
## [0.23.0] - 2025-07-14

### Breaking Change

- Drop Airflow 2.2 Support by @pankajastro in
[#388](#388)
- Drop Python 3.8 support by @pankajastro in
[#435](#435)
- Drop Airflow 2.3 Support by @pankajastro in
[#456](#456)

### Airflow 3 Support

- Add tools for Dag-factory Airflow3 testing by @pankajastro in
[#395](#395)
- Fix schedule args for Airflow 3 by @pankajastro in
[#413](#413)
- Fix import path for BranchPythonOperator, PythonOperator and
PythonSensor by @pankajastro in
[#414](#414)
- Add scheduling docs for Airflow 3 by @pankajastro in
[#424](#424)
- Enable Airflow 3 tests in CI by @pankajastro in
[#436](#436)
- Add env AUTO_CONVERT_TO_AF3 in Dockerfile by @pankajastro in
[#455](#455)
- Validate DAG's on Airflow 3 by @pankajastro in
[#457](#457)
- Refactor schedule to use the Python object @pankajastro in
[#458](#458)
- Fix CI and import issues for Airflow 3 compatibility @pankajastro in
[#463](#463)

### Added

- Add support for defining inlets by @IvanSviridov in
[#380](#380)
- Add HttpOperator JSON serialization support with tests by @a-chumagin
in [#382](#382)
- Add support for custom Python object by @pankajastro in
[#444](#444)
- Support env var in default by @gyli in
[#452](#452)
- Pass default arguments via dictionary in .py file by @jroachgolf84 in
[#465](#465)

### Fixed

- Remediated ``default`` behavior, added documentation by @jroachgolf84
in [#378](#378)
- Upgrade `apache-airflow-providers-cncf-kubernetes` provider by
@pankajastro in
[#407](#407)
- Include error message trace in exception by @pankajastro in
[#408](#408)

### Docs

- Remove Unreleased heading section from the CHANGELOG.md by @pankajkoti
in [#365](#365)
- Add Documentation for Conditional Dataset Scheduling with dag-factory
by @ErickSeo in
[#367](#367)
- Add copy right in docs footer by @pankajastro in
[#371](#371)
- Updating docs for callbacks by @jroachgolf84 in
[#375](#375)
- Add stable/latest version in docs by @pankajastro in
[#391](#391)
- Migrate old content to new documentation structure by @pankajastro in
[#393](#393)
- Update Airflow supported version 2.3+ in docs by @pankajastro in
[#412](#412)
- Doc: Add step to fork repo in contributing guide by @pankajastro in
[#427](#427)
- Add setting CONFIG_ROOT_DIR in the contribution doc by @gyli in
[#432](#432)
- Add DAG example showcasing runtime params usage by @pankajastro in
[#449](#449)
- Add jinja2 template example by @pankajastro in
[#450](#450)

### Others

- Add Scraf Pixels for telemetry by @pankajastro in
[#373](#373)
- feat: bumped http provider versions to 2.0+ by @a-chumagin in
[#389](#389)
- Add --verbosity debug in astro-cli cmd by @pankajastro in
[#390](#390)
- Add missing Python file for dynamic task example by @pankajastro in
[#392](#392)
- Pin apache-airflow-providers-cncf-kubernetes<10.4.2 by @pankajastro in
[#400](#400)
- Add script to check version and tag by @pankajastro in
[#395](#394)
- Assert DagRunState in integration test by @pankajastro in
[#415](#415)
- Move to uv for package management by @jlaneve in
[#419](#419)
- Install uv in CI by @jlaneve in
[#421](#421)
- Bump astral-sh/setup-uv from 5 to 6 by @dependabot in
[#423](#423)
- Add Airflow 2.11 in test matrix by @pankajastro in
[#425](#425)
- fix example_dag_factory.yml typo causing catchup: false to not be
respected by @RNHTTR in
[#431](#431)
- Remove expandvars in utils.get_python_callable by @gyli in
[#440](#440)
- Delete unused img folder by @pankajastro in
[#446](#446)
- Clean print statement by @pankajastro
[#447](#447)
- Add hatch to uv dev dependencies by @gyli in
[#453](#453)
- Add Authorize Job in CI by @pankajastro in
[#460](#460)
- Remove PyPI token for releasing packages by @tatiana in
[#461](#461)
- Change CI on trigger event to pull_request from pull_request_target by
@pankajkoti in
[#464](#464)
- Update cicd.yaml: Use pull_request for authorize as we don't have
pull_request_target event configured by @pankajkoti in
[#466](#466)
- Add environment for pypi publish job by @pankajastro in
[#478](#478)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tatiana tatiana tatiana approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Feature] Support for inlets in dag-factory

5 participants

@IvanSviridov @codecov-commenter @matveykortsev @pankajastro @tatiana