Implement schedule on @asset

[Lee-W]

Description

Schedules

This will probably be achieve by using AssetRef to access Asset by name

This one could be deferred to "Data Completeness"

Use case/motivation

Rationale

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[uranusjr]

PR to add asset reference support #45028

[uranusjr]

There’s a second part to come later to implement modern schedule dates (logical date at the end, no data intervals).

[uranusjr]

PR for asset ref merged.

[uranusjr]

There’s still this part of the AIP left https://cwiki.apache.org/confluence/x/RA2TEg

The key difference is, since we are decoupling partitioning from scheduling, the schedule parameter no longer controls the interval, i.e. not designed around the logical/execution date. It simply controls when the next round should happen.

It is expected that scheduling of non-asset workflows (DAGs) will also be changed in a similar way to match the behavior for assets. Existing operators must be reviewed to ensure they account for the new scheduling semantic, but we should provide a transition interface to assist rewrites and better allow providers to develop a implementation both compatible to 2 and 3. The timetable protocol will also require new methods to implement the new semantic, but both old and new should be able to both implemented on the same class, with each major version only calling methods its uses.

This behaviour is partially in line with the amendments proposed for AIP-83, and will be implemented together with changes to the DAG class. https://cwiki.apache.org/confluence/x/Ngv0Ew

Specifically:

• A DAG run backing an @asset materialisation will have its logical date and data interval values set to null in the database.
• The @asset function’s execution context dict will not contain keys logical_date (and derived values such as ds), data_interval_start, and data_interval_end.
• The run will be represented in user-facing interfaces by the run_after date (in line with all logical-date-less DAG runs in the amendment).

[phanikumv]

Depends on #46192