[Feature] Refactor and add support for schedule conditions in DAG configuration:

[ErickSeo]

Description

This feature introduces a enhancement to DAG scheduling in Airflow, enabling support for dynamic schedules based on dataset conditions. By leveraging dataset filters and logical conditions, users can now create more flexible and precise scheduling rules tailored to their workflows.

Key Features:

• Condition-Based Scheduling: Allows defining schedules using logical conditions between datasets (e.g., ('dataset_1' & 'dataset_2') | 'dataset_3'), enabling workflows to trigger dynamically based on dataset availability.

• Dynamic Dataset Processing: Introduced the process_file_with_datasets function to evaluate and process dataset URIs from external files, supporting both simple and condition-based schedules.

• Improved Dataset Evaluation: Developed the evaluate_condition_with_datasets function to transform dataset URIs into valid variable names and evaluate logical conditions securely.

Workflow Example:
Given the following condition:

example_custom_config_condition_dataset_consumer_dag:
  description: "Example DAG consumer custom config condition datasets"
  schedule:
    file: $CONFIG_ROOT_DIR/datasets/example_config_datasets.yml
    datasets:  "((dataset_custom_1 & dataset_custom_2) | dataset_custom_3)"
  tasks:
    task_1:
      operator: airflow.operators.bash_operator.BashOperator
      bash_command: "echo 'consumer datasets'"

example_without_custom_config_condition_dataset_consumer_dag:
  description: "Example DAG consumer custom config condition datasets"
  schedule:
    datasets: "((s3://bucket-cjmm/raw/dataset_custom_1 & s3://bucket-cjmm/raw/dataset_custom_2) | s3://bucket-cjmm/raw/dataset_custom_3)"
  tasks:
    task_1:
      operator: airflow.operators.bash_operator.BashOperator
      bash_command: "echo 'consumer datasets'"

example_without_custom_config_condition_dataset_consumer_dag:
  description: "Example DAG consumer custom config condition datasets"
  schedule:
    datasets: 
      !or  
        - !and  
          - "s3://bucket-cjmm/raw/dataset_custom_1"  
          - "s3://bucket-cjmm/raw/dataset_custom_2"
        - "s3://bucket-cjmm/raw/dataset_custom_3"
  tasks:
    task_1:
      operator: airflow.operators.bash_operator.BashOperator
      bash_command: "echo 'consumer datasets'"

The system evaluates the datasets, ensuring valid references, and schedules the DAG dynamically when the condition resolves to True.

Example Use Case:
Consider a data pipeline that processes files only when multiple interdependent datasets are updated. With this feature, users can create dynamic DAG schedules that automatically adjust based on dataset availability and conditions, optimizing resource allocation and execution timing.

Images:

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 97.43590% with 3 lines in your changes missing coverage. Please review.

Project coverage is 94.01%. Comparing base (48e5575) to head (fb33b67).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
dagfactory/utils.py	88.00%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #320      +/-   ##
==========================================
+ Coverage   93.62%   94.01%   +0.39%     
==========================================
  Files          10       11       +1     
  Lines         784      886     +102     
==========================================
+ Hits          734      833      +99     
- Misses         50       53       +3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[tatiana]

Hi @ErickSeo, this feature is super exciting. It significantly improves how DAG factory handles datasets. I'm sorry for the delay in reviewing it. I've left some comments in-line and am happy to discuss alternatives.

If we're able to reach a consensus on the implementation, we can aim to release this feature in DAG Factory 0.22, planned to be released on 10 January 2025.

[tatiana]

This is a very exciting feature, @ErickSeo ! Thank you very much for adding support to a more powerful way of scheduling DAGs in DAG Factory.

Thank you for addressing all the feedback. I'm sure there will be use-cases we did not think about - but we can always iterate over them.

[pankajkoti]

Yes, indeed thanks a lot for the contribution and quickly addressing the review comments, much helpful @ErickSeo 🚀