Improve dataset/asset experience in Cosmos

[tatiana]

While validating Cosmos Airflow Asset support in Airflow 3.1, I noticed that we could improve the overall experience.

Previously, Cosmos required both openlineage-integration-common and openlineage-airflow or apache-airflow-providers-openlineage to be installed to emit Airflow Datasets or Airflow Assets. However, the only necessary dependency is openlineage-integration-common.

Users will need either openlineage-airflow or apache-airflow-providers-openlineage for emitting OpenLineage events - but not Airflow Datasets and Aliasses.

This PR improves this behaviour by adding openlineage-integration-common as a dependency of the Cosmos standard installation - and not failing to emit datasets if openlineage-airflow or apache-airflow-providers-openlineage are not installed.

It was validated with Airflow 3.1.0 and validates with the following three DAGs:

Producer Cosmos DbtDag

Existing basic_cosmos_dag example

import os
from datetime import datetime
from pathlib import Path

from cosmos import DbtDag, ProfileConfig, ProjectConfig
from cosmos.profiles import PostgresUserPasswordProfileMapping

DEFAULT_DBT_ROOT_PATH = Path(__file__).parent / "dbt"
DBT_ROOT_PATH = Path(os.getenv("DBT_ROOT_PATH", DEFAULT_DBT_ROOT_PATH))
DBT_PROJECT_NAME = os.getenv("DBT_PROJECT_NAME", "jaffle_shop")
DBT_PROJECT_PATH = DBT_ROOT_PATH / DBT_PROJECT_NAME


profile_config = ProfileConfig(
    profile_name="default",
    target_name="dev",
    profile_mapping=PostgresUserPasswordProfileMapping(
        conn_id="example_conn",
        profile_args={"schema": "public"},
        disable_event_tracking=True,
    ),
)

basic_cosmos_dag = DbtDag(
    # dbt/cosmos-specific parameters
    project_config=ProjectConfig(DBT_PROJECT_PATH),
    profile_config=profile_config,
    operator_args={
        "install_deps": True,  # install any necessary dependencies before running any dbt command
        "full_refresh": True,  # used only in dbt commands that support this flag
    },
    # normal dag parameters
    schedule="@daily",
    start_date=datetime(2023, 1, 1),
    catchup=False,
    dag_id="basic_cosmos_dag",
    default_args={"retries": 0},
)

Consumer DAG scheduled based on an Asset

from datetime import datetime
from airflow import DAG
from airflow.sdk.definitions.asset import Asset
from airflow.operators.empty import EmptyOperator


with DAG(
    "dataset_triggered_dag",
    description="A DAG that should be triggered via Dataset",
    start_date=datetime(2024, 9, 1),
    schedule=[Asset(uri="postgres://0.0.0.0:5432/postgres/public/orders")],
) as dag:
    t1 = EmptyOperator(
        task_id="task_1",
    )
    t2 = EmptyOperator(
        task_id="task_2",
    )

    t1 >> t2

Consumer DAG scheduled based on a AssetAlias

from datetime import datetime
from airflow import DAG
from airflow.sdk.definitions.asset import AssetAlias
from airflow.operators.empty import EmptyOperator


with DAG(
    "datasetalias_triggered_dag",
    description="A DAG that should be triggered via Dataset alias",
    start_date=datetime(2024, 9, 1),
    schedule=[AssetAlias(name="basic_cosmos_dag__orders__run")],
) as dag:

    t3 = EmptyOperator(
        task_id="task_3",
    )

    t3

[netlify]

✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.

Name	Link
🔨 Latest commit	ea1919f
🔍 Latest deploy log	https://app.netlify.com/projects/sunny-pastelito-5ecb04/deploys/68ee13ab3949f000086f0815

[Copilot]

Pull Request Overview

This PR improves the dataset/asset experience in Cosmos by making openlineage-integration-common a required dependency while clarifying that openlineage-airflow or apache-airflow-providers-openlineage are only needed for emitting OpenLineage events, not for Airflow Datasets/Assets functionality.

• Added openlineage-integration-common as a core dependency in pyproject.toml
• Refactored import logic to separate OpenLineage artifact processing from OpenLineage event emission
• Updated variable names and comments to better distinguish between OpenLineage common functionality and Airflow provider functionality

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
pyproject.toml	Added openlineage-integration-common as required dependency
cosmos/operators/local.py	Refactored OpenLineage imports and clarified distinction between artifact processing and event emission

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[codecov]

Codecov Report

❌ Patch coverage is 62.50000% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 97.79%. Comparing base (8436351) to head (ea1919f).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
cosmos/operators/local.py	62.50%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2030      +/-   ##
==========================================
- Coverage   97.84%   97.79%   -0.06%     
==========================================
  Files          89       89              
  Lines        5624     5623       -1     
==========================================
- Hits         5503     5499       -4     
- Misses        121      124       +3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[tatiana]

BTW: The lines not covered by tests were already not covered by tests before.

[pankajastro]

I’ve posted a few quick questions; otherwise, everything looks good. Thanks!