How to use a custom virtual environment with

[gmyrianthous]

I have a custom virtual environment with two dependencies (and no airflow dependency installed in it), that I would like to use along with ExternalPythonOperator however I am unsure what argument to pass to the python argument. Can someone shed some light on this?

from airflow.operators.python import ExternalPythonOperator
... 
    t = ExternalPythonOperator(
        task_id='test_external_venv',
        python='PATH_TO_?',
        expect_airflow=False,
        python_callable=my_python_function,
    )

Assuming the virtual environment is placed under /home/dbt-airflow-venv/, should the python argument point to python (i.e. '/home/dbt-airflow-venv/bin/python3') or somewhere else?

[Taragolis]

Yep it should be path to python executable in venv, for reference ExternalPythonOperator

[Taragolis]

I'm not familiar with Managed Airflow services and the way how to created venv into them.
Are you create this virtual environment locally first and somehow copy them? If so

• did you use same Operation system and architecture (X86_64 vs ARM64)?
• Are you create without symlink, e.g. you should use --copies flag, otherwise instead of actual binary to python you would have symlink

[heidi-creo]

Hi, is there any resolution here? I'm also interested in using an ExternalPythonOperator in Cloud Composer, and I'm not sure how it would be possible to install the virtual environment and make it available to Cloud Composer

[gmyrianthous]

My suggestion would be to use KubernetesPodOperator instead

[micahjsmith]

@gmyrianthous python virtualenvs are not portable so you can't simply upload it to GCS and expect it to work on the cloud composer worker (assuming this is what you did)

[potiuk]

Also - it's more of a question to Cloud Composer team than here - you should open a ticket there. Ideally there should be a way to add a way to build a custom image where you could add installation of such venv, but how to do it , and whether it's possible, is entirely up to a deployment of Airflow.