Fix `test_integration_virtualenv_operator` by pankajastro · Pull Request #1718 · astronomer/astronomer-cosmos

pankajastro · 2025-04-28T15:36:58Z

closes: #1707

For AF3, it appears that the render_template is attempting to resolve the virtualenv_dir templated field path, which leads to an error during execution.

>\nERROR    airflow.sdk.definitions._internal.abstractoperator:abstractoperator.py:382 Exception rendering Jinja template for task \'seed\', field \'virtualenv_dir\'. Template: PosixPath(\'/tmp/persistent-venv2\')\nTraceback (most recent call last):\n  File "/home/runner/.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.10-3.0-1.9/lib/python3.10/site-packages/airflow/sdk/definitions/_internal/abstractoperator.py", line 379, in _do_render_template_fields\n    rendered_content = self.render_template(value, context, jinja_env, seen_oids)\n  File "/home/runner/.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.10-3.0-1.9/lib/python3.10/site-packages/airflow/sdk/definitions/_internal/templater.py", line 182, in render_template\n    return resolve(context)\n  File "/opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/pathlib.py", line 1077, in resolve\n    s = self._accessor.realpath(self, strict=strict)\n  File "/opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/posixpath.py", line 396, in realpath\n    path, ok = _joinrealpath(filename[:0], filename, strict, {})\n  File "/opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/posixpath.py", line 431, in _joinrealpath\n    st = os.lstat(newpath)\nFileNotFoundError: [Errno 2] No such file or directory: \'/tmp/persistent-venv2\'\nERROR    airflow.task:taskinstance.py:2457 Task failed with exception\nTraceback (most recent call last):\n  File "/home/runner/.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.10-3.0-1.9/lib/python3.10/site-packages/airflow/models/taskinstance.py", line 1728, in _run_raw_task\n    TaskInstance._execute_task_with_callbacks(\n  File "/home/runner/.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.10-3.0-1.9/lib/python3.10/site-packages/airflow/models/taskinstance.py", line 2028, in _execute_task_with_callbacks\n    task_orig = self.render_templates(context=context, jinja_env=jinja_env)\n  File "/home/runner/.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.10-3.0-1.9/lib/python3.10/site-packages/airflow/models/taskinstance.py", line 2840, in render_templates\n    original_task.render_template_fields(context, jinja_env)\n  File "/home/runner/.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.10-3.0-1.9/lib/python3.10/site-packages/airflow/sdk/bases/operator.py", line 1558, in render_template_fields\n    self._do_render_template_fields(self, self.template_fields, context, jinja_env, set())\n  File "/home/runner/.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.10-3.0-1.9/lib/python3.10/site-packages/airflow/sdk/definitions/_internal/abstractoperator.py", line 379, in _do_render_template_fields\n    rendered_content = self.render_template(value, context, jinja_env, seen_oids)\n  File "/home/runner/.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.10-3.0-1.9/lib/python3.10/site-packages/airflow/sdk/definitions/_internal/templater.py", line 182, in render_template\n    return resolve(context)\n  File "/opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/pathlib.py", line 1077, in resolve\n    s = self._accessor.realpath(self, strict=strict)\n  File "/opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/posixpath.py", line 396, in realpath\n    path, ok = _joinrealpath(filename[:0], filename, strict, {})\n  File "/opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/posixpath.py", line 431, in _joinrealpath\n    st = os.lstat(newpath)\nFileNotFoundError: [Errno 2] No such file or

This PR introduces a pre-task step to generate the virtualenv_dir value before it's used in the template rendering process, preventing the resolution error.

netlify · 2025-04-28T15:37:16Z

✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.

Name	Link
🔨 Latest commit	`ab82177`
🔍 Latest deploy log	https://app.netlify.com/sites/sunny-pastelito-5ecb04/deploys/680fbb8a125f6f0008df426c

cloudflare-workers-and-pages · 2025-04-28T15:37:48Z

Deploying astronomer-cosmos with Cloudflare Pages

Latest commit:	`ab82177`
Status:	✅ Deploy successful!
Preview URL:	https://e3b01ecc.astronomer-cosmos.pages.dev
Branch Preview URL:	https://fix-1707.astronomer-cosmos.pages.dev

View logs

Remove template

codecov · 2025-04-28T17:57:21Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 97.50%. Comparing base (f9bdcc1) to head (ab82177).
Report is 3 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1718   +/-   ##
=======================================
  Coverage   97.50%   97.50%           
=======================================
  Files          83       83           
  Lines        5049     5049           
=======================================
  Hits         4923     4923           
  Misses        126      126

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

pankajkoti · 2025-04-29T06:05:27Z

 )

 with DAG("example_virtualenv_mini", start_date=datetime(2022, 1, 1)) as dag:
+    pre = BashOperator(task_id="pre", bash_command="mkdir -p /tmp/persistent-venv2")


I know this is still is draft, but taking an early review. I don't think this could be the right approach. Ideally DbtSeedVirtualenvOperator itself should create this path.

also, what if we try "/tmp/persistent-venv2" instead of Path("/tmp/persistent-venv2") in the argument value for the operator? does it have an effect in case of Airflow 3 and help with the rendering and at the same the operator execution is not impacted?

We can reflect that in our docs that the type needs to be a string and not Path.

I think from a rendering point of view it should work, but type for virtualenv_dir is Path or None

https://github.com/astronomer/astronomer-cosmos/blob/main/cosmos/operators/virtualenv.py#L79C9-L79C23

I have created #1721 with your suggestion if CI is happy we can close this PR and merge #1721

yes, the CI was unhappy, but I pushed a commit 1f1947c that seems to help

tatiana · 2025-04-29T08:10:55Z

As discussed in https://github.com/astronomer/astronomer-cosmos/pull/1718/files#r2065555001, I agree that we should implement the solution as part of the operator itself, rather than expecting end-users to create a separate task to create the directory. The other PR #1721 looks more promising, even though it still needs more work. I'm therefore closing this PR for now.

pankajastro temporarily deployed to internal April 28, 2025 15:37 — with GitHub Actions Inactive

pankajastro force-pushed the fix-1707 branch from f66a9c4 to c0f48c1 Compare April 28, 2025 15:37

pankajastro temporarily deployed to internal April 28, 2025 15:38 — with GitHub Actions Inactive

pankajastro temporarily deployed to internal April 28, 2025 17:30 — with GitHub Actions Inactive

pankajastro added 2 commits April 28, 2025 23:01

Fix test_integration_virtualenv_operator

8509943

Remove template

Fix command

ab82177

pankajastro force-pushed the fix-1707 branch from ce11e22 to ab82177 Compare April 28, 2025 17:31

pankajastro temporarily deployed to internal April 28, 2025 17:32 — with GitHub Actions Inactive

pankajastro marked this pull request as ready for review April 29, 2025 06:02

pankajkoti reviewed Apr 29, 2025

View reviewed changes

pankajkoti mentioned this pull request Apr 29, 2025

Change type of virtualenv_dir argument from path to string #1721

Closed

tatiana closed this Apr 29, 2025

pankajastro deleted the fix-1707 branch May 4, 2025 06:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix `test_integration_virtualenv_operator`#1718

Fix `test_integration_virtualenv_operator`#1718
pankajastro wants to merge 2 commits into
mainfrom
fix-1707

pankajastro commented Apr 28, 2025 •

edited

Loading

Uh oh!

netlify Bot commented Apr 28, 2025 •

edited

Loading

Uh oh!

cloudflare-workers-and-pages Bot commented Apr 28, 2025 •

edited

Loading

Uh oh!

codecov Bot commented Apr 28, 2025 •

edited

Loading

Uh oh!

pankajkoti Apr 29, 2025 •

edited

Loading

Uh oh!

pankajkoti Apr 29, 2025 •

edited

Loading

Uh oh!

pankajastro Apr 29, 2025

Uh oh!

pankajastro Apr 29, 2025

Uh oh!

pankajastro Apr 29, 2025

Uh oh!

pankajkoti Apr 29, 2025

Uh oh!

tatiana commented Apr 29, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

pankajastro commented Apr 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

netlify Bot commented Apr 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.

Uh oh!

cloudflare-workers-and-pages Bot commented Apr 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploying astronomer-cosmos with Cloudflare Pages

Uh oh!

codecov Bot commented Apr 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

pankajkoti Apr 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pankajkoti Apr 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pankajastro Apr 29, 2025

Choose a reason for hiding this comment

Uh oh!

pankajastro Apr 29, 2025

Choose a reason for hiding this comment

Uh oh!

pankajastro Apr 29, 2025

Choose a reason for hiding this comment

Uh oh!

pankajkoti Apr 29, 2025

Choose a reason for hiding this comment

Uh oh!

tatiana commented Apr 29, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

pankajastro commented Apr 28, 2025 •

edited

Loading

netlify Bot commented Apr 28, 2025 •

edited

Loading

cloudflare-workers-and-pages Bot commented Apr 28, 2025 •

edited

Loading

codecov Bot commented Apr 28, 2025 •

edited

Loading

pankajkoti Apr 29, 2025 •

edited

Loading

pankajkoti Apr 29, 2025 •

edited

Loading