Solve circular import error in Astro runtime while debugging by tatiana · Pull Request #2538 · astronomer/astronomer-cosmos

tatiana · 2026-04-07T20:23:07Z

Problem

When Cosmos is loaded inside Astro Runtime, plugin discovery triggers a circular import that prevents Cosmos from loading:

❯ 2026-04-07T19:17:55.923162Z [error    ] Failed to import plugin cosmos [airflow.plugins_manager] loc=plugins_manager.py:261                                        
Traceback (most recent call last):                                                                                                                                   
  File "/usr/local/lib/python3.12/site-packages/airflow/plugins_manager.py", line 253, in load_entrypoint_plugins                                                    
    plugin_class = entry_point.load()                                                                                                                                
                   ^^^^^^^^^^^^^^^^^^                                                                                                                                
  File "/usr/local/lib/python3.12/importlib/metadata/__init__.py", line 205, in load                                                                                 
    module = import_module(match.group('module'))                                                                                                                    
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                    
  File "/usr/local/lib/python3.12/importlib/__init__.py", line 90, in import_module                                                                                  
    return _bootstrap._gcd_import(name[level:], package, level)                                                                                                      
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                      
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import                                                                                                    
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load                                                                                                 
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked                                                                                        
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked                                                                                                  
  File "<frozen importlib._bootstrap_external>", line 999, in exec_module                                                                                            
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed                                                                                       
  File "/usr/local/lib/python3.12/site-packages/cosmos/plugin/__init__.py", line 21, in <module>                                                                     
    from .airflow3 import CosmosAF3Plugin as CosmosPlugin  # type: ignore[assignment]  # noqa: F401                                                                  
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                            
  File "/usr/local/lib/python3.12/site-packages/cosmos/plugin/airflow3.py", line 22, in <module>                                                                     
    from cosmos import telemetry                                                                                                                                     
  File "/usr/local/lib/python3.12/site-packages/cosmos/telemetry.py", line 16, in <module>                                                                           
    from cosmos.log import get_logger                                                                                                                                
  File "/usr/local/lib/python3.12/site-packages/cosmos/log.py", line 5, in <module>                                                                                  
    from cosmos.settings import rich_logging                                                                                                                         
ImportError: cannot import name 'rich_logging' from partially initialized module 'cosmos.settings' (most likely due to a circular import)                            
(/usr/local/lib/python3.12/site-packages/cosmos/settings.py)

How to reproduce

  docker exec -it $(docker ps -q -f name=scheduler) python -c \                                                                                                      
    "from cosmos.profiles import GoogleCloudServiceAccountDictProfileMapping; \
     m = GoogleCloudServiceAccountDictProfileMapping(conn_id='gcp_gs_conn', \                                                                                        
     profile_args={'dataset': 'astro_prod_cosmos_test'}); print(m.env_vars)"

Root cause

The import chain during plugin discovery is:

  cosmos.plugin.__init__                                                                                                                                             
    → cosmos.plugin.airflow3  (line 26: from cosmos import telemetry)
      → cosmos.telemetry       (line 15: from cosmos import constants, settings)
        → cosmos.settings      (starts executing, defines module-level variables...)                                                                                 
          → cosmos.constants   (imported at line 14 of settings.py — fine)                                                                                           
        → cosmos.log           (line 16: from cosmos.log import get_logger)                                                                                          
          → cosmos.settings    (line 5: from cosmos.settings import rich_logging)                                                                                    
            ✗ cosmos.settings is still being initialized — rich_logging not yet defined

cosmos.telemetry imports both cosmos.settings (line 15) and cosmos.log (line 16) at the module level. When cosmos.settings is imported, it starts executing but hasn't reached the rich_logging = ... assignment (line 35) before cosmos.log tries to import rich_logging from it.

In standard Airflow, this doesn't manifest because the import order completes cosmos.settings before cosmos.log is needed. In Astro Runtime, there is probably some specific code that triggers plugin discovery in a different order than the one that exposes the cycle.

Fix

Move the from cosmos.settings import rich_logging import from module level in cosmos/log.py into the get_logger() function body.

Validation

I validated by cloning and changing:
https://github.com/astronomer/cosmos-demo

So that the requirements.txt installs Cosmos using:

astronomer-cosmos @ https://github.com/astronomer/astronomer-cosmos/archive/refs/heads/fix-astro-runtime-circular-import.zip

cosmos.log imported cosmos.settings.rich_logging at module level, which caused a circular import when Airflow plugin discovery triggered the chain: cosmos.plugin → cosmos.telemetry → cosmos.log → cosmos.settings (still initializing). Move the import inside get_logger() so it only runs when the function is first called, by which time cosmos.settings is fully initialized. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

This PR aims to prevent an Airflow plugin-discovery circular import by deferring cosmos.settings.rich_logging access until get_logger() is called, instead of importing it at cosmos.log module import time.

Changes:

Removed the module-level from cosmos.settings import rich_logging import in cosmos.log.
Added a deferred import of rich_logging inside get_logger().

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

rich_logging is now imported inside get_logger() from cosmos.settings, so tests must monkeypatch it there instead of on the cosmos.log module. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

from cosmos.settings import rich_logging can still fail if get_logger() is called while cosmos.settings is partially initialized (e.g. cosmos.telemetry calls get_logger() at module level during plugin discovery). Use getattr with a False default instead, which safely returns the default when the attribute hasn't been defined yet. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

codecov · 2026-04-07T21:43:12Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.08%. Comparing base (678e508) to head (e251233).
⚠️ Report is 4 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2538   +/-   ##
=======================================
  Coverage   98.08%   98.08%           
=======================================
  Files         103      103           
  Lines        7482     7483    +1     
=======================================
+ Hits         7339     7340    +1     
  Misses        143      143

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Break circular import between cosmos.config and cosmos.profiles by moving the BaseProfileMapping import behind TYPE_CHECKING. The isinstance check in get_profile_type() uses a local import instead. This prevents ImportError when Airflow plugin discovery triggers cosmos.config while cosmos.profiles is still initializing. Guard CosmosRichLogger.handle() against None msg to prevent TypeError when logging None values. Add autouse fixture in conftest.py that demotes CosmosRichLogger instances back to logging.Logger after each test, preventing test_log.py from polluting the logging cache for other tests. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

The try/except ImportError was masking legitimate import errors. Python returns the partially-initialized module during circular imports, so the bare import always succeeds. getattr with a False default is sufficient to handle the case where rich_logging hasn't been defined yet. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

pankajkoti

Does this happen with Cosmos 1.14.0 on runtime images in Astro? If so, do we need a patch release out? I did not observe this when testing Cosmos 1.14.0a8 with runtime images for both 3.1 and 3.2 alpha.

Either way, I think is a good fix, thanks. I am going ahead with the merge, but would be nice to confirm if we need a patch release out with this fix.

tatiana · 2026-04-08T08:35:42Z

@pankajkoti This does not happen when we do astro dev start or astro dev restart. It does happen, though, when we are trying to troubleshoot and attempt to run the commands mentioned in the PR description.

It happens both with Cosmos 1.14.0 and 1.13.1 (I asked @pankajastro to try to reproduce yesterday, and he confirmed he observed it as well) (https://astronomer.slack.com/archives/C06RRC8MFT5/p1775591027977759?thread_ts=1775589526.276809&cid=C06RRC8MFT5))

Since it's been there since at least February 2026 and no one has reported it, we're fine waiting for other bug fixes and shipping it as part of 1.14.1.

@tatiana

Bug Fixes * Fix ``ExecutionMode.WATCHER`` producer retry behaviour by @tatiana in #2559 * Prevent watcher producer skip propagating to downstream tasks via gateway task by @johnhoran and @tatiana in #2597 * Keep watcher sensor polling when producer is still running by @pankajkoti in #2592 * Fix circular import error in Cosmos plugin discovery under Astro Runtime by @tatiana in #2538 * Fix ``CosmosRichLogger`` crash on ``None`` log message by @tatiana in #2540 * Enable inlets and outlets using dbt Fusion on Airflow 3 by @ichirotakami in #2561 * Fix incorrectly skipped source downstream tasks in ``ExecutionMode.WATCHER`` by @pankajastro in #2563 * Fix duplicate logs in ``dbt build`` when source freshness is enabled by @pankajastro in #2564 * Warn and normalize when ``source_rendering_behavior=None`` is passed by @pankajastro in #2570 * Gracefully handle ``Variable.set()`` failures on Astro Remote Execution by @hkc-8010 in #2573 * Skip malformed YAML selectors instead of failing entirely by @YourRoyalLinus in #2577 Docs * Update watcher test behavior docs for Cosmos 1.14.0 by @tatiana in #2549 * Add redirect for moved partial-parsing docs page by @tatiana in #2550 * Document ``ExecutionMode.WATCHER`` and ``depends_on_past`` limitation by @tatiana in #2602 * Restore memory-optimised imports docs for Cosmos < 1.14.0 by @pankajkoti in #2604 Others * Speed up Airflow 3.1+ integration tests by caching InProcessExecutionAPI by @pankajkoti in #2547 * Improve stability of cache hash unit tests by @tatiana in #2539 * Fix mypy 1.20.0 type check failures by @pankajkoti in #2546 * Fix CI failures caused by docs build memory exhaustion by @pankajkoti in #2580 * Fix dbt Fusion broken integration tests by @tatiana in #2581 * Fix flaky ``cosmos_manifest_selectors_example`` DAG in CI by @pankajkoti in #2593 * Reduce pre-commit autoupdate frequency PRs by @tatiana in #2544 * Bump ``reviewdog/action-actionlint`` from 1.71.0 to 1.72.0 by @dependabot in #2542 * Skip watcher gateway test on Airflow 3.0 by @tatiana in #2607 closes: astronomer/oss-integrations-private#381

Copilot AI review requested due to automatic review settings April 7, 2026 20:23

tatiana requested review from a team, corsettigyg, dwreeves and jbandoro as code owners April 7, 2026 20:23

tatiana requested review from pankajastro and pankajkoti April 7, 2026 20:23

tatiana temporarily deployed to internal April 7, 2026 20:23 — with GitHub Actions Inactive

Copilot started reviewing on behalf of tatiana April 7, 2026 20:23 View session

Copilot AI reviewed Apr 7, 2026

View reviewed changes

Comment thread cosmos/log.py Outdated

tatiana and others added 2 commits April 7, 2026 17:32

Fix test_log tests to patch cosmos.settings instead of cosmos.log

c81f171

rich_logging is now imported inside get_logger() from cosmos.settings, so tests must monkeypatch it there instead of on the cosmos.log module. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

tatiana temporarily deployed to internal April 7, 2026 20:46 — with GitHub Actions Inactive

Merge branch 'main' into fix-astro-runtime-circular-import

16d7879

Copilot AI review requested due to automatic review settings April 7, 2026 20:46

tatiana temporarily deployed to internal April 7, 2026 20:47 — with GitHub Actions Inactive

Copilot started reviewing on behalf of tatiana April 7, 2026 20:47 View session

Copilot AI reviewed Apr 7, 2026

View reviewed changes

Comment thread cosmos/log.py Outdated

tatiana temporarily deployed to internal April 7, 2026 23:19 — with GitHub Actions Inactive

tatiana changed the title ~~Defer rich_logging import in cosmos.log to break circular import~~ Solve circular import error in Astro runtime while debugging Apr 7, 2026

Copilot AI review requested due to automatic review settings April 7, 2026 23:32

tatiana temporarily deployed to internal April 7, 2026 23:32 — with GitHub Actions Inactive

Copilot started reviewing on behalf of tatiana April 7, 2026 23:33 View session

Copilot AI reviewed Apr 7, 2026

View reviewed changes

Comment thread cosmos/config.py Outdated

Update cosmos/config.py

e251233

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings April 7, 2026 23:37

tatiana temporarily deployed to internal April 7, 2026 23:38 — with GitHub Actions Inactive

Copilot started reviewing on behalf of tatiana April 7, 2026 23:38 View session

Copilot AI reviewed Apr 7, 2026

View reviewed changes

pankajkoti approved these changes Apr 8, 2026

View reviewed changes

pankajkoti assigned tatiana Apr 8, 2026

pankajkoti merged commit bfe69d4 into main Apr 8, 2026
88 checks passed

pankajkoti deleted the fix-astro-runtime-circular-import branch April 8, 2026 07:11

tatiana added this to the Cosmos 1.14.1 milestone Apr 8, 2026

pankajkoti mentioned this pull request Apr 23, 2026

Release 1.14.1 #2601

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Solve circular import error in Astro runtime while debugging#2538

Solve circular import error in Astro runtime while debugging#2538
pankajkoti merged 7 commits into
mainfrom
fix-astro-runtime-circular-import

tatiana commented Apr 7, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

codecov Bot commented Apr 7, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

pankajkoti left a comment •

edited

Loading

Uh oh!

Uh oh!

tatiana commented Apr 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

tatiana commented Apr 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

codecov Bot commented Apr 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

pankajkoti left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tatiana commented Apr 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

tatiana commented Apr 7, 2026 •

edited

Loading

codecov Bot commented Apr 7, 2026 •

edited

Loading

pankajkoti left a comment •

edited

Loading