Fix broken CI due to Pydantic conflicts

[tatiana]

Pydantic recently released 2.11.7 (Jun 14, 2025), which seems to have broken a few of our test cases, including the following integration tests:

• Async tests
• Kubernetes
• Performance tests

As of now, dbt-core versions until 1.10 depend on Pydantic < 2.0 (https://github.com/dbt-labs/dbt-core/blob/8873581c5afd44196c229d2fbf92880b164652ef/core/setup.py#L85), while Airflow 2.10, 2.11 and 3.0 rely on Pydantic > 2.0. Although these are incompatible, until Pydantic==2.11.0, both libraries could co-exist in the same Python virtual environment without significant issues. However, from this version, numerous issues began to occur, as illustrated below.

For now, we're pinning to a version of Pydantic that works with both libraries, as illustrated in this successful Github run:
https://github.com/astronomer/astronomer-cosmos/actions/runs/15755062499

Follow-up tickets:

• Refactor test installation scripts & validate versions by the end of all of them #1816
• Fix CI Run-Integration-Tests (3.9, 3.0, 1.9) #1817

Error 1

  File "/home/runner/work/astronomer-cosmos/astronomer-cosmos/cosmos/dbt/graph.py", line 216, in run_command_with_subprocess
    raise CosmosLoadDbtException(f"Unable to run {command} due to the error:\n{details}")
cosmos.dbt.graph.CosmosLoadDbtException: Unable to run ['/home/runner/.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.11-3.0-1.9/bin/dbt', 'deps', '--project-dir', '/tmp/tmpb6u3jz1n', '--profiles-dir', '/tmp/tmpw1ykmjqx', '--profile', 'default', '--target', 'dev'] due to the error:
stderr: Traceback (most recent call last):
  File "/home/runner/.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.11-3.0-1.9/bin/dbt", line 5, in <module>
    from dbt.cli.main import cli
  File "/home/runner/.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.11-3.0-1.9/lib/python3.11/site-packages/dbt/cli/__init__.py", line 1, in <module>
    from .main import cli as dbt_cli  # noqa
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.11-3.0-1.9/lib/python3.11/site-packages/dbt/cli/main.py", line 15, in <module>
    from dbt.cli import requires
  File "/home/runner/.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.11-3.0-1.9/lib/python3.11/site-packages/dbt/cli/requires.py", line 34, in <module>
    from dbt.parser.manifest import parse_manifest
  File "/home/runner/.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.11-3.0-1.9/lib/python3.11/site-packages/dbt/parser/manifest.py", line 118, in <module>
    from dbt.utils.artifact_upload import add_artifact_produced
  File "/home/runner/.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.11-3.0-1.9/lib/python3.11/site-packages/dbt/utils/artifact_upload.py", line 6, in <module>
    from pydantic import BaseSettings
  File "/home/runner/.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.11-3.0-1.9/lib/python3.11/site-packages/pydantic/__init__.py", line 426, in __getattr__
    return _getattr_migration(attr_name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.11-3.0-1.9/lib/python3.11/site-packages/pydantic/_migration.py", line 296, in wrapper
    raise PydanticImportError(
pydantic.errors.PydanticImportError: `BaseSettings` has been moved to the `pydantic-settings` package. See https://docs.pydantic.dev/2.11/migration/#basesettings-has-moved-to-pydantic-settings for more details.

For further information visit https://errors.pydantic.dev/2.11/u/import-error

This error was happening because we were installing dbt-core==1.10, instead of dbt-core==1.9, and it had a conflicting version of Pydantic about Airflow 2.9 and 3.0.

"Error 2"

____________________ ERROR collecting tests/test_version.py ____________________
tests/test_version.py:7: in <module>
    import cosmos
cosmos/__init__.py:15: in <module>
    from cosmos.airflow.dag import DbtDag
cosmos/airflow/dag.py:9: in <module>
    from airflow.models.dag import DAG
../../../.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.11-2.8-1.9/lib/python3.11/site-packages/airflow/models/dag.py:94: in <module>
    from airflow.jobs.job import run_job
../../../.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.11-2.8-1.9/lib/python3.11/site-packages/airflow/jobs/job.py:35: in <module>
    from airflow.serialization.pydantic.job import JobPydantic
../../../.local/share/hatch/env/virtual/astronomer-cosmos/Za_bFbg4/tests.py3.11-2.8-1.9/lib/python3.11/site-packages/airflow/serialization/pydantic/job.py:33: in <module>
    class JobPydantic(BaseModelPydantic):
pydantic/main.py:221: in pydantic.main.ModelMetaclass.__new__
    ???
pydantic/fields.py:504: in pydantic.fields.ModelField.infer
    ???
pydantic/fields.py:434: in pydantic.fields.ModelField.__init__
    ???
pydantic/fields.py:544: in pydantic.fields.ModelField.prepare
    ???
pydantic/fields.py:568: in pydantic.fields.ModelField._set_default_and_type
    ???
pydantic/fields.py:437: in pydantic.fields.ModelField.get_default
    ???
pydantic/utils.py:694: in pydantic.utils.smart_deepcopy
    ???
/opt/hostedtoolcache/Python/3.11.13/x64/lib/python3.11/copy.py:172: in deepcopy
    y = _reconstruct(x, memo, *rv)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
/opt/hostedtoolcache/Python/3.11.13/x64/lib/python3.11/copy.py:271: in _reconstruct
    state = deepcopy(state, memo)
            ^^^^^^^^^^^^^^^^^^^^^
/opt/hostedtoolcache/Python/3.11.13/x64/lib/python3.11/copy.py:146: in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
/opt/hostedtoolcache/Python/3.11.13/x64/lib/python3.11/copy.py:231: in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
                             ^^^^^^^^^^^^^^^^^^^^^
/opt/hostedtoolcache/Python/3.11.13/x64/lib/python3.11/copy.py:161: in deepcopy
    rv = reductor(4)
         ^^^^^^^^^^^
E   TypeError: cannot pickle '_thread.RLock' object
=========================== short test summary info ============================
ERROR tests/airflow/test_graph.py - TypeError: cannot pickle '_thread.RLock' object
ERROR tests/dbt/parser/test_output.py - TypeError: cannot pickle '_thread.RLock' object
(...)!!!!!!!!!!!!!!!!!!! Interrupted: 60 errors during collection !!!!!!!!!!!!!!!!!!!`

[netlify]

✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.

Name	Link
🔨 Latest commit	0e21a5a
🔍 Latest deploy log	https://app.netlify.com/projects/sunny-pastelito-5ecb04/deploys/6854180ef93462000825109d

[tatiana]

Since tests passed in https://github.com/astronomer/astronomer-cosmos/actions/runs/15759300044 and the comments were addressed, we're going ahead to merge this