Non-Deterministic Field Resolution

[enrico-stauss]

Hi,

While I genuinely like pydantic and pydantic settings for many use-cases, I have been batteling it for a few hour now, just to discover that there seems to be non-deterministic behaviour when resolving fields.
It is a bit hard to put in words, so let's provide the basic schema first:

from pydantic import Field, AliasChoices
from pydantic_settings import BaseSettings, SettingsConfigDict

class NestedSettings(BaseSettings):
    model_config = SettingsConfigDict(
        validate_by_alias=True,
        validate_by_name=True,
        extra="allow",
    )
    user: str = Field(validation_alias=AliasChoices("user", "db_user"))

class AppSettings(BaseSettings):
    model_config = SettingsConfigDict(
        env_file_encoding="utf-8",
        env_nested_delimiter="__",
        env_nested_max_split=2,
    )
    connection: NestedSettings

and here is a corresponding unit test which essentially contains 4 times the same test case:

import os
from schemata import AppSettings

class TestMWE:
    def test_1(self):
        os.environ["DB_USER"] = "inner"
        os.environ["CONNECTION__DB_USER"] = "outer"
        settings = AppSettings(**{"connection": {"user": "explicit"}})
        assert settings.connection.user == "explicit"

    def test_2(self):
        os.environ["DB_USER"] = "inner"
        os.environ["CONNECTION__DB_USER"] = "outer"
        settings = AppSettings(**{"connection": {"user": "explicit"}})
        assert settings.connection.user == "outer"

    def test_3(self):
        os.environ["DB_USER"] = "inner3"
        os.environ["CONNECTION__DB_USER"] = "outer3"
        settings = AppSettings(**{"connection": {"user": "explicit3"}})
        assert settings.connection.user == "explicit3"

    def test_4(self):
        os.environ["DB_USER"] = "inner4"
        os.environ["CONNECTION__DB_USER"] = "outer4"
        settings = AppSettings(**{"connection": {"user": "explicit4"}})
        assert settings.connection.user == "outer4"

Now I execute this test in a docker container with --build --rm to ensure a fresh and clean environment, but what I get is, that the test cases that fail flip from time to time. Here are 2 subsequent executions:

$ docker compose run --build --rm test test_mwe.py
[+] Building 1.8s (13/13) FINISHED                                                                                                           docker:default
=================================================================== test session starts ====================================================================
platform linux -- Python 3.11.13, pytest-8.4.0, pluggy-1.6.0
configfile: pyproject.toml
plugins: anyio-4.9.0
collected 4 items

test_mwe.py .F.F                                                                                                                            [100%]

========================================================================= FAILURES =========================================================================
______________________________________________________________________ TestMWE.test_2 ______________________________________________________________________

self = <py_tests.test_mwe.TestMWE object at 0x74022ba6bd50>

    def test_2(self):
        os.environ["DB_USER"] = "inner"
        os.environ["CONNECTION__DB_USER"] = "outer"
        settings = AppSettings(**{"connection": {"user": "explicit"}})
>       assert settings.connection.user == "outer"
E       AssertionError: assert 'explicit' == 'outer'
E
E         - outer
E         + explicit

py_tests/test_mwe.py:15: AssertionError
______________________________________________________________________ TestMWE.test_4 ______________________________________________________________________

self = <py_tests.test_mwe.TestMWE object at 0x74022ba009d0>

    def test_4(self):
        os.environ["DB_USER"] = "inner4"
        os.environ["CONNECTION__DB_USER"] = "outer4"
        settings = AppSettings(**{"connection": {"user": "explicit4"}})
>       assert settings.connection.user == "outer4"
E       AssertionError: assert 'explicit4' == 'outer4'
E
E         - outer4
E         + explicit4

py_tests/test_mwe.py:27: AssertionError
===================================================================== warnings summary =====================================================================
../usr/local/lib/python3.11/site-packages/pydantic/_internal/_fields.py:198
  /usr/local/lib/python3.11/site-packages/pydantic/_internal/_fields.py:198: UserWarning: Field name "schema" in "DBConnectionSettings" shadows an attribute in parent "BaseSettings"
    warnings.warn(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================================================= short test summary info ==================================================================
FAILED py_tests/test_mwe.py::TestMWE::test_2 - AssertionError: assert 'explicit' == 'outer'
FAILED py_tests/test_mwe.py::TestMWE::test_4 - AssertionError: assert 'explicit4' == 'outer4'
========================================================== 2 failed, 2 passed, 1 warning in 0.73s ==========================================================

$ docker compose run --build --rm test py_tests/test_mwe.py
[+] Building 1.2s (13/13) FINISHED                                                                                                           docker:default
=================================================================== test session starts ====================================================================
platform linux -- Python 3.11.13, pytest-8.4.0, pluggy-1.6.0
configfile: pyproject.toml
plugins: anyio-4.9.0
collected 4 items

py_tests/test_mwe.py F.F.                                                                                                                            [100%]

========================================================================= FAILURES =========================================================================
______________________________________________________________________ TestMWE.test_1 ______________________________________________________________________

self = <py_tests.test_mwe.TestMWE object at 0x70986c834a10>

    def test_1(self):
        os.environ["DB_USER"] = "inner"
        os.environ["CONNECTION__DB_USER"] = "outer"
        settings = AppSettings(**{"connection": {"user": "explicit"}})
>       assert settings.connection.user == "explicit"
E       AssertionError: assert 'outer' == 'explicit'
E
E         - explicit
E         + outer

py_tests/test_mwe.py:9: AssertionError
______________________________________________________________________ TestMWE.test_3 ______________________________________________________________________

self = <py_tests.test_mwe.TestMWE object at 0x70986b19ff90>

    def test_3(self):
        os.environ["DB_USER"] = "inner3"
        os.environ["CONNECTION__DB_USER"] = "outer3"
        settings = AppSettings(**{"connection": {"user": "explicit3"}})
>       assert settings.connection.user == "explicit3"
E       AssertionError: assert 'outer3' == 'explicit3'
E
E         - explicit3
E         + outer3

py_tests/test_mwe.py:21: AssertionError
===================================================================== warnings summary =====================================================================
../usr/local/lib/python3.11/site-packages/pydantic/_internal/_fields.py:198
  /usr/local/lib/python3.11/site-packages/pydantic/_internal/_fields.py:198: UserWarning: Field name "schema" in "DBConnectionSettings" shadows an attribute in parent "BaseSettings"
    warnings.warn(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================================================= short test summary info ==================================================================
FAILED py_tests/test_mwe.py::TestMWE::test_1 - AssertionError: assert 'outer' == 'explicit'
FAILED py_tests/test_mwe.py::TestMWE::test_3 - AssertionError: assert 'outer3' == 'explicit3'
========================================================== 2 failed, 2 passed, 1 warning in 0.68s ==========================================================

As you can see, in the first execution, tests 2 and 4 fail, but in the second, test 1 and 3 fail. Please help me out here, this is driving me crazy :D

Cheers Enrico

[hramezani]

Thanks @enrico-stauss for creating the issue.

I didn't check the issue carefully. But the nested model should inherit from pydantic.BaseModel. so probably your problem can be fixed by changing:

class NestedSettings(BaseModel):

https://docs.pydantic.dev/latest/concepts/pydantic_settings/#parsing-environment-variable-values

Let me know if the problem exists after my suggestion.

[enrico-stauss]

This does seem to resolve the issue with my MWE being non-deterministic. I however relied on nesting BaseSettings instances for a two reasons:

1. Being able to use the inner class standalone
2. Enable resolving from 3 sources, which in my example would be init args -> CONNECTION__DB_USER -> DB_USER

Even if that is not supported by Pydantic, I would have hoped for deterministic behaviour (may it be an error or unexpected behaviour).

Anyway, thanks for the quick clarification!
Kind Regards
Enrico

[hramezani]

I ran the tests with pytest multiple times and the results were the same:

from pydantic import Field, AliasChoices, BaseModel
from pydantic_settings import BaseSettings, SettingsConfigDict
import os

class NestedSettings(BaseModel):
    model_config = SettingsConfigDict(
        validate_by_alias=True,
        validate_by_name=True,
        extra="allow",
    )
    user: str = Field(validation_alias=AliasChoices("user", "db_user"))

class AppSettings(BaseSettings):
    model_config = SettingsConfigDict(
        env_file_encoding="utf-8",
        env_nested_delimiter="__",
        env_nested_max_split=2,
    )
    connection: NestedSettings



def test_1():
    os.environ["DB_USER"] = "inner"
    os.environ["CONNECTION__DB_USER"] = "outer"
    settings = AppSettings(**{"connection": {"user": "explicit"}})
    assert settings.connection.user == "explicit"

def test_2():
    os.environ["DB_USER"] = "inner"
    os.environ["CONNECTION__DB_USER"] = "outer"
    settings = AppSettings(**{"connection": {"user": "explicit"}})
    assert settings.connection.user == "outer"

def test_3():
    os.environ["DB_USER"] = "inner3"
    os.environ["CONNECTION__DB_USER"] = "outer3"
    settings = AppSettings(**{"connection": {"user": "explicit3"}})
    assert settings.connection.user == "explicit3"

def test_4():
    os.environ["DB_USER"] = "inner4"
    os.environ["CONNECTION__DB_USER"] = "outer4"
    settings = AppSettings(**{"connection": {"user": "explicit4"}})
    assert settings.connection.user == "outer4"

┏━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┓
┃  File      ┃  Function  ┃  Function Line  ┃  Error Line  ┃  Error           ┃
┡━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩
│  test8.py  │  test_2    │  30             │  34          │  AssertionError  │
│  test8.py  │  test_4    │  42             │  46          │  AssertionError  │
└────────────┴────────────┴─────────────────┴──────────────┴──────────────────┘

[enrico-stauss]

Yeah, as I said, with the inner model inheriting from BaseModel, the behaviour becomes deterministic.
The 2 inconveniences I listed unfortunately remain.

[hramezani]

Enable resolving from 3 sources, which in my example would be init args -> CONNECTION__DB_USER -> DB_USER

Can you explain more? would be good to represent the problem with a small code or test.

[enrico-stauss]

This should depict my intentions:

from pydantic import Field, AliasChoices, BaseModel
from pydantic_settings import BaseSettings, SettingsConfigDict
import os

class NestedSettings(BaseSettings):
    model_config = SettingsConfigDict(
        validate_by_alias=True,
        validate_by_name=True,
        extra="allow",
    )
    user: str = Field(validation_alias=AliasChoices("user", "inner_user"))


class AppSettings(BaseSettings):
    model_config = SettingsConfigDict(
        env_file_encoding="utf-8",
        env_nested_delimiter="__",
        env_prefix="outer_",
        env_nested_max_split=2,
    )
    conn: NestedSettings = Field(default_factory=NestedSettings)

def test_env_inner():
    os.environ.clear()
    os.environ["INNER_USER"] = "inner"
    settings = AppSettings()
    assert settings.conn.user == "inner"
    
def test_env_outer():
    os.environ.clear()
    os.environ["OUTER_CONN__INNER_USER"] = "outer"
    settings = AppSettings()
    assert settings.conn.user == "outer"

def test_precedence_outer():
    os.environ.clear()
    os.environ["INNER_USER"] = "inner"
    os.environ["OUTER_CONN__INNER_USER"] = "outer"
    settings = AppSettings()
    assert settings.conn.user == "outer"

def test_precedence_explicit():
    os.environ.clear()
    os.environ["INNER_USER"] = "inner"
    os.environ["OUTER_CONN__INNER_USER"] = "outer"
    settings = AppSettings(**{"conn": {"user": "explicit"}})
    assert settings.conn.user == "explicit"

it would now be up to the user to set env>init args precedence per model, but in principle I want to have the clear resolution order, s.t. if a value is provided at the initializer, this takes precedence. If no value is provided at the init, the env var that maps to the top level settings class takes precedence and if no value is provided to the top level settings class, the nested settings class itself tries to resolve a value from the environment. In principle this works, if it were not for the indeterministic behaviour.

If you run my tests a few times, you'll see that ~50% of the times all do pass.

One last quirk. If I change to this:

class NestedSettings(BaseSettings):
    model_config = SettingsConfigDict(
        validate_by_alias=True,
        validate_by_name=True,
        extra="allow",
        env_prefix="inner_"
    )
    user: str

I would have expected it to work identically, but I do get these errors:

__________________________________________________________________________________________________________ test_env_outer ___________________________________________________________________________________________________________ 

    def test_env_outer():
        os.environ.clear()
        os.environ["OUTER_CONN__INNER_USER"] = "outer"
>       settings = AppSettings()

...

E       pydantic_core._pydantic_core.ValidationError: 1 validation error for AppSettings
E       conn.user
E         Field required [type=missing, input_value={'inner_user': 'outer'}, input_type=dict]
E           For further information visit https://errors.pydantic.dev/2.11/v/missing

.venv\lib\site-packages\pydantic_settings\main.py:176: ValidationError
_______________________________________________________________________________________________________ test_precedence_outer _______________________________________________________________________________________________________ 

    def test_precedence_outer():
        os.environ.clear()
        os.environ["INNER_USER"] = "inner"
        os.environ["OUTER_CONN__INNER_USER"] = "outer"
        settings = AppSettings()
>       assert settings.conn.user == "outer"

E       AssertionError: assert 'inner' == 'outer'
E         
E         - outer
E         + inner

which is weird because Field required [type=missing, input_value={'inner_user': 'outer'}, input_type=dict] should be resolved as validate_by_alias=True, validate_by_name=True, are both set.

[hramezani]

it would now be up to the user to set env>init args precedence per model, but in principle I want to have the clear resolution order, s.t. if a value is provided at the initializer, this takes precedence. If no value is provided at the init, the env var that maps to the top level settings class takes precedence and if no value is provided to the top level settings class, the nested settings class itself tries to resolve a value from the environment. In principle this works, if it were not for the indeterministic behaviour.

This seems a custom behavior. pydantic-settings only initializes the parent class( AppSettings in you example) and tries to resolve values for it's field. that's why we suggest to not user BaseSettings in nested classes.

you still can have your custome settings source inherited from env settings source.

take a look at: https://docs.pydantic.dev/latest/concepts/pydantic_settings/#customise-settings-sources

[enrico-stauss]

I understand that you do not advise nesting BaseSettings objects, but this restricts modularity and - as I tried to explain - prohibits the 3rd stage of env resolution in my example. If I change to:

class NestedSettings(BaseModel):
    model_config = SettingsConfigDict(
        validate_by_alias=True,
        validate_by_name=True,
        extra="allow",
    )
    user: str = Field(validation_alias=AliasChoices("user", "inner_user"))

the test_env_inner will always fail.

I can accept that this is a non-supported usecase, but I'd use this opportunity to place a feature request: Allow Nesting BaseSettings` Much necessary functionality seems to be there already, if the non-deterministic behaviour is resolved.

[hramezani]

Yes, I got your point. But unfortunately, we can't support it in V2. nested models have to be used as a nested model, and it is better not to inherit from BaseSettings. You can still inherit from BaseSettings, but you have to care about the potential problems.

I prefer not to accept it as a feature request. But we can keep the issue open for a while to see if other users are interested.

[enrico-stauss]

I took a bit of time to look into the actual pydantic settings code and found the culprit that causes the problem I described. This is in fact not specific to nesting BaseSettings models, but it much rather is a resolution issue of the InitSettingsSource class.

Check out this commit on my fork which introduces a unit test that simply populates two of the defined alias fields simultaneously. Sometimes this will work - and others it won't, because we cannot know for sure that the set.pop() will yield the expected element if multiple values are inside init_kwarg_name.

Now I also added a subsequent commit that provides the fix, but with that fix, another unit test fails. It seems that resolving this issue requires quite a bit of cleanup throughout the SettingsSources implementations.

I'd be happy to submit a PR that fixes the bug, but before investing more time, I'd need feedback if you are interested in that.

Cheers Enrico

[hramezani]

Thanks @enrico-stauss for the investigation.

If the fix is an easy one, we can merge it.

[hramezani]

Fixed in 0497ef2