Setting Nested Value Via Environment Variable Removing All Nested Values

[FancyGecko]

Build: 2.0.0

When I use set_env() and only want to change one nested value via environment variables it is removing all the nested values.

config.yaml

parent_value:
  nested_value1: foo
  nested_value2: bar

code

import confuse

config = confuse.Configuration('app', __name__)
config.set_file("config.yaml")
config.set_env()

print(config['parent_value'].get())

Output: Not setting an environment variable

OrderedDict([('nested_value1', 'foo'), ('nested_value2', 'bar')])

Output: Setting environment variable export APP_PARENT_VALUE__NESTED_VALUE1=newValue

{'nested_value1': 'newValue'}

Expected Outcome:

{'nested_value1': 'newValue', 'nested_value2': 'bar'}

[sampsyo]

Hello! Unfortunately, this is a symptom of a deeper issue having to do with the way that overlays and get() work, which are covered in #143 as well.

Here's a script demonstrating the issue without environment variables:

import confuse

# Setup.
s = confuse.ConfigSource({'a': {'b': 'c', 'd': 'e'}})
r = confuse.RootView([s])
print(r['a'].get())

# Overriding seems to affect the entire `a` dict.
r['a']['b'].set('c1')
print(r['a']['b'].get())
print(r['a'].get())

# However, `d` is still there.
print(r['a']['d'].get())

The output is:

{'b': 'c', 'd': 'e'}
c1
{'b': 'c1'}
e

The upshot is that the other keys in the a dict are still there, but you can't see them by doing r['a'].get(). Code that instead iterates over a, as in for view in r['a'], will still see all the keys.

I totally agree that this behavior is not intuitive, especially for environment variable overrides. It would be great to do something better with this, perhaps with an alternative to get with different "merging" behavior. But that's a tricky conceptual problem…

[arroyoj]

In addition to iterating over the keys in the view like @sampsyo suggested, you could also use a Template with get() to make sure none of the keys are skipped. I'm not sure if this fits your use case, but for example, you could do the following, building on your sample code:

import confuse

config = confuse.Configuration('app', __name__)
config.set_file("config.yaml")
config.set_env()
parent_template = {'nested_value1': str, 'nested_value2': str}

print(config['parent_value'].get())
print(config['parent_value'].get(parent_template))

When a dict is passed to get() as above, it is converted to a MappingTemplate. Using the template also gives you validation. In the example above, the template indicates that the values for those keys must be strings.

Without setting environment variables, the output is

OrderedDict([('nested_value1', 'foo'), ('nested_value2', 'bar')])
{'nested_value1': 'foo', 'nested_value2': 'bar'}

With setting the env var (export APP_PARENT_VALUE__NESTED_VALUE1=newValue), the output when using the template matches what you expected:

{'nested_value1': 'newValue'}
{'nested_value1': 'newValue', 'nested_value2': 'bar'}