python-configuration

A library to load configuration parameters hierarchically from multiple sources and formats

This library is intended as a helper mechanism to load configuration files hierarchically.

Supported Formats

The python-configuration library supports the following configuration formats and sources:

• Python files
• Dictionaries
• Environment variables
• Filesystem paths
• JSON files
• INI files
• dotenv type files
• Optional support for:
  • YAML files: requires yaml
  • TOML files: requires tomli for Python < 3.11
  • Azure Key Vault credentials: requires azure-keyvault
  • AWS Secrets Manager credentials: requires boto3
  • GCP Secret Manager credentials: requires google-cloud-secret-manager
  • Hashicorp Vault credentials: requires hvac

Installing

To install the library:

pip install python-configuration

To include the optional TOML and/or YAML loaders, install the optional dependencies toml and yaml. For example,

pip install python-configuration[toml,yaml]

Without the optional dependencies, the TOML (Python < 3.11) and YAML loaders will not be available, and attempting to use them will raise an exception.

Getting started

python-configuration converts the various config types into dictionaries with dotted-based keys. For example, given this JSON configuration

{
    "a": {
        "b": "value"
    }
}

We can use the config_from_json method to parse it:

from config import config_from_json

cfg = config_from_json("my_config_file.json", read_from_file=True)

(Similar methods exist for all the other supported configuration formats (eg. config_from_toml, etc.).)

We are then able to refer to the parameters in the config above using any of:

cfg['a.b']
cfg['a']['b']
cfg['a'].b
cfg.a.b

and extract specific data types such as dictionaries:

cfg['a'].as_dict == {'b': 'value'}

This is particularly useful in order to isolate group parameters. For example, with the JSON configuration

{
  "database.host": "something",
  "database.port": 12345,
  "database.driver": "name",
  "app.debug": true,
  "app.environment": "development",
  "app.secrets": "super secret",
  "logging": {
    "service": "service",
    "token": "token",
    "tags": "tags"
  }
}

one can retrieve the dictionaries as

cfg.database.as_dict()
cfg.app.as_dict()
cfg.logging.as_dict()

or simply as

dict(cfg.database)
dict(cfg.app)
dict(cfg.logging)

Configuration

There are two general types of objects in this library. The first one is the Configuration, which represents a single config source. The second is a ConfigurationSet that allows for multiple Configuration objects to be specified.

Single Config

Python Files

To load a configuration from a Python module, the config_from_python can be used. The first parameter must be a Python module and can be specified as an absolute path to the Python file or as an importable module.

Optional parameters are the prefix and separator. The following call

config_from_python('foo.bar', prefix='CONFIG', separator='__')

will read every variable in the foo.bar module that starts with CONFIG__ and replace every occurrence of __ with a .. For example,

# foo.bar
CONFIG__AA__BB_C = 1
CONFIG__AA__BB__D = 2
CONF__AA__BB__D = 3

would result in the configuration

{
    'aa.bb_c': 1,
    'aa.bb.d': 2,
}

Note that the single underscore in BB_C is not replaced and the last line is not prefixed by CONFIG.

Dictionaries

Dictionaries are loaded with config_from_dict and are converted internally to a flattened dict.

{
    'a': {
        'b': 'value'
    }
}

becomes

{
    'a.b': 'value'
}

Environment Variables

Environment variables starting with prefix can be read with config_from_env:

config_from_env(prefix, separator='_')

Filesystem Paths

Folders with files named as xxx.yyy.zzz can be loaded with the config_from_path function. This format is useful to load mounted Kubernetes ConfigMaps or Secrets.

JSON, INI, .env, YAML, TOML

JSON, INI, YAML, TOML files are loaded respectively with config_from_json, config_from_ini, config_from_dotenv, config_from_yaml, and config_from_toml. The parameter read_from_file controls whether a string should be interpreted as a filename.

Caveats

In order for Configuration objects to act as dict and allow the syntax dict(cfg), the keys() method is implemented as the typical dict keys. If keys is an element in the configuration cfg then the dict(cfg) call will fail. In that case, it's necessary to use the cfg.as_dict() method to retrieve the dict representation for the Configuration object.

The same applies to the methods values() and items().

Configuration Sets

Configuration sets are used to hierarchically load configurations and merge settings. Sets can be loaded by constructing a ConfigurationSet object directly or using the simplified config function.

To construct a ConfigurationSet, pass in as many of the simple Configuration objects as needed:

cfg = ConfigurationSet(
    config_from_env(prefix=PREFIX),
    config_from_json(path, read_from_file=True),
    config_from_dict(DICT),
)

The example above will read first from Environment variables prefixed with PREFIX, and fallback first to the JSON file at path, and finally use the dictionary DICT.

The config function simplifies loading sets by assuming some defaults. The example above can also be obtained by

cfg = config(
    ('env', PREFIX),
    ('json', path, True),
    ('dict', DICT),
)

or, even simpler if path points to a file with a .json suffix:

cfg = config('env', path, DICT, prefix=PREFIX)

The config function automatically detects the following:

• extension .py for python modules
• dot-separated python identifiers as a python module (e.g. foo.bar)
• extension .json for JSON files
• extension .yaml for YAML files
• extension .toml for TOML files
• extension .ini for INI files
• extension .env for dotenv type files
• filesystem folders as Filesystem Paths
• the strings env or environment for Environment Variables

Merging Values

ConfigurationSet instances are constructed by inspecting each configuration source, taking into account nested dictionaries, and merging at the most granular level. For example, the instance obtained from cfg = config(d1, d2) for the dictionaries below

d1 = {'sub': {'a': 1, 'b': 4}}
d2 = {'sub': {'b': 2, 'c': 3}}

is such that cfg['sub'] equals

{'a': 1, 'b': 4, 'c': 3}

Note that the nested dictionaries of 'sub' in each of d1 and d2 do not overwrite each other, but are merged into a single dictionary with keys from both d1 and d2, giving priority to the values of d1 over those from d2.

Caveats

As long as the data types are consistent across all the configurations that are part of a ConfigurationSet, the behavior should be straightforward. When different configuration objects are specified with competing data types, the first configuration to define the elements sets its datatype. For example, if in the example above element is interpreted as a dict from environment variables, but the JSON file specifies it as anything else besides a mapping, then the JSON value will be dropped automatically.

Other Features

String Interpolation

When setting the interpolate parameter in any Configuration instance, the library will perform a string interpolation step using the str.format syntax. In particular, this allows to format configuration values automatically:

cfg = config_from_dict({
    "percentage": "{val:.3%}",
    "with_sign": "{val:+f}",
    "val": 1.23456,
    }, interpolate=True)

assert cfg.val == 1.23456
assert cfg.with_sign == "+1.234560"
assert cfg.percentage == "123.456%"

Validation

Validation relies on the jsonchema library, which is automatically installed using the extra validation. To use it, call the validate method on any Configuration instance in a manner similar to what is described on the jsonschema library:

schema = {
    "type" : "object",
    "properties" : {
        "price" : {"type" : "number"},
        "name" : {"type" : "string"},
    },
}

cfg = config_from_dict({"name" : "Eggs", "price" : 34.99})
assert cfg.validate(schema)

cfg = config_from_dict({"name" : "Eggs", "price" : "Invalid"})
assert not cfg.validate(schema)

# pass the `raise_on_error` parameter to get the traceback of validation failures
cfg.validate(schema, raise_on_error=True)
# ValidationError: 'Invalid' is not of type 'number'

To use the format feature of the jsonschema library, the extra dependencies must be installed separately as explained in the documentation of jsonschema.

from jsonschema import Draft202012Validator

schema = {
    "type" : "object",
    "properties" : {
        "ip" : {"format" : "ipv4"},
    },
}

cfg = config_from_dict({"ip": "10.0.0.1"})
assert cfg.validate(schema, format_checker=Draft202012Validator.FORMAT_CHECKER)

cfg = config_from_dict({"ip": "10"})
assert not cfg.validate(schema, format_checker=Draft202012Validator.FORMAT_CHECKER)

# with the `raise_on_error` parameter:
c.validate(schema, raise_on_error=True, format_checker=Draft202012Validator.FORMAT_CHECKER)
# ValidationError: '10' is not a 'ipv4'

Extras

The config.contrib package contains extra implementations of the Configuration class used for special cases. Currently the following are implemented:

• AzureKeyVaultConfiguration in config.contrib.azure, which takes Azure Key Vault credentials into a Configuration-compatible instance. To install the needed dependencies execute

  pip install python-configuration[azure]

• AWSSecretsManagerConfiguration in config.contrib.aws, which takes AWS Secrets Manager credentials into a Configuration-compatible instance. To install the needed dependencies execute

  pip install python-configuration[aws]

• GCPSecretManagerConfiguration in config.contrib.gcp, which takes GCP Secret Manager credentials into a Configuration-compatible instance. To install the needed dependencies execute

  pip install python-configuration[gcp]

• HashicorpVaultConfiguration in config.contrib.vault, which takes Hashicorp Vault credentials into a Configuration-compatible instance. To install the needed dependencies execute

  pip install python-configuration[vault]

Features

• Load multiple configuration types
• Hierarchical configuration
• Ability to override with environment variables
• Merge parameters from different configuration types

Contributing

If you'd like to contribute, please fork the repository and use a feature branch. Pull requests are welcome.

See CONTRIBUTING.md for the details.

Links

• Repository: https://github.com/tr11/python-configuration
• Issue tracker: https://github.com/tr11/python-configuration/issues
• Documentation: https://python-configuration.readthedocs.io

Licensing

The code in this project is licensed under MIT license.