Support default values for configuration environment variables

[Mario-Hofstaetter]

Is your feature request related to a problem? Please describe.

When using Configuration Environment Variables,

if a variable is NOT SET on otelcol start, the value will be empty and probably break configuration.

Please provide a way to set default values in case ENV variables are not available.

Describe the solution you'd like

Allow definition of a default value if an ENV variable is not set. Via ${VAR:default_value} , example:

processors:
  attributes/example:
    actions:
      - key: "${DB_KEY:CustomerDefaultKey}"
        action: "${OPERATION:CustomerDefaultOperation}"

This mimics the syntax of grafana promtail

Another example is docker-compose , where "Default values can be defined inline using typical shell syntax"
https://github.com/docker/docker.github.io/blob/db650b4772f605683ba4143d0b0de44d0a187db4/compose/compose-file/index.md#interpolation

• ${VARIABLE:-default} evaluates to default if VARIABLE is unset or
  empty in the environment.
• ${VARIABLE-default} evaluates to default only if VARIABLE is unset
  in the environment.

and more, which is more complex however.

Describe alternatives you've considered

For example, grafana allows to Override configuration with environment variables

For OTEL Collector, this could look like this. Given this configuration file (section):

exporters:
  jaeger:
    endpoint: my_default_server:14250

Providing an environment variable crafted like so:

export OTELCOL_EXPORTERS_JAEGER_ENDPOINT=some_other_server:23456

Could overwrite the value from default configuration or configuration files.

(Grafana is supporting even more variable substituation ways, but this is certainly overkill for OTEL Collector.)

At the moment, the configuration file (including ENV vars within the file) is the only way to configure OTEL Collector (?), which may be a deliberate design choice. It is not possible to overwrite specific configuration keys by using:

• CLI Flags
• ENV variables directly addressing those keys (so described approach above)

Which is not bad by all means, since mixing all those possiblities creates complex and confusing configuration.

However ENV variables and CLI Flags are easier to use when running OtelCol as a container, in comparison to a config file. When not running on kubernetes opentelemetry-operator is no alternative.

[bogdandrutu]

It is not possible to overwrite specific configuration keys by using:

This is wrong, look into "yaml:" provider (https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/mapprovider/yamlmapprovider/mapprovider.go) you can do something like:

--config=file:my_file --config="yaml:service::telemetry::metrics::address: localhost:8000"

Or using the --set flag.

Allow definition of a default value if an ENV variable is not set. Via ${VAR:default_value}

We will follow the same model as defined by our providers so will be something like ${env:FOO?default=bar} or something like that. I would like to see a PR that changes the envmapprovider to support default value.

[jpkrohling]

This is wrong, look into "yaml:" provider

Is this documented somewhere? This does deserve to be available in user-facing docs. I can write that doc if we have readme files available for all the supported providers.

[neelayu]

@bogdandrutu

We will follow the same model as defined by our providers so will be something like ${env:FOO?default=bar} or something like that. I would like to see a PR that changes the envmapprovider to support default value.

Correct me if I am wrong, but the envmapprovider is basically a replacement for file or yaml map providers, i.e you can have a partial or full configuration stored in an environment variable which you can specify in the --config flag.
What the need is to support the the substitution in expandmapconverter, similar to what we had in #4458

I believe these are two different provisions.
If you agree, I can re-open the previous PR to fit this use-case.

[bogdandrutu]

We are very close to propose for merging #4742 which will use the same providers also inside the config yaml.

[DewaldDeJager]

We are very close to propose for merging #4742 which will use the same providers also inside the config yaml.

I see this PR was merged. Does that mean we can now use this syntax for default values? ${env:FOO?default=bar}

CC @mx-psi

[mx-psi]

We are very close to propose for merging #4742 which will use the same providers also inside the config yaml.

I see this PR was merged. Does that mean we can now use this syntax for default values? ${env:FOO?default=bar}

CC @mx-psi

No, this only means that you can use ${env:FOO} instead of the legacy ${FOO}. We don't support the default= syntax currently.

[ringerc]

See #2534 (comment) for a workaround

It looks like someone did a draft PR at #4458 but it got stalled.

[systems1]

$FOO and ${FOO}` both do not work (exporters)

if they are removed or never included

and ${env:FOO} doesnt work in processors
but $FOO and ${FOO} both work in processors
processors:
attributes:
actions:
- action: insert
key: env
#value: dev # works in all
value: ${env:DEPLOY_ENV} # doesnt work
#value: $DEPLOY_ENV # works
#value: "${DEPLOY_ENV}" # works

tested with otel-col-contriub 0.91.0 and 0.72.0

No, this only means that you can use ${env:FOO} instead of the legacy ${FOO}. We don't support the default= syntax currently.

[mx-psi]

@systems1 this seems like a separate issue, can you report this on a new issue and post a full configuration example that reproduces the issue?

[yurishkuro]

The ticket is 2+ years old, and there is no path forward, #4458 with shell syntax was not accepted.

Is the consensus to go with ${env:VAR?default=xyz} syntax?

[mx-psi]

@yurishkuro The OpenTelemetry Configuration WG now admits the shell syntax for the SDK configuration files. Since we aim to be aligned with them, it makes sense in my opinion to add support in the Collector for this as well.

[jvoravong]

I’d like to bump prioritization for this ticket due to the recent changes introduced in issue #9532. The 9532 change, introduced via a feature gate, breaks my previous usage of optional environment variables in collector configurations.

Before #9532:
Optional env var OTEL_MEMORY_LIMIT_MIB could be left empty, and limit_mib would default to an appropriate value.

memory_limiter:
  check_interval: 2s
  limit_mib: ${OTEL_MEMORY_LIMIT_MIB}

After #9532:
If env var OTEL_MEMORY_LIMIT_MIB is left empty the following error occurs due to stricter type checking:

Error: failed to get config: cannot unmarshal the configuration: 1 error(s) decoding:
error decoding 'processors': error reading configuration for "memory_limiter": 1 error(s) decoding:
'limit_mib' expected type 'uint32', got unconvertible type 'string', value: '460'
Hint: Temporarily restore the previous behavior by disabling 
      the `confmap.strictlyTypedInput` feature gate. More details at:
      https://github.com/open-telemetry/opentelemetry-collector/issues/10552

To work around this, I would like to use default values for configuration environment variables as proposed in this ticket. I would use env var OTEL_MEMORY_LIMIT_MIB in my configuration like such:

memory_limiter:
  check_interval: 2s
  limit_mib: ${OTEL_MEMORY_LIMIT_MIB:-512}

This will help avoid breaking changes while retaining previous functionality.

[mx-psi]

@jvoravong I don't get the same error as you do. With this configuration file:

Config file (click to expand)

receivers:
  nop:

exporters:
  nop:

processors:
  memory_limiter:
    check_interval: 2s
    limit_mib: ${OTEL_MEMORY_LIMIT_MIB}

service:
  pipelines:
    metrics:
        receivers: [nop]
        exporters: [nop]
        processors: [memory_limiter]

I get the following error:

2024-07-24T16:24:25.375+0200	warn	[email protected]/provider.go:51	Configuration references unset environment variable	{"name": "OTEL_MEMORY_LIMIT_MIB"}
Error: invalid configuration: processors::memory_limiter: 'limit_mib' or 'limit_percentage' must be greater than zero
2024/07/24 16:24:25 collector server run finished with error: invalid configuration: processors::memory_limiter: 'limit_mib' or 'limit_percentage' must be greater than zero

Which I believe is the same error you would have gotten before the feature gate.

Could you give more details about your setup? Are you using multiple configuration files? Are you using a Helm chart?

[cforce]

Currently Otle colector yaml file do not support default values like

${OTEL_MEMORY_LIMIT_MIB:-512}
only they support ev vars without default
${OTEL_MEMORY_LIMIT_MIB}

Please support default values

[cforce]

Highly appreciated feature . Tx a lot @yurishkuro @mx-psi for contributing 👏