Missing support for variable interpolation in .env files

[hashkool]

podman-compose deviates from docker-compose in

• that it does not interpolate strings in .env files, and
• that it does not know about host env vars in .env (like $HOME, $PWD etc)

To Reproduce

FILE: .env

A=bar
B=foo$A
C=$HOME/foo
D="${XDG_DATA_HOME:-$HOME/.local/share}/blabla"

FILE: docker-compose.yml

version: "4"
services:
    variables:
        image: busybox
        command: ["/bin/busybox", "sh", "-c", "export | grep EXAMPLE"]
        environment:
            EXAMPLE_SIMPLE: "$B"
            EXAMPLE_HOSTVAR_SIMPLE: $C
            EXAMPLE_HOSTVAR_COMPLEX: $D

Commands

$ podman-compose config

Expected behavior
All the variables should be substituted as you would expect. Note that docker-compose config just works as expected.
Also no need to explicitly pass around variables with -e from when invoking docker-compose.

Actual behavior
Actual behaviour is that $B evaluates to literally foo$A, meaning it doesn't interpolate its value.
$C and $D results in errors, as podman-compose doesn't know about $HOME at all.

Output

$ podman-compose version
podman-compose version: 1.0.6
['podman', '--version', '']
using podman version: 4.5.0
podman-compose version 1.0.6
podman --version 
podman version 4.5.0

Environment:
Linux 6.1.31-2

Additional context

Current tests in repo do not test equality of output between docker-compose and podman-compose. That would possibly make your life easier to keep feature parity.

[herzenschein]

I can confirm this. It's easy to reproduce with the docker.io/postgres image and a sample env file like this:

DB_NAME=databasename
DB_USER=databaseuser
DB_PASS=databasepassword

POSTGRES_DB=${DB_NAME}
POSTGRES_USER=${DB_USER}
POSTGRES_PASSWORD=${DB_PASS}

Something like this causes postgres to fail with the following error in podman-compose logs:

FATAL:  invalid character in extension owner: must not contain any of ""$'\"

This issue also seems very similar to #721, they might be duplicates.

Meanwhile, using the following in the compose.yaml with --env-file works just fine.

    environment:
      POSTGRES_DB: ${DB_NAME}
      POSTGRES_USER: ${DB_USER}
      POSTGRES_PASSWORD: ${DB_PASS}

[Franbo]

I'm also seeing that complex variables from the .env file like ${SAMPLE:-$PWD} also do not get the $PWD portion resolved in the yaml compose file, such as for volume mounting.

[meanderix]

AFAICT the .env-file is processed in two ways:

1. Using the python-dotenv-library (which supports expansions within the .env-file; i.e. any expansion can be reused within compose.yaml).
2. As an --env-file argument passed to podman when starting the pods (this will not resolve any inline expansions within the .env-file, so a work-around is to manually do this in compose.yaml as stated above).

[cyqsimon]

I'm also seeing that complex variables from the .env file like ${SAMPLE:-$PWD} also do not get the $PWD portion resolved in the yaml compose file, such as for volume mounting.

Ran into this a moment ago. Didn't run into exactly this, but the equivalent thing in compose.yml where environment variable replacement is already supported. Wanted to write something like this - ${OVERRIDE_DIR:-${HOME}/some/dir}:/other/dir:ro,z but it was only interpolated once into - ${OVERRIDE_DIR:-/home/me/some/dir}:/other/dir:ro,z.

I took a look at the source code, and it seems like there is no recursion to handle nested interpolation (which is part of the spec):

podman-compose/podman_compose.py

Lines 267 to 282 in d704622

	elif is_str(value):
	def convert(m):
	if m.group("escaped") is not None:
	return "$"
	name = m.group("named") or m.group("braced")
	value = subs_dict.get(name)
	if value == "" and m.group("empty"):
	value = None
	if value is not None:
	return str(value)
	if m.group("err") is not None:
	raise RuntimeError(m.group("err"))
	return m.group("default") or ""
	value = var_re.sub(convert, value)

Should be a very easy fix though; will submit a PR in these few days.

Edit 1: After some initial experimentation, it seems like I was overly optimistic in assessing the difficulty of this task. It's pretty evident to me now that it would be practically impossible to implement a reliable nested interpolation parser without bringing in a proper lexer dependency (or worse, raw dogging one ourselves). Regex simply isn't cutting it. I'll continue to work on it but don't expect anything quick.

Edit 2: After looking away for a few days and coming back in a less sleep-deprived state, I am finding the problem slightly less intimidating but still quite tricky. I think I will be able to make a regex-based POC but there are some unresolved questions, mainly regarding ambiguities in the interpolation spec. We can discuss them in the PR.

Edit 3: While writing the PR comments, the more I wrote the more problems I found, and it is quickly spiralling out of control. Recursive string replacement is just not going to be nearly good enough, because fundamentally we cannot distinguish between a lexical token (e.g. $$) and a concatenated string (e.g. $$${FOO} where ${FOO} is $ -> $$). And if we start doing structured replacements, we might as well be writing an AST (which I'm certainly not doing in Python). So I guess I won't be submitting a PR. Sorry. ¯\_(ツ)_/¯

[bcspragu]

I hit this today with something like:

environment:
  - DATABASE_USER=${DATABASE_USER:-postgres}
  - DATABASE_URL=${DATABASE_URL:-postgresql://$DATABASE_URL:somepassword@host/database}

No variation of the last DATABASE_URL (eg. $DATABASE_URL or ${DATABASE_URL}) gets resolved, it always stays as a literal postgresql://$DATABASE_URL:somepassword@host/database