Save secret questions answers in seperate file

[traverseda]

Actual Situation

I'm using copier to deploy several docker containers and automatically configure stuff like single-sign-on. I have a secrets_seed setting that I'm setting using secret: true in copier.yml.

The user must put in exactly the same secret_seed value every time or else things will break. This is out of my control as fixing it would require forking the docker containers for several popular projects, which only set secrets inside the container the first time it's run. Cycling secrets is outside of the scope of my project.

_skip_if_exists can be used to reduce the damage from a user entering the wrong secret_seed value, but isn't a perfect solutions either it means I can't add new secrets to an existing secrets.env file. I'd need to make a secrets.env.d file and docker doesn't support globing for loading env files, so it would make the project much messier.

Desired Situation

I'd like to be able to save secrets into a separate yaml answers file that isn't checked into git, so that it can be included in backups but not in the git repo itself. This also reduces the risk of user-error from entering the wrong secrets_seed value.

This separate secrets.yaml file could be stored in backups along-side user data, but not checked into git.

Proposed solution

No response

[sisp]

Just an alternative idea: How about encrypting answers to secret questions and recording them like regular answers in the answers file? We could probably make this an opt-in feature that requires an encryption key (or key pair in case we want to support asymmetric encryption for some reason).

[traverseda]

I don't see much use of encrypting the secret_seed in my particular use case, as I need to store the generated secrets in cleartext to pass them to various services. Anyone who can see the in-use directory structure can see the secrets.

Right now my solutions is a wrapper script, I generate a SECRET_SEED file, and I wrap copier update --data secret_seed=$(cat SECRET_SEED).

I definitely think the encrypted secrets makes sense for a lot of projects, most of them even. Just for my particular use-case, it would complicate things if multiple sysadmin work on the generated project tree, and the secrets need to be stored in cleartext anyway.

[pawamoy]

If Copier was able to read and combine multiple answers files, you could generate the secrets YAML file yourself:

secret1: "{{ secret1 }}"
secret2: "{{ secret2 }}"

Then

copier update --answers-file .copier-answers.yaml --answers-file secrets.yaml

[traverseda]

As an aside, I've done devops for a long time and I think using copier is a really exciting approach that gives the benefits of central project updates while allowing deep customization where it's needed. Beautiful project.

[sisp]

I don't see much use of encrypting the secret_seed in my particular use case, as I need to store the generated secrets in cleartext to pass them to various services. Anyone who can see the in-use directory structure can see the secrets.

The idea behind encrypting secrets is all about allowing them to be versioned with Git as part of the answers file. Thus, you could Git-ignore the encryption key file instead of writing the cleartext secret to a separate answers file. See also, e.g., Sealed Secrets.

About supporting multiple answers files: How would we specify in which file an answer gets recorded? Now I get it, we would simply render a custom YAML file containing the secrets in the answers file format. But supporting multiple answers files might be error-prone because only one file should contain metadata (keys with leading underscore like _src and _commit).

[traverseda]

Yeah, less a full answers file and more a way to chainmap multiple yaml files together. One answers file and we can handle custom integrating on our own.

[yajo]

Are you using raw docker or docker compose?

[yajo]

Thanks everyone for your comments.

In docker compose you can add an extra .env file for the services that need it. That file can be skipped if exists and git-ignored. I've used this for years and it works very well.

In simple docker, you can have a similar result by mounting your separate secrets as a volume and reading the secrets file from your app code, or re-exporting them in the entrypoint.

So, this is not a problem in the copier side, AFACS. You just need to use the right tool for the job.

I hope you can fix your situation with these tips.