Understanding Salt with Hashicorp Vault

[elwood218]

How Vault is integrated into Vault?

According to following documentation I understand this example as all Minions need access to Vault, correct? -> https://salt-extensions.github.io/saltext-vault/topics/basic_configuration.html

But I don't want allow any Minion to access Vault. I want that only the Masters are allowed to access Vault and then I would write pillar files where I access a secret with:

secret: '{{ salt.vault.read_secret("secret/data/foo", key="bar") }}'

But for this I also have configured it like in the basic configuration on the master at least. But I did not put the right to create vault app-roles. For me it should work like that:

1. I put once a secret-id (most at startup or if the service was longer down and could not renew its token)
2. The secret id will than get a token where it is allowed to read certain secrets and then the masters should renew their token all the time until for any reason (like reboot) it cannot renew in a certain time. Then I would put a new secret-id - but which can be only used once.

BUT obviusly it doesn't work like that. At least for me. Can someone explain how that should be done because the documentation unfortunately does not explain it and the different options.

PS: I also would let master read all secrets and then use that as Pillar directly for the minions if that is possible without let each minion access the vault?! But I wonder then how I could tell that this minion is only allowed to read this certain data. With the way above I can at least tell in top.sls.

Kind regards
Mathias

[elwood218]

If it is not working like that I would probably do a workaround and install Vault Agent as proxy and let the masters talk to the local Vault agent with a token which never expires. (at least in theory..)

But maybe it would work because at the moment I still got an error when I try it like menioned above but don't know why. If I use Approle with other Software it works. It tells me following error and I don't know why it would create a token other than I defined in the approle..

salt.exceptions.SaltRenderError: Problem running salt function in Jinja template: Failed to read secret! CommandExecutionError: {'error': 'VaultInvocationError: child policies must be subset of parent'}

As I am quite new to salt I need to check how I can debug that and see what policies are tried..

[dmurphy18]

FYI use the Salt Vault Extension, it is much more up to date, and that is the way Salt is heading with 3008, see https://github.com/salt-extensions/saltext-vault

[elwood218]

@dmurphy18 Thank you. That is exactly what I am referring to. The link of the "basic config" is the one on that github page you posted. But unfortunately I don't understand it completely. I don't see why the master must be able to create approles for the minions.. but I guess this would only be needed if the minion also has access to Vault. That is what I still try to figure out.

[elwood218]

I have debugged the request and found the problem. So it is reading at least now. The problem was that there are default policies which are set. So I have set my policies which also the secret-id had - so it then was a "subset". This had to be set in master config.

vault:
  policies:
    assign:
      - policy1
      - policy2

But I still don't know how to use that in the correct way and how is the workflow and if the token is renewed in the background because I don't want to put a secret-id which runs forever.

And @dmurphy18 you probably wanted to tell me that I shall open an discussion in their github repo because it is an extension and not included anymore, right?! :)

[lkubb]

But I don't want allow any Minion to access Vault. I want that only the Masters are allowed to access Vault

The Vault integration works under the following assumptions:

1. The master orchestrates minion authentication/authorization only, it does not have access to any secrets.
2. Minions request Vault server configuration + authentication credentials and access Vault directly. The master assigns Vault policies (collections of related authorizations you need to define yourself) to these credentials based on the vault:policies:assign configuration, which is usually templated because otherwise, all minions would receive the same permissions.
3. A practical exception to (1) is minion pillar compilation, where the master impersonates each minion by issuing minion-specific credentials to itself. This avoids distributing secrets to minions they are not supposed to have access to.

I'm pretty sure omitting the peer_run master configuration (which allows minions to request credentials) also breaks (3), so there is currently no way of avoiding minions requesting Vault configuration/credentials. You can still block connections on the network layer.

This fundamental architecture is not going to change. It would be fairly straightforward to disallow non-impersonated authentication requests though, for which you can create a feature request at https://github.com/salt-extensions/saltext-vault/issues.

Note that pillar compilation can be a performance bottleneck for the master and minions only receive access to the secrets they are supposed to be able to read, so in general it's recommended for minions to do the lookups themselves. Be absolutely sure that the described behavior is what you need and be aware of the tradeoffs.

I would write pillar files where I access a secret

It's not recommended to access the Vault modules during regular .sls pillar compilation because of circular reference issues, unless you're not using the pillar to assign policies to your minions (which one generally should do). The dedicated pillar module does not have this limitation.

But I wonder then how I could tell that this minion is only allowed to read this certain data

A minion is allowed access to any paths its policies allow it to, this question is not specific to the Vault integration. You can show policies a minion receives based on your master configuration via salt-run vault.show_policies <minion_id>.

For me it should work like that [...]

In general, one would add a token/secret ID without a cap on the number of uses, but with one on the validity time. When using Salt 3007.x or the extension, the master automatically renews the current token during requests. While it does not have a dedicated renewal background process, the token is usually kept alive just by the base activity.

Afaik you can use the env SDB module to source the VAULT_TOKEN from the Vault Agent (https://salt-extensions.github.io/saltext-vault/ref/configuration.html#token) (Edit: This might be broken though - if you decide to test it and it does not work, feel free to submit a bug report).

Can someone explain how that should be done

The common setups are documented in the guide you linked. I hope this post helps explain the concepts, but feel free to ask any further questions.

[elwood218]

@lkubb Really thank you for that fast answer and that detailed explanation and clarification of the concept!
In my view it would be better if only the master would need access to the vault but I am not that deep into the whole Salt so probably as you said the performance could be later a problem and maybe it is also more secure if each minion by itself can access it. I thought at least it would be better to limit that access and I was also not sure if each "minion" also if there can no minion be installed can make use of that if "master" should not be allowed to read secrets.

I didn't want to put secrets on master pillar directly as they are not encrypted and the secrets are in the vault anyway - so there is no need to put and manage secrets in multiple stores. That's why I liked the idea of integrate Vault.

I will test that setup then with access by each minion. But I also wonder how much load many minion would make on Vault then and how often master will do that. I will test and see in practice now - also how good the renewal works.