Skip to content

Commit

Permalink
auth/kubernetes: support for dynamically reloading short-lived tokens (
Browse files Browse the repository at this point in the history
…#13595)

* auth/kubernetes: support for short-lived tokens

* Uplift new version of Kubernetes auth plugin that does not store the
  service account token persistently to Vault storage.

* Update the documentation to recommend local token again when running
  Vault inside cluster.

Signed-off-by: Tero Saarni <[email protected]>

* Added changelog entry

Signed-off-by: Tero Saarni <[email protected]>

* clarification to changelog entry, executed go mod tidy

* clarifications and added targeted release version
  • Loading branch information
tsaarni authored and tvoran committed Jan 19, 2022
1 parent 9f2f924 commit f30e438
Show file tree
Hide file tree
Showing 5 changed files with 137 additions and 10 deletions.
3 changes: 3 additions & 0 deletions changelog/13595.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
```release-note:improvement
auth/kubernetes: Added support for dynamically reloading short-lived tokens for better Kubernetes 1.21+ compatibility
```
2 changes: 1 addition & 1 deletion go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@ require (
github.com/hashicorp/vault-plugin-auth-gcp v0.11.3
github.com/hashicorp/vault-plugin-auth-jwt v0.11.4
github.com/hashicorp/vault-plugin-auth-kerberos v0.5.0
github.com/hashicorp/vault-plugin-auth-kubernetes v0.11.3
github.com/hashicorp/vault-plugin-auth-kubernetes v0.7.1-0.20220107030939-d289258274b7
github.com/hashicorp/vault-plugin-auth-oci v0.9.0
github.com/hashicorp/vault-plugin-database-couchbase v0.5.1
github.com/hashicorp/vault-plugin-database-elasticsearch v0.9.1
Expand Down
4 changes: 2 additions & 2 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -942,8 +942,8 @@ github.com/hashicorp/vault-plugin-auth-jwt v0.11.4 h1:rL/hvd7uGB8CGpw1FKxxUD/dBJ
github.com/hashicorp/vault-plugin-auth-jwt v0.11.4/go.mod h1:jzjDdssus8sw8G6NOP7kNFMEeIvrjXvPHUR3pEn5+r0=
github.com/hashicorp/vault-plugin-auth-kerberos v0.5.0 h1:oORxeqOraVVLQrb+z3fj5JayPmH/JBxJWGywZ8ZRJt0=
github.com/hashicorp/vault-plugin-auth-kerberos v0.5.0/go.mod h1:eqjae8tMBpAWgJNk1NjV/vtJYXQRZnYudUkBFowz3bY=
github.com/hashicorp/vault-plugin-auth-kubernetes v0.11.3 h1:VTl62rRNhcALzsLw8romBZfTRpVna2IeLTN0kAQyXvY=
github.com/hashicorp/vault-plugin-auth-kubernetes v0.11.3/go.mod h1:HNcW9fnQIKw9g5qnxRQn6pHfGnSuVwBJAGb/v2/2dvw=
github.com/hashicorp/vault-plugin-auth-kubernetes v0.7.1-0.20220107030939-d289258274b7 h1:/VoIuHApeOStEIgLIF8J77OsLuCPLEhsfGnt3iYEivw=
github.com/hashicorp/vault-plugin-auth-kubernetes v0.7.1-0.20220107030939-d289258274b7/go.mod h1:HNcW9fnQIKw9g5qnxRQn6pHfGnSuVwBJAGb/v2/2dvw=
github.com/hashicorp/vault-plugin-auth-oci v0.9.0 h1:5wuHuPsW/MM5x0yvbr5ZwFLviNdF7q2t+z9saL7zjcI=
github.com/hashicorp/vault-plugin-auth-oci v0.9.0/go.mod h1:Cn5cjR279Y+snw8LTaiLTko3KGrbigRbsQPOd2D5xDw=
github.com/hashicorp/vault-plugin-database-couchbase v0.5.1 h1:WsXcOHHVwphwsrNGxpxRHcFzVgApN17ZNiE5RVD+q78=
Expand Down
2 changes: 2 additions & 0 deletions website/content/api-docs/auth/kubernetes.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -29,8 +29,10 @@ access the Kubernetes API.

- `kubernetes_host` `(string: <required>)` - Host must be a host string, a host:port pair, or a URL to the base of the Kubernetes API server.
- `kubernetes_ca_cert` `(string: "")` - PEM encoded CA cert for use by the TLS client used to talk with the Kubernetes API. NOTE: Every line must end with a newline: `\n`
If not set, the local CA cert will be used if running in a Kubernetes pod.
- `token_reviewer_jwt` `(string: "")` - A service account JWT used to access the TokenReview
API to validate other JWTs during login. If not set,
the local service account token is used if running in a Kubernetes pod, otherwise
the JWT submitted in the login payload will be used to access the Kubernetes TokenReview API.
- `pem_keys` `(array: [])` - Optional list of PEM-formatted public keys or certificates
used to verify the signatures of Kubernetes service account
Expand Down
136 changes: 129 additions & 7 deletions website/content/docs/auth/kubernetes.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -98,8 +98,128 @@ management tool.
This role authorizes the "vault-auth" service account in the default
namespace and it gives it the default policy.

For the complete list of configuration options, please see the [API
documentation](/api/auth/kubernetes).
For the complete list of configuration options, please see the [API
documentation](/api/auth/kubernetes).

## Kubernetes 1.21

Starting in version [1.21][k8s-1.21-changelog], the Kubernetes
`BoundServiceAccountTokenVolume` feature defaults to enabled. This changes the
JWT token mounted into containers by default in two ways that are important for
Kubernetes auth:

- It has an expiry time and is bound to the lifetime of the pod and service account.
- The value of the JWT's `"iss"` claim depends on the cluster's configuration.

The changes to token lifetime are important when configuring the
[`token_reviewer_jwt`](/api-docs/auth/kubernetes#token_reviewer_jwt) option.
If a short-lived token is used,
Kubernetes will revoke it as soon as the pod or service account are deleted, or
if the expiry time passes, and Vault will no longer be able to use the
`TokenReview` API. See [How to work with short-lived Kubernetes tokens][short-lived-tokens]
below for details on handling this change.

In response to the issuer changes, Kubernetes auth has been updated in Vault
1.9.0 to not validate the issuer by default. The Kubernetes API does the same
validation when reviewing tokens, so enabling issuer validation on the Vault
side is duplicated work. Without disabling Vault's issuer validation, it is not
possible for a single Kubernetes auth configuration to work for default mounted
pod tokens with both Kubernetes 1.20 and 1.21. Note that auth mounts created
before Vault 1.9 will maintain the old default, and you will need to explicitly
set `disable_iss_validation=true` before upgrading Kubernetes to 1.21. See
[Discovering the service account `issuer`](#discovering-the-service-account-issuer)
below for guidance if you wish to enable issuer validation in Vault.

[k8s-1.21-changelog]: https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.21.md#api-change-2
[short-lived-tokens]: #how-to-work-with-short-lived-kubernetes-tokens

### How to work with short-lived Kubernetes tokens

There are a few different ways to configure auth for Kubernetes pods when
default mounted pod tokens are short-lived, each with their own tradeoffs. This
table summarizes the options, each of which is explained in more detail below.

| Option | All tokens are short-lived | Can revoke tokens early | Other considerations |
| ------------------------------------ | -------------------------- | ----------------------- | --------------------------------------------------------------- |
| Use local token as reviewer JWT | Yes | Yes | Requires Vault (1.10+) to be deployed on the Kubernetes cluster |
| Use client JWT as reviewer JWT | Yes | Yes | Operational overhead |
| Use long-lived token as reviewer JWT | No | Yes | |
| Use JWT auth instead | Yes | No | |

-> **Note:** By default, Kubernetes currently extends the lifetime of admission
injected service account tokens to a year to help smooth the transition to
short-lived tokens. If you would like to disable this, set
[--service-account-extend-token-expiration=false][k8s-extended-tokens] for
`kube-apiserver` or specify your own `serviceAccountToken` volume mount. See
[here](/docs/auth/jwt/oidc_providers#specifying-ttl-and-audience) for an example.

[k8s-extended-tokens]: https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/#options

#### Use local service account token as the reviewer JWT

When running Vault in a Kubernetes pod the recommended option is to use the pod's local
service account token. Vault will periodically re-read the file to support
short-lived tokens. To use the local token and CA certificate, omit
`token_reviewer_jwt` and `kubernetes_ca_cert` when configuring the auth method.
Vault will attempt to load them from `token` and `ca.crt` respectively inside
the default mount folder `/var/run/secrets/kubernetes.io/serviceaccount/`.

```bash
vault write auth/kubernetes/config \
kubernetes_host=https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_PORT
```

!> **Note:** Requires Vault 1.10+. In earlier versions the service account
token and CA certificate is read once and stored in Vault storage.
When the service account token expires or is revoked, Vault will no longer be
able to use the `TokenReview` API and client authentication will fail.

#### Use the Vault client's JWT as the reviewer JWT

When configuring Kubernetes auth, you can omit the `token_reviewer_jwt`, and Vault
will use the Vault client's JWT as its own auth token when communicating with
the Kubernetes `TokenReview` API. If Vault is running in Kubernetes, you also need
to set `disable_local_ca_jwt=true`.

This means Vault does not store any JWTs and allows you to use short-lived tokens
everywhere but adds some operational overhead to maintain the cluster role
bindings on the set of service accounts you want to be able to authenticate with
Vault. Each client of Vault would need the `system:auth-delegator` ClusterRole:

```bash
kubectl create clusterrolebinding vault-client-auth-delegator \
--clusterrole=system:auth-delegator \
--group=group1 \
--serviceaccount=default:svcaccount1 \
...
```

#### Continue using long-lived tokens

The default Kubernetes secret created for a service account is still long lived,
and can be used as the `token_reviewer_jwt` without needing to refresh it. To
find the secret, run:

```bash
kubectl get secret "$(kubectl get serviceaccount default -o jsonpath='{.secrets[0].name}')"
```

Using this maintains previous workflows but does not fully take advantage of the
new default short-lived tokens.

#### Use JWT auth

Kubernetes auth is specialized to use Kubernetes' `TokenReview` API. However, the
JWT tokens Kubernetes generates can also be verified using Kubernetes as an OIDC
provider. The JWT auth method documentation has [instructions][k8s-jwt-auth] for
setting up JWT auth with Kubernetes as the OIDC provider.

[k8s-jwt-auth]: /docs/auth/jwt/oidc_providers#kubernetes

This solution allows you to use short-lived tokens for all clients and removes
the need for a reviewer JWT. However, the client tokens cannot be revoked before
their TTL expires, so it is recommended to keep the TTL short with that
limitation in mind.

### Discovering the service account `issuer`

Expand Down Expand Up @@ -253,6 +373,7 @@ func getSecretWithKubernetesAuth() (string, error) {
return value, nil
}
```

</CodeBlockConfig>

<CodeBlockConfig lineNumbers>
Expand All @@ -267,7 +388,7 @@ using VaultSharp.V1.Commons;

namespace Examples
{
public class KubernetesAuthExample
public class KubernetesAuthExample
{
const string DefaultTokenPath = "path/to/service-account-token";

Expand All @@ -289,24 +410,25 @@ namespace Examples

// Get the path to service account token or fall back on default path
string pathToToken = String.IsNullOrEmpty(Environment.GetEnvironmentVariable("SA_TOKEN_PATH")) ? DefaultTokenPath : Environment.GetEnvironmentVariable("SA_TOKEN_PATH");
string jwt = File.ReadAllText(pathToToken);
string jwt = File.ReadAllText(pathToToken);

IAuthMethodInfo authMethod = new KubernetesAuthMethodInfo(roleName, jwt);
var vaultClientSettings = new VaultClientSettings(vaultAddr, authMethod);

IVaultClient vaultClient = new VaultClient(vaultClientSettings);
IVaultClient vaultClient = new VaultClient(vaultClientSettings);

// We can retrieve the secret after creating our VaultClient object
Secret<SecretData> kv2Secret = null;
kv2Secret = vaultClient.V1.Secrets.KeyValue.V2.ReadSecretAsync(path: "/creds").Result;

var password = kv2Secret.Data.Data["password"];

return password.ToString();
}
}
}
```

</CodeBlockConfig>

</CodeTabs>

0 comments on commit f30e438

Please sign in to comment.