Do not store local service account token and CA to config. #122

tsaarni · 2021-10-14T11:13:41Z

Overview

Kubernetes 1.21 switched to new ID tokens which are now bound to a specific pod. When pod is deleted and recreated the old token will be invalidated and not accepted by Kubernetes API server anymore. Tokens will also expire and Kubernetes will renew them periodically. The new tokens cause major problem with Vault since currently, even when using local tokens, the token and CA certificate are read when the authentication method is configured and stored persistently to config.

Kubernetes authentication backend will stop working when:

the pod where the token happened to be copied from restarts (=pod gets recreated)
token expires

Vault will not recover automatically, it requires reconfiguration of the authentication method.

Design of Change

This change avoids storing local token and local CA certificate in config. Instead, they are loaded lazily when needed and kept in memory only. Token will also re-read periodically to facilitate token rotation and to avoid authentication failure due to expired token.

Related Issues/Pull Requests

Fixes #121
Updates #95

Tests

Acceptance tests pass.
New tests added.
Manual test on Kubernetes pass..

Backwards compatibility

I believe that previously GET /auth/kubernetes/config returned a copy of local CA file. This is not done anymore. The justification is that local file is not part of stored configuration. CA file is still returned if user explicitly configured it with POST /auth/kubernetes/config.

Contributor Checklist

Add relevant docs to upstream Vault repository, or sufficient reasoning why docs won’t be added yet
Issue link Kubernetes Auth Method documentation not compatible with Kubernetes v1.21+ vault#12855
Add output for any tests not ran in CI to the PR description (eg, acceptance tests)
Backwards compatible: see details under heading "Backwards compatibility" above

When defaulting to local JWT token and CA certificate in a pod, always read them from local filesystem and do not store them persistenly with the config. Token will be re-read periodically to avoid using expired token. The change allows running Vault on Kubernetes 1.21 and newer, which switched to ID token that is bound to the pod and will expire. Signed-off-by: Tero Saarni <[email protected]>

backend.go

…onfig

tsaarni · 2021-11-10T06:49:32Z

@hsimon-hashicorp Kindly asking if someone could take a look at this PR? Currently Vault's Kubernetes service account authentication method is broken and it would be really great to get it working again!

heatherezell · 2021-11-18T20:15:01Z

Chatted with @tvoran about this briefly; this is a note to myself to circle back on this later.

tomhjp · 2021-11-24T18:26:42Z

Hi @tsaarni, thanks for the PR! Sorry for the delay reviewing, but we're definitely interested in this feature. I haven't reviewed in detail yet, but will try to do so this week.

tomhjp

Looking good 👍 I'd like to remove the CA changes from the scope if possible, and I also want to make sure this is as robust as practically possible, so one comment on the caching approach.

path_config.go

backend.go

tsaarni

Thank you for the review @tomhjp, looking forward for further comments & doing final adjustments to the PR!

BTW Sometimes I find github really confusing 🤔 My response to ^^^ above ^^^ review comment is only visible in Files Changed view...

tomhjp

Sorry for the delay - I got part-way through a review and then got to the point where I was going to need to do some more involved testing for backwards compatibility/different deployment scenarios etc. I'm still yet to do that, so some of the comments are only semi-actionable. I'll leave what I've got here for now and revisit once I've had a chance to do the testing.

backend.go

caching_file_reader.go

path_config_test.go

path_config.go

backend.go

caching_file_reader_test.go

tsaarni · 2021-12-07T18:13:03Z

BTW I think it could be nice if there was a test environment with "fake" Kubernetes API server, allowing more thorough testing without compiling this plugin into Vault and running that against actual Kubernetes, in cluster and outside.

tvoran

I really like this approach! Left a few comments, but so far it has tested well for me for both short and long-lived JWTs, and when using the client token as the reviewer JWT.

I think in the future we may want to expose the local JWT and CA paths as options, but I think we can worry about that in another PR when the need arises.

path_config.go

backend.go

path_config.go

backend.go

tsaarni · 2021-12-14T19:37:57Z

I've realized that I should rework the lock handling a bit: I need to have write lock for backend, when loading local files and assigning them to variables in kubeAuthBackend.loadConfig().

backend.go

caching_file_reader.go

tsaarni · 2021-12-18T14:15:54Z

I wrote my manual test procedures here https://gist.github.com/tsaarni/66b085e0244fe1b0635cc1f806b5c334

tomhjp

LGTM, the test updates are really nice. Thanks for contributing such a significant feature! I'll let @tvoran have one last look but I think we can land this soon.

tvoran

This looks great and worked well for me when I did some testing. Just one more small suggestion and then I think it'll be good to go.

path_login.go

backend.go

…to client token

tvoran

Thanks again!

When defaulting to local JWT token and CA certificate in a pod, always read them from local filesystem and do not store them persistently with the config. Token will be re-read periodically to avoid using expired token. The change allows running Vault on Kubernetes 1.21 and newer, which switched to ID token that is bound to the pod and will expire. Signed-off-by: Tero Saarni <[email protected]> * review comment fix: load only token or ca cert if other is given in config * changed the reload period to 1 minute * fixed review comments * take lock also on alias lookahead path * more review fixes * proposal to fix the read/write lock issue * fixed typo * cachedFile by value to avoid mutation while not holding log * acquire lock in same place as in pathLogin * added debug log entry when local token is not found and falling back to client token

When defaulting to local JWT token and CA certificate in a pod, always read them from local filesystem and do not store them persistently with the config. Token will be re-read periodically to avoid using expired token. The change allows running Vault on Kubernetes 1.21 and newer, which switched to ID token that is bound to the pod and will expire. Signed-off-by: Tero Saarni <[email protected]> * review comment fix: load only token or ca cert if other is given in config * changed the reload period to 1 minute * fixed review comments * take lock also on alias lookahead path * more review fixes * proposal to fix the read/write lock issue * fixed typo * cachedFile by value to avoid mutation while not holding log * acquire lock in same place as in pathLogin * added debug log entry when local token is not found and falling back to client token Co-authored-by: Tero Saarni <[email protected]>

zohebs341 · 2022-02-17T19:24:28Z

Hi All,

We are facing a similar issue with Vault in our production workloads & and creating downtime. Can someone assist me with the vault part? I mean resolution for this.

A few days back we upgraded our Azure AKS Cluster from 1.20.7 to 1.21.7 and updated "disable_iss_validation=true" in vault config. It worked for some time, but intermittently we are facing issues with application pods which are crashing with permission denied errors and in vault logs, I can see below errors.

After going through multiple issues/docs, what I understood is vault is not able to load/read dynamically the service account/JWT tokens of pods, as it's changing frequently/expiring every hour in K8s version 1.21.7. Please assist me on this
Error From Vault pod logs:

[ERROR] auth.kubernetes.auth_kubernetes_34b3b46a: login unauthorized due to: lookup failed: service account unauthorized; this could mean it has been deleted or recreated with a new token
Error From Application Pod Logs:

Error making API request.

Code: 403. Errors:

1 error occurred:
permission denied
(retry attempt 7 after "16s")

Config File:

aultWriteAuthKubeConfig() {

echo && echo "[INFO]:: Writing Kubernetes auth config..."

vault write auth/kubernetes/config \

token_reviewer_jwt="$(cat ${KUBERNETES_SERVICE_TOKEN})" \

kubernetes_host=${KUBERNETES_HOST} \

kubernetes_ca_cert=@${KUBERNETES_CA_CERT} \

disable_iss_validation=true

krishnamohan987 · 2022-02-19T13:25:20Z

@zohebs341 upgrade vault to V1.9.3 or + , it will support for dynamically reloading short-lived tokens.

zohebs341 · 2022-02-19T13:52:54Z

@krishnamohan987 Thank you Krishna. We will try upgrading version.

OneideLuizSchneider · 2022-02-19T22:28:54Z

@krishnamohan987 Hi, the helm chart is still on v1.9.2, does it take a while to update there?

.

krishnamohan987 · 2022-02-20T04:52:20Z

@OneideLuizSchneider you can override image version in values yaml while you apply the chart. it works. downlaod the image ready and push to your registry.

tsaarni force-pushed the fix-local-service-account branch from 2fa1796 to b657e15 Compare October 14, 2021 18:29

tsaarni mentioned this pull request Oct 18, 2021

Kubernetes Auth Method documentation not compatible with Kubernetes v1.21+ hashicorp/vault#12855

Closed

RemcoBuddelmeijer reviewed Oct 27, 2021

View reviewed changes

backend.go Outdated Show resolved Hide resolved

backend.go Outdated Show resolved Hide resolved

backend.go Show resolved Hide resolved

review comment fix: load only token or ca cert if other is given in c…

a8d2533

…onfig

tsaarni mentioned this pull request Oct 27, 2021

Add possibility to rotate already defined token_reviewer_jwt tokens (backwards compatibility) hashicorp/vault#12937

Closed

RemcoBuddelmeijer mentioned this pull request Oct 28, 2021

Remove persisting of service account token Nordix/vault-plugin-auth-kubernetes#1

Closed

tomhjp mentioned this pull request Nov 24, 2021

Docs to clarify k8s auth options with short-lived tokens hashicorp/vault#13275

Merged

RemcoBuddelmeijer mentioned this pull request Nov 24, 2021

Remove usage of token_reviewer_jwt completely #128

Closed

tomhjp reviewed Nov 26, 2021

View reviewed changes

path_config.go Show resolved Hide resolved

backend.go Outdated Show resolved Hide resolved

tsaarni commented Nov 29, 2021

View reviewed changes

changed the reload period to 1 minute

0997bc2

tomhjp reviewed Dec 2, 2021

View reviewed changes

tsaarni added 2 commits December 6, 2021 23:20

fixed review comments

910097d

take lock also on alias lookahead path

9684ecb

tvoran reviewed Dec 14, 2021

View reviewed changes

path_config.go Outdated Show resolved Hide resolved

backend.go Outdated Show resolved Hide resolved

path_config.go Outdated Show resolved Hide resolved

path_config.go Outdated Show resolved Hide resolved

backend.go Outdated Show resolved Hide resolved

more review fixes

93e5047

tsaarni commented Dec 14, 2021

View reviewed changes

backend.go Outdated Show resolved Hide resolved

proposal to fix the read/write lock issue

b3e3a5a

calvn reviewed Dec 18, 2021

View reviewed changes

caching_file_reader.go Show resolved Hide resolved

tsaarni added 2 commits December 18, 2021 12:33

fixed typo

067a44d

cachedFile by value to avoid mutation while not holding log

8aea07a

tomhjp mentioned this pull request Jan 5, 2022

Best Practice / Docs clarification on token_reviewer_jwt #106

Closed

tomhjp approved these changes Jan 5, 2022

View reviewed changes

tvoran reviewed Jan 6, 2022

View reviewed changes

path_login.go Outdated Show resolved Hide resolved

aquire lock in same place as in pathLogin

859e023

tomhjp reviewed Jan 6, 2022

View reviewed changes

backend.go Outdated Show resolved Hide resolved

added debug log entry when local token is not found and falling back …

4b6ee54

…to client token

tvoran mentioned this pull request Jan 7, 2022

Vault agent can't authenticate using k8s 1.21 hashicorp/vault-helm#562

Closed

tvoran approved these changes Jan 7, 2022

View reviewed changes

tvoran merged commit d289258 into hashicorp:main Jan 7, 2022

tsaarni mentioned this pull request Jan 7, 2022

auth/kubernetes: support for dynamically reloading short-lived tokens hashicorp/vault#13595

Merged

tvoran mentioned this pull request Jan 18, 2022

Backport #122 to vault 1.9 #131

Merged

hc-github-team-secure-vault-core mentioned this pull request Jan 19, 2022

Backport of auth/kubernetes: support for dynamically reloading short-lived tokens into release/1.9.x hashicorp/vault#13698

Merged

hc-github-team-secure-vault-core mentioned this pull request Jan 20, 2022

Backport of Docs to clarify k8s auth options with short-lived tokens into release/1.9.x hashicorp/vault#13720

Merged

RemcoBuddelmeijer mentioned this pull request Feb 1, 2022

Authentication failures Hashicorp Vault on Kubernetes 1.21 hashicorp/vault#13867

Closed

tvoran mentioned this pull request Feb 8, 2022

Use pod service account instead of provided JWT for authentication #95

Closed

Oro mentioned this pull request Feb 24, 2022

Kubernetes 1.21 - Serviceregistration is not reloading the service account token hashicorp/vault#14225

Closed

tsaarni mentioned this pull request Dec 7, 2022

Ensure a consistent TLS configuration #173

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Do not store local service account token and CA to config. #122

Do not store local service account token and CA to config. #122

tsaarni commented Oct 14, 2021 •

edited

Loading

tsaarni commented Nov 10, 2021

heatherezell commented Nov 18, 2021

tomhjp commented Nov 24, 2021

tomhjp left a comment

tsaarni left a comment •

edited

Loading

tomhjp left a comment

tsaarni commented Dec 7, 2021

tvoran left a comment •

edited

Loading

tsaarni commented Dec 14, 2021 •

edited

Loading

tsaarni commented Dec 18, 2021

tomhjp left a comment

tvoran left a comment

tvoran left a comment

zohebs341 commented Feb 17, 2022

krishnamohan987 commented Feb 19, 2022

zohebs341 commented Feb 19, 2022

OneideLuizSchneider commented Feb 19, 2022

krishnamohan987 commented Feb 20, 2022

Do not store local service account token and CA to config. #122

Do not store local service account token and CA to config. #122

Conversation

tsaarni commented Oct 14, 2021 • edited Loading

Overview

Design of Change

Related Issues/Pull Requests

Tests

Backwards compatibility

Contributor Checklist

tsaarni commented Nov 10, 2021

heatherezell commented Nov 18, 2021

tomhjp commented Nov 24, 2021

tomhjp left a comment

Choose a reason for hiding this comment

tsaarni left a comment • edited Loading

Choose a reason for hiding this comment

tomhjp left a comment

Choose a reason for hiding this comment

tsaarni commented Dec 7, 2021

tvoran left a comment • edited Loading

Choose a reason for hiding this comment

tsaarni commented Dec 14, 2021 • edited Loading

tsaarni commented Dec 18, 2021

tomhjp left a comment

Choose a reason for hiding this comment

tvoran left a comment

Choose a reason for hiding this comment

tvoran left a comment

Choose a reason for hiding this comment

zohebs341 commented Feb 17, 2022

krishnamohan987 commented Feb 19, 2022

zohebs341 commented Feb 19, 2022

OneideLuizSchneider commented Feb 19, 2022

krishnamohan987 commented Feb 20, 2022

tsaarni commented Oct 14, 2021 •

edited

Loading

tsaarni left a comment •

edited

Loading

tvoran left a comment •

edited

Loading

tsaarni commented Dec 14, 2021 •

edited

Loading