Enable bound SA tokens by marun · Pull Request #718 · openshift/cluster-kube-apiserver-operator

marun · 2020-01-16T10:42:24Z

Implements operator support for openshift/enhancements#150

TODO

- Add key management controller and test coverage of same
- Ensure keys are configured in the apiserver pods
- Ensure configuration of issuer and audience
- Unskip test coverage for the TokenRequest API
- Ensure e2e is passing consistently

marun · 2020-01-16T20:02:29Z

/retest

sttts · 2020-01-20T09:06:09Z

bindata/v4.1.0/kube-apiserver/defaultconfig.yaml

+serviceAccountIssuer: auth.openshift.io
+apiAudiences:
+  - auth.openshift.io
+serviceAccountSigningKeyFile: /etc/kubernetes/static-pod-certs/secrets/service-account-signing-key/service-account.key


isn't there are need to override this for bootstrapping? Or do we get this key from the render step that early?

The intent is not to enable bound tokens in the bootstrapping phase, since as per a discussion on the enhancement there does not appear to be a need for that. Will it be necessary to set these values in code that can detect that bootstrapping is complete rather than here?

I think we're ok not having bound tokens available from the bootstrap kubeapiserver. we asked in the enhancement.

My question was more whether this path is a valid one during bootstrap or whether the process dies if not.

Ah, ok. Will test tomorrow. Maybe the operator will have to detect that it is past the bootstrap phase and compose the config accordingly in code?

There are two override yaml files, one for the bootstrap phase, one for after. Just set sensible values for each.

pkg/operator/boundsatokensignercontroller/controller.go

sttts · 2020-01-20T09:11:53Z

pkg/operator/boundsatokensignercontroller/controller.go

+		c.queue.AddAfter(workQueueKey, readyInterval+10*time.Second)
+	}
+
+	certConfigMap, err := c.configMapClient.ConfigMaps(targetNamespace).Get(CertConfigMapName, metav1.GetOptions{})


what happens if the secret is changed, but the config map is not?

If its the operator secret that is changing, that is not a problem because it is not used directly by the apiserver instances. In the case of the operand secret, notice that promotion occurs only after the configmap has been successfully updated.

Hopefully the functional separation makes my intent clear?

pkg/operator/boundsatokensignercontroller/controller.go

sttts · 2020-01-20T09:14:07Z

pkg/operator/boundsatokensignercontroller/controller.go

+	// Giving time for apiserver instances to pick up the change in public keys before
+	// changing the private key minimizes the potential for one or more apiservers to
+	// issue tokens signed by the new private key that apiservers without the
+	// corresponding public key are unable to validate.


so we lack back-pressure again here, right? If rolling update is blocked for some reason, we might get into trouble.

Is this still a problem now that actual state is considered rather than just waiting for a random amount of time?

Will read the new code. Looking at the deployed state should be enough.

sttts · 2020-01-20T09:15:03Z

pkg/operator/boundsatokensignercontroller/controller.go

+	// corresponding public key are unable to validate.
+	//
+	// TODO(marun) Find a more accurate indication that all apiservers are capable of
+	// validating tokens signed by the new private key.


In the etcd encryption code, we wait until all API servers have settled on the same revision, and that there is no new pending revision. Maybe that approach works here too?

+1 for having the CM revisioned.

Are the service account tokens private and pub keys read dynamically since we're just swapping them here without redeployment?

Can you point me to the etcd code in question? And should there be a corresponding update to the kcmo token controller?

re: dynamic key reads - afaict it's enough to just update the resource. Any changes to the resources/config that influence the state of apiserver pods prompt a redeployment of a single pod and only if that redeployment is successful will the change be rolled out to all pods.

For this code, you're even ok to avoid demanding a stable level, you just need each revision on nodes to include the cert for your key.

Done, PTAL.

pkg/operator/boundsatokensignercontroller/controller.go

pkg/operator/resourcesynccontroller/resourcesynccontroller.go

stlaz · 2020-01-20T12:50:09Z

bindata/v4.1.0/kube-apiserver/defaultconfig.yaml

 serviceAccountPublicKeyFiles:
  - /etc/kubernetes/static-pod-resources/configmaps/sa-token-signing-certs
+  - /etc/kubernetes/static-pod-certs/configmaps/bound-sa-token-signing-certs
+serviceAccountIssuer: auth.openshift.io


this should be mentioned as a default value in openshift/api#569

Ok, it wasn't clear to me that a value that wasn't set by default at the API level should be documented for the API type. Done.

stlaz · 2020-01-20T12:57:13Z

pkg/operator/boundsatokensignercontroller/controller.go

+	TokenReadyAnnotation = "kube-apiserver.openshift.io/ready-to-use"
+	readyInterval        = 5 * time.Minute
+
+	CertConfigMapName = "bound-sa-token-signing-certs"


nit: there will never be actual certs in this CM, will they?

No, but I'm being consistent with the name of the controller-manager equivalent (sa-tokens-signing-certs) which has effectively the same content.

interesting, ok

pkg/operator/resourcesynccontroller/resourcesynccontroller.go

stlaz · 2020-01-20T13:12:03Z

pkg/operator/boundsatokensignercontroller/controller.go

+	}
+	needKeypair := errors.IsNotFound(err) || len(signingSecret.Data[PrivateKeyKey]) == 0 || len(signingSecret.Data[PublicKeyKey]) == 0
+	if needKeypair {
+		newSecret, err := newSigningSecret()


While ApplySecret has certain output, a human-friendly log line might be helpful here

pkg/operator/boundsatokensignercontroller/controller.go

stlaz · 2020-01-20T14:03:16Z

pkg/operator/boundsatokensignercontroller/controller.go

+	// corresponding public key are unable to validate.
+	//
+	// TODO(marun) Find a more accurate indication that all apiservers are capable of
+	// validating tokens signed by the new private key.


+1 for having the CM revisioned.

Are the service account tokens private and pub keys read dynamically since we're just swapping them here without redeployment?

stlaz · 2020-01-20T14:39:35Z

pkg/operator/boundsatokensignercontroller/controller.go

+	go wait.Until(func() {
+		ticker := time.NewTicker(time.Minute)
+		defer ticker.Stop()
+
+		for {
+			c.queue.Add(workQueueKey)
+			select {
+			case <-ticker.C:
+			case <-stopCh:
+				return
+			}
+		}
+
+	}, time.Minute, stopCh)


It this correct? That inner ticker looks quite unnecessary given that wait.Until has its own internal timer which does the same, this basically runs wait.Until and freezes it on its first loop?

I think you're right - it should be sufficient to call func() {c.queue.Add(workQueueKey)} as the argument to wait.Util. Updated.

deads2k · 2020-01-20T21:24:22Z

bindata/v4.1.0/kube-apiserver/defaultconfig.yaml

 serviceAccountPublicKeyFiles:
  - /etc/kubernetes/static-pod-resources/configmaps/sa-token-signing-certs
+  - /etc/kubernetes/static-pod-certs/configmaps/bound-sa-token-signing-certs
+serviceAccountIssuer: auth.openshift.io


please pass as a flag, not as a struct value. Same with all these values.

You're looking for apiServerArguments above and add comments to help future me who won't remember this as well.

deads2k · 2020-01-20T21:28:36Z

pkg/operator/boundsatokensignercontroller/controller.go

+	}
+	needKeypair := errors.IsNotFound(err) || len(signingSecret.Data[PrivateKeyKey]) == 0 || len(signingSecret.Data[PublicKeyKey]) == 0
+	if needKeypair {
+		klog.Infof("Creating a new signing secret for bound service account tokens.")


if it's important enough for an info message, it's important enough for an event. If it isn't important enough for an event, then it belongs at a lower level.

Which would you prefer - even or lower level log?

deads2k · 2020-01-20T21:31:27Z

pkg/operator/boundsatokensignercontroller/controller.go

+	// did not have the latest public key would not be able to
+	// validate those new tokens.
+	TokenReadyAnnotation = "kube-apiserver.openshift.io/ready-to-use"
+	readyInterval        = 5 * time.Minute


You have the ability to inspect the revisions to figure out which ones contains the key in question. You can then inspect the levels on the nodes (the kubeapiservers.operator.openshift.io) to know if the nodes actually have the levels required. No need for time.

deads2k · 2020-01-20T21:32:11Z

pkg/operator/boundsatokensignercontroller/controller.go

+		// passed (see comment above the ready interval constant). Do not return
+		// immediately to ensure that the new public key can be set in the configmap
+		// in advance of promotion.
+		c.queue.AddAfter(workQueueKey, readyInterval+10*time.Second)


if you make your decision based on levels of the configmap actually on the nodes and trigger based on updates to kubeapiserver.operator.openshift.io, you don't need to have this delay.

deads2k · 2020-01-20T21:36:59Z

pkg/operator/boundsatokensignercontroller/controller.go

+	return ret
+}
+
+func (c *BoundSATokenSignerController) sync() error {


please break this function into logical bits with no data based between them. I see

create a secret if needed. depends on nothing outside the secrets

update configmap if needed. This can retrieve the current secret. A stale lister will always get another event notification, so staleness of a cache doesn't matter and this can then be contained.

promoting a key. This is based on the current secret in the operator namespace, the current secret in the operand namespace, the revisions on the nodes, the content of the configmaps on those nodes. All of which can be cached.

If any step has an error, the other steps should still be run.

stlaz

looking better, found some issues

pkg/operator/boundsatokensignercontroller/controller.go

stlaz · 2020-01-21T12:40:08Z

pkg/operator/boundsatokensignercontroller/controller.go

+	return nil
+}
+
+// ensureOperandSigningSecret ensures that the current signing key is


missing a verb here

pkg/operator/boundsatokensignercontroller/controller.go

stlaz · 2020-01-21T13:04:12Z

pkg/operator/boundsatokensignercontroller/controller.go

+			break
+		}
+	}
+	return hasValue


return false

stlaz · 2020-01-21T13:04:20Z

pkg/operator/boundsatokensignercontroller/controller.go

+	hasValue := false
+	for _, value := range configMap.Data {
+		if value == desiredValue {
+			hasValue = true


return true

stlaz · 2020-01-21T13:04:45Z

pkg/operator/boundsatokensignercontroller/controller.go

+			return false, err
+		}
+		found := false
+		for _, value := range configMap.Data {


configMapHasValue ?

sttts · 2020-01-21T14:38:02Z

pkg/operator/boundsatokensignercontroller/controller.go

+		err := syncMethod()
+		if err != nil {
+			utilruntime.HandleError(err)
+			syncFailed = true


usually we aggregate the errors in a slice and then return NewAggregate(errs).

I'd prefer to log the error raw to simplify troubleshooting. afaict wrapping errors in any way (including NewAggregate) changes the file+line that is logged. Is that ok, or do you require that I use NewAggregate?

marun · 2020-01-21T17:10:30Z

Updated with fixup, PTAL

marun · 2020-01-22T04:10:15Z

/retest

stlaz · 2020-01-22T11:25:35Z

/retest

stlaz · 2020-01-22T13:42:31Z

/retest

sttts · 2020-01-23T16:05:28Z

pkg/operator/boundsatokensignercontroller/controller.go

+	return ret
+}
+
+func (c *BoundSATokenSignerController) sync() bool {


the inverse is more standard: true = ok

Done. All the more important given that the call site was assuming true = ok too.

sttts · 2020-01-23T16:08:59Z

pkg/operator/boundsatokensignercontroller/controller.go

+// ensureOperatorSigningSecret ensures the existence of a secret in the operator
+// namespace containing an RSA keypair used for signing and validating bound service
+// account tokens.
+func (c *BoundSATokenSignerController) ensureOperatorSigningSecret() error {


ensureNextOperatorSigningSecret

sttts · 2020-01-23T16:09:37Z

pkg/operator/boundsatokensignercontroller/controller.go

+}
+
+// newSigningSecret creates a new secret populated with a new keypair.
+func newSigningSecret() (*corev1.Secret, error) {


newNextSigningSecret

sttts · 2020-01-23T16:12:27Z

pkg/operator/boundsatokensignercontroller/controller.go

+		// minimize the potential for not being able to validate issued tokens.
+		nextKeyIndex := len(configMap.Data) + 1
+		nextKeyKey := ""
+		for len(nextKeyKey) == 0 {


is the len comparison even needed with the break?

pkg/operator/boundsatokensignercontroller/controller.go

sttts · 2020-01-23T16:19:43Z

pkg/operator/boundsatokensignercontroller/controller.go

+	} else {
+		// Update the operand secret only if the current public key has been synced to
+		// all nodes.
+		syncRequired, err = c.publicKeySyncedToAllNodes(currPublicKey)


s/syncRequired/syncAllowed/

sttts · 2020-01-23T16:22:47Z

bindata/v4.1.0/kube-apiserver/boundsatokenconfig.yaml

+  - /etc/kubernetes/static-pod-resources/configmaps/sa-token-signing-certs
+  # The following path contains the public keys needed to verify bound sa
+  # tokens. This is only supported post-bootstrap.
+  - /etc/kubernetes/static-pod-resources/configmaps/bound-sa-token-signing-certs


just copy these into the overrides file

Which overrides file? afaict the only ones available are for bootstrap. Would you prefer that I add another post-bootstrap override file rather than adding this feature-specific one?

https://github.com/openshift/cluster-kube-apiserver-operator/blob/master/bindata/bootkube/config/config-overrides.yaml

This file has been renamed to config-overrides.yaml as requested.

sttts · 2020-01-23T16:23:11Z

pkg/operator/targetconfigcontroller/targetconfigcontroller.go

 		"config.yaml",
 		specialMergeRules,
 		defaultConfig,
+		boundSATokenConfig,


didn't we have the overrides yaml here as well?

afaict the overrides yaml are only for bootstrap:

https://github.com/openshift/cluster-kube-apiserver-operator/tree/master/bindata/bootkube/config

I don't know why there are 2 files for bootstrap. I do know that putting these options in the either of the bootstrap overrides will break bootstrapping because the bound token keypair is only created post-bootstrap.

https://github.com/openshift/cluster-kube-apiserver-operator/blob/master/bindata/bootkube/config/config-overrides.yaml

I've posted a new PR to document the requirement to put post-bootstrap overrides in the renamed file: #731

marun · 2020-01-23T18:57:42Z

@sttts Updated, PTAL

This controller is modeled after the one that manages key material for legacy sa tokens in cluster-kube-controller-manager-operator, but is simplified by not having to enable bound sa tokens during bootstrap.

marun · 2020-01-24T00:34:36Z

@sttts Updated, PTAL

marun · 2020-01-24T06:27:35Z

/test all

marun · 2020-01-24T09:03:56Z

@damemi I've added the DO NOT MERGE Bump revision limit in TestRevisionLimit commit to bump the apparently magic revision limit in the TestRevisionLimit. This PR is likely increasing the number of revisions in the process of coordinating the keypair configuration required to enable bound sa tokens. I'd appreciate your input as to whether changing the magic number is an acceptable workaround of if more involved changes are required.

sttts · 2020-01-24T09:39:15Z

test/e2e/operator_test.go

 	totalRevisionLimit := operatorSpec.SucceededRevisionLimit + operatorSpec.FailedRevisionLimit
 	if operatorSpec.SucceededRevisionLimit == 0 {
-		totalRevisionLimit += 5
+		totalRevisionLimit += 10


For a follow-up: @damemi you wrote this original test. Why was this 5? Magic number?

I now get the +=5 I believe (it's the default value?). But why does the loop below not wait for the revision pruning controller to do its work, but errors the test immediately?

The addition of bound token configuration - specifically keypair management - appears to be adding to the number of revisions just enough to break the test. The test needs to be updated to allow the pruning controller time to work.

sttts · 2020-01-24T10:32:11Z

test/e2e/operator_test.go

 		// Check total+1 to account for possibly a current new revision that just hasn't pruned off the oldest one yet.
 		if len(newRevisions) > int(totalRevisionLimit)+1 {
-			t.Errorf("more revisions (%v) than total allowed (%v): %+v", len(revisions), totalRevisionLimit, revisions)
+			// TODO(marun) If number of revisions has been exceeded, need to give time for the pruning controller to


sttts · 2020-01-24T10:32:25Z

/lgtm
/approve

openshift-ci-robot · 2020-01-24T10:33:17Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: marun, sttts

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [sttts]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-bot · 2020-01-24T12:38:27Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-01-24T13:29:35Z

/retest

Please review the full test history for this PR and help us cut down flakes.

This reverts commit 303d32d, reversing changes made to 7e19f69.

openshift-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jan 16, 2020

openshift-ci-robot requested review from mfojtik and sttts January 16, 2020 10:43

sttts reviewed Jan 20, 2020

View reviewed changes

pkg/operator/boundsatokensignercontroller/controller.go Outdated Show resolved Hide resolved

sttts reviewed Jan 20, 2020

View reviewed changes

pkg/operator/boundsatokensignercontroller/controller.go Show resolved Hide resolved

sttts reviewed Jan 20, 2020

View reviewed changes

pkg/operator/boundsatokensignercontroller/controller.go Outdated Show resolved Hide resolved

sttts reviewed Jan 20, 2020

View reviewed changes

pkg/operator/boundsatokensignercontroller/controller.go Outdated Show resolved Hide resolved

sttts reviewed Jan 20, 2020

View reviewed changes

pkg/operator/boundsatokensignercontroller/controller.go Outdated Show resolved Hide resolved

sttts reviewed Jan 20, 2020

View reviewed changes

pkg/operator/boundsatokensignercontroller/controller.go Show resolved Hide resolved

sttts reviewed Jan 20, 2020

View reviewed changes

pkg/operator/resourcesynccontroller/resourcesynccontroller.go Show resolved Hide resolved

stlaz suggested changes Jan 20, 2020

View reviewed changes

deads2k reviewed Jan 20, 2020

View reviewed changes

marun changed the title ~~WIP Enable bound SA tokens~~ Enable bound SA tokens Jan 21, 2020

openshift-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 21, 2020

stlaz suggested changes Jan 21, 2020

View reviewed changes

sttts reviewed Jan 21, 2020

View reviewed changes

sttts reviewed Jan 23, 2020

View reviewed changes

pkg/operator/boundsatokensignercontroller/controller.go Show resolved Hide resolved

sttts reviewed Jan 23, 2020

View reviewed changes

marun and others added 3 commits January 23, 2020 16:03

Add controller to manage key material for bound sa tokens

40859c7

This controller is modeled after the one that manages key material for legacy sa tokens in cluster-kube-controller-manager-operator, but is simplified by not having to enable bound sa tokens during bootstrap.

Configure bound service account tokens

07cd609

Update bindata

fac4e70

sttts reviewed Jan 24, 2020

View reviewed changes

openshift-ci-robot assigned sttts Jan 24, 2020

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Jan 24, 2020

openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 24, 2020

openshift-merge-robot merged commit 303d32d into openshift:master Jan 24, 2020

sttts mentioned this pull request Jan 24, 2020

Remove unused post-bootstrap overrides from bootstrap bindata #731

Closed

deads2k added a commit to deads2k/cluster-kube-apiserver-operator that referenced this pull request Jan 27, 2020

Revert "Merge pull request openshift#718 from marun/bound-sa-tokens"

c42a5e4

This reverts commit 303d32d, reversing changes made to 7e19f69.

deads2k added a commit to deads2k/cluster-kube-apiserver-operator that referenced this pull request Jan 27, 2020

Revert "Merge pull request openshift#718 from marun/bound-sa-tokens"

d431463

This reverts commit 303d32d, reversing changes made to 7e19f69.

marun deleted the bound-sa-tokens branch February 4, 2020 14:33

Conversation

marun commented Jan 16, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

marun commented Jan 16, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

marun commented Jan 16, 2020 •

edited

Loading