Support a provisioning token for the Machine Config Server #443
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,147 @@ | ||
| --- | ||
| title: mco-auth-token | ||
| authors: | ||
| - "@cgwalters" | ||
| reviewers: | ||
| - "@crawford" | ||
| approvers: | ||
| - "@ashcrow" | ||
| - "@crawford" | ||
| - "@imcleod" | ||
| - "@runcom" | ||
| creation-date: 2020-08-05 | ||
| last-updated: 2020-08-19 | ||
| status: provisional | ||
| see-also: | ||
| replaces: | ||
| superseded-by: | ||
| --- | ||
|
|
||
| # Support a provisioning token for the Machine Config Server | ||
|
|
||
| ## Release Signoff Checklist | ||
|
|
||
| - [ ] Enhancement is `implementable` | ||
| - [ ] Design details are appropriately documented from clear requirements | ||
| - [ ] Test plan is defined | ||
| - [ ] Graduation criteria for dev preview, tech preview, GA | ||
| - [ ] User-facing documentation is created in [openshift-docs](https://github.com/openshift/openshift-docs/) | ||
|
|
||
| ## Summary | ||
|
|
||
| This enhancement proposes that for new (e.g. 4.7+) installations a "provisioning token" is required by default | ||
| to access the Machine Config Server. | ||
|
|
||
| For IPI/machineAPI scenarios, this will be handled fully automatically. | ||
|
|
||
| For user-provisioned installations, openshift-install will generate a token internally via the equivalent of | ||
|
|
||
| ``` | ||
| $ dd if=/dev/urandom bs=32 count=1 | base64 > token | ||
| $ oc -n openshift-config create secret generic machineconfig-auth --from-file=token | ||
| ``` | ||
|
|
||
| And then this token will be injected into the "pointer ignition configs" it creates as a URL parameter, or | ||
| potentially `Authorization: bearer`. | ||
|
|
||
|
|
||
| ## Motivation | ||
|
|
||
| The default Ignition configuration contains secrets (e.g. the pull secret, initial kubeconfig), and we want to avoid it being accessible both inside and outside the cluster. | ||
|
|
||
| ### Goals | ||
|
|
||
| - Increase security | ||
| - Avoid breaking upgrades | ||
| - Easy to implement | ||
| - Easy to understand for operators | ||
| - Eventually support being replaced by something more sophisticated (e.g. TPM2 attestation) | ||
| - Avoid too strong dependence on platform-specific functionality (i.e. UPI AWS should remain close to UPI bare metal) | ||
|
|
||
| ### Non-Goals | ||
|
|
||
| - Completely rework major parts of the provisioning process today | ||
|
|
||
| ## Proposal | ||
|
|
||
| Change the MachineConfigServer to support requiring a token in order to retrieve the Ignition config for new installs. | ||
|
|
||
| In IPI/machineAPI managed clusters, `openshift-install` and the MCO would automatically handle | ||
| this end to end; the initial implementation would work similarly to the UPI case, but may change | ||
| in the future to be even stronger. | ||
|
|
||
| ### Implementation Details/Notes/Constraints | ||
|
|
||
| For UPI, the documentation would need to be enhanced to describe this. | ||
|
|
||
| For IPI, it's now possible to implement automatic rotation because the `user-data` provided to nodes is owned by the MCO: https://github.com/openshift/enhancements/blob/master/enhancements/machine-config/user-data-secret-managed.md | ||
|
|
||
| In order to avoid race conditions, the MCS might support the "previous" token for a limited period of time (e.g. one hour/day). | ||
|
Contributor
There was a problem hiding this comment. Rotating the token will have implications for requirements for the ability to stage bare metal nodes before delivering them to remote sites for telco customers. If the worker node is installed on day 1, then it takes 3 days for it to be delivered to the site of a tower where it is powered back on and connected to the cluster, the token it has will have expired.
Member
Author
There was a problem hiding this comment. Thanks for looking at this! The short answer would be "don't rotate the token". A better fix needs a lot more information on exactly what "stage" means. I think a generally good workflow we want to support in CoreOS is using I think the answer is probably just "don't rotate the token while any nodes are being staged". Particularly for bare metal as I mention at the end, I think the clearly best solution is using TPM2 (e.g. in this case the "staging site" can communicate the hardware identity of the machine to the target in advance even), but that's much more involved.
Contributor
There was a problem hiding this comment.
I think that means there would be no IPI for bare metal with remote workers, which would defeat the goal.
I'm not sure what the policy about linking to product requirement tickets in GitHub reviews is, so I'll try to summarize. We need to install the host at one location so that it is configured to join its cluster properly (that may mean the pointer config, or something else, it's unclear). Then some of the hosts will be physically moved to different remote sites, plugged into the network, and they need to rejoin the cluster without any user intervention (we can't assume the person installing the hardware is qualified to do more than rack it and plug it in). IPI for bare metal does not use coreos-installer, it writes a disk image to a block device, including the pointer ignition configuration. See https://github.com/openshift/enhancements/blob/master/enhancements/baremetal/baremetal.md It sounds like you're suggesting that during staging the image and ignition pointer should be written to the host, but the host shouldn't be booted until it is at the remote location to preserve the pointer configuration. Is that right? I don't expect that to be acceptable given the usual burn-in and validation requirements for these sorts of use cases.
Is the lifetime for the automated rotation configurable? What happens if the token does rotate after a host leaves the staging site and before it is connected back to the cluster? Does the user ship the host back to the staging site to be re-imaged?
Member
Author
There was a problem hiding this comment. OK I had missed you're talking about IPI metal, not UPI. I think we probably need a whole section on this - the core problem here is that the "automatic IPI" case hinges on the MCO managing the user data, but it's not clear to me that applies in IPI/machineAPI metal (does it?). In other words, forget about staging for a second - would this enhancement work as described in IPI metal or not? Then once we have that part figured out...we can think about how we handle rotation. It might be that for IPI metal, we don't rotate by default - or we add an
Member
Author
There was a problem hiding this comment.
To rephrase; does https://github.com/openshift/enhancements/blob/master/enhancements/machine-config/user-data-secret-managed.md apply in IPI metal today?
Contributor
There was a problem hiding this comment. Metal IPI uses the user-data secret to pass the ignition pointer configuration to the host via cloudinit. So, yes, I think the user-data-secret-managed enhancement applies to metal IPI as well.
Contributor
There was a problem hiding this comment. For metal-ipi we also need to consider the issue of how to access the MCO managed config from the cluster-api-provider ref openshift/machine-config-operator#1690 - today we only provide the pointer config via the config-drive, but that means a chicken/egg issue with advanced network configurations. With this proposal, would we expect a privileged in-cluster controller to be able to access the MCS, or would we instead have to retrieve the underlying MachineConfig directly?
Member
Author
There was a problem hiding this comment. I don't think this proposal makes hacking things absent openshift/machine-config-operator#1690 significantly more difficult - a privileged pod could get the token to talk to the MCS too. (It doesn't help either of course) |
||
|
|
||
| #### IPI/machineAPI details | ||
|
|
||
| It's probably easiest to start out by automatically generating a rotating secret that works the same way as UPI, and also (per above) support the previous token for a period of time. | ||
|
|
||
| ### Risks and Mitigations | ||
|
|
||
| #### Troubleshooting | ||
|
|
||
| Debugging nodes failing in Ignition today is painful; usually doing so requires looking at the console. There's no tooling or automation around this. We will need to discuss this in a "node provisioning" section in the documentation. | ||
|
|
||
| This scenario would be easier to debug if [Ignition could report failures to the MCS](https://github.com/coreos/ignition/issues/585). | ||
|
|
||
| #### Disaster recovery | ||
|
|
||
| See: [Disaster recovery](https://docs.openshift.com/container-platform/4.5/backup_and_restore/disaster_recovery/about-disaster-recovery.html) | ||
|
|
||
| This proposes that the secret for provisioning a node is stored in the cluster itself (in the IPI case). But so is the configuration for the cluster, so this is not a new problem. | ||
|
|
||
| Note that the token is only necessary when Ignition is run - which is the first boot of a node before it joins the cluster. If a node is just shut down and restarted, access to the token isn't required. Hence there are no concerns if e.g. the whole control plane (or full cluster) is shut down and restarted. | ||
|
|
||
| In the case of e.g. trying to reprovision a control plane node, doing that today already requires the Machine Config Server to run, which requires a cluster functioning enough to run it. The auth token doesn't impose any additional burden to that. | ||
|
|
||
| That said, see "Rotating the token" below. | ||
|
|
||
| #### Non-Ignition components fetching the Ignition config | ||
|
|
||
| As part of the Ignition spec 3 transition, we needed to deal with non-Ignition consumers of the Ignition config, such as [openshift-ansible](github.com/openshift/openshift-ansible/) and Windows nodes. These should generally instead fetch the rendered `MachineConfig` object from the cluster, and not access the MCS port via TCP. | ||
|
Contributor
There was a problem hiding this comment. Is that how these components work today and if not, is it possible for them to do so? There's a relatively large amount of setup that needs to happen before a new client is able to fetch objects from the API server and I don't want to gloss over that without being sure it won't be a problem.
Member
Author
There was a problem hiding this comment. Yes, that's how it works today, see the code around here:
Looks like the ansible code accepts a kubeconfig. Things like rhel7 and Windows MCO should also just be able to run as a pod; what's the problem with that?
Contributor
There was a problem hiding this comment. Okay, it's good to see that the existing implementations are compatible. Can you update the wording to reflect that? Specifically, reading "should generally" doesn't give me confidence. |
||
|
|
||
| #### Rotating the token | ||
|
|
||
| In UPI scenarios, we should document rotating the token, though doing so incurs some danger if the administrator forgets to update the pointer configuration. We should recommend against administrators rotating the token *without* also implementing a "periodic reprovisioning" policy to validate that they can restore workers (and ideally control plane) machines. | ||
|
Contributor
There was a problem hiding this comment. Customers do not read documentation; especially if they believe that they already know how something works. Is there anything we can do in the UPI case to make it more obvious when an admin forgets to update the pointer config (without sacrificing security)?
Member
Author
There was a problem hiding this comment. I can't think of anything other than having a mechanism to report failures per above into something that propagates into the cluster. Well I guess in the IaaS case we could write code that tries to screen-scrape the console; we have some code for that today in the coreos testing infra (kola). We've actually already introduced a lot of pain around the pointer config for UPI around the Ignition Spec 3 transition, and then we accidentally added even more pain with the Go change for certs; see openshift/oc#628 (comment) Although at least for existing clusters that were upgraded this wouldn't apply by default (unless the admin enables the token).
Contributor
There was a problem hiding this comment. Having the MCS report auth failures would be a good start, but I was also thinking in the direction of what we can do on each node. If a node fails to provision, administrators are going to look at the console for hints. If we can make it extremely clear what is going on, it's going to save them time and frustration. |
||
|
|
||
| Or better, migrate to an IPI/machineAPI managed cluster. | ||
|
|
||
| ### Upgrades | ||
|
|
||
| This will not apply by default on upgrades, even in IPI/machineAPI managed clusters (to start). However, an administrator absolutely could enable this "day 2". Particularly for "static" user provisioned infrastructure where the new nodes are mostly manually provisioned, the burden of adding the token into the process of provisioning would likely be quite low. | ||
cgwalters marked this conversation as resolved.
Show resolved
Hide resolved
Contributor
There was a problem hiding this comment. Unless there is a solid reason, we should avoid fracturing the fleet of clusters by their initial version. This especially applies to security improvements. We don't need to commit to upgrading everyone into this new scheme in the first iteration, but I would like to see a plan for getting out of the hole before we jump into it.
Member
Author
There was a problem hiding this comment. In UPI cases we don't control the pointer config, so there's no way we can automate this right? We could add some sort of alerting and ask admins to do the switch I guess?
Contributor
There was a problem hiding this comment.
Right, not with this proposal. And that's my concern. I'd prefer a solution that is less hands-on and possible for us to roll out as an upgrade. |
||
|
|
||
| For existing machineAPI managed installs, it should be possible to automatically adapt existing installs to use this, but we would need to very carefully test and roll out such a change; that would be done as a separate phase. | ||
|
|
||
| ## Alternatives | ||
|
|
||
| #### Firewalling | ||
|
|
||
| Not obvious to system administrators in e.g. bare metal environments and difficult to enforce with 3rd party SDNs. | ||
|
|
||
| #### Move all secret data out of Ignition into the pointer config | ||
|
|
||
| Move the pull secret and bootstrap kubeconfig into the "pointer Ignition". In all cloud environments that we care about, access to the "user data" in which the pointer Ignition is stored is secured. It can't be accessed outside the cluster because it's a link local IP address, and the OpenShift SDN blocks it from the pod network (see references). | ||
|
|
||
| Risk: In UPI scenarios, particularly e.g. bare metal and vSphere type UPI that often involves a manually set up webserver, we do not currently require confidentiality for the pointer configuration. This would need to change - it'd require documentation. | ||
|
|
||
| #### Encrypt secrets in Ignition | ||
|
|
||
| Similar to above, we have a bootstrapping problem around fetching the key. | ||
|
Contributor
There was a problem hiding this comment. Do we? As long as the secrets aren't required to gain access to the API server, we could protect the decryption key behind the CSR process. Can you expand on this alternative?
Member
Author
There was a problem hiding this comment. It seems equivalent to the above - the pull secret is a secret so we'd need to solve all the problems there. |
||
|
|
||
| #### Improved node identity | ||
|
Contributor
There was a problem hiding this comment. I don't follow how this is an alternative to the proposal. I agree we should do more to attest, but I don't see how this protects the Ignition configs served by the MCS.
Member
Author
There was a problem hiding this comment. Added: In this model we'd change the Ignition process to require the node prove is identity, and the MCS would verify that. See https://cloud.google.com/compute/docs/instances/verifying-instance-identity#verifying |
||
|
|
||
| Docs for instance identity in GCP: https://cloud.google.com/compute/docs/instances/verifying-instance-identity | ||
| We may be able to do this in other clouds too. [This blog post](https://googleprojectzero.blogspot.com/2020/10/enter-the-vault-auth-issues-hashicorp-vault.html) discusses a vulnerability in Vault but shows an abstraction over cloud identity systems. | ||
|
|
||
| In this model we'd change the Ignition process to require the node prove is identity, by passing a signed token or equivalent and the MCS would verify that. | ||
|
|
||
| Another approach on bare metal may be using [Trusted Platform Module](https://en.wikipedia.org/wiki/Trusted_Platform_Module)s for the whole provisioning process, including [CSR approval](https://github.com/openshift/cluster-machine-approver). | ||
|
|
||
| Also related: https://kubernetes.io/docs/reference/access-authn-authz/bootstrap-tokens/ | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As much as possible, OpenShift generates secrets within the cluster so that it doesn't need to worry about securing that information at rest or over the wire. Adding this new functionality would impose a new constraint on administrators (if they want to enjoy the benefit of this mechanism): they would need to run the installation from a secure machine and they would need to securely store the pointer Ignition configs. Without a robust rotation mechanism, that's going to be a pretty tough sell.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What does this mean? Surely if the installation machine is compromised in some way then that undermines everything?
That is taken care of already in clouds - accessing the user data is a privileged operation and we block it from pods. The in-cluster pointer config is already a
secrettype.In bare metal UPI yes; we'd need to document best practices here. In most bare metal UPI the simple fix here would be to just turn off the webserver hosting the pointer config until you need to do provisioning.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Anywhere the token is stored, it needs to be secured - the chain is only as strong as its weakest link. If we are generating the token on someone's laptop, then they need to ensure that there is nothing snooping, otherwise, they cannot be sure that the token is safe. If there was a rotation mechanism, we could significantly reduce the risk for this particular vector, but it still doesn't address the other locations that this token is stored. Turning on and off a web server based on when the cluster needs to be scaled is not an option. Not only is that cumbersome and error-prone, but most customers' IT is not structured in a way that would even make this possible.