[thought-experiment] single-node cluster static pod creation #302
Conversation
openshift-ci-robot
added
the
do-not-merge/hold
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: deads2k The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci-robot
added
the
approved
| We can create a new kind of render command which takes existing inputs *and* config.openshift.io resources. | ||
| Similar to how we built the original disaster recovery for certificates, we can factor the command to run the various | ||
| control loops "in order". | ||
| We can initialize our control loops using fake clients and wire listeners to synthetically update indexers backing fake | ||
| listers. | ||
| This is like we do for unit tests, only wired into the update reactors for the client. | ||
| If we separate the reactive bits of the control loops, the informer watch triggers adn the like, from the data input bits | ||
| (I think this is possible), we can have very high fidelity. | ||
| In the kube-apiserver, the ordering would like this for instance: | ||
| 1. cert-rotation - we need to create certs | ||
| 2. encryption - this would need a special mode to say: just encrypt it right away | ||
| 3. bound tokens - this creates some secrets for us | ||
| 4. static-resources - this creates targets, SAs, and stuff | ||
| 5. config observation - we need to set the operator observed config to be able to generate the final config. | ||
| 6. target config - writes the kube-apiserver configmap | ||
| 7. resource sync - copies bits from A to B | ||
| 8. loop through config observation, target config, resource sync one more time (yeah, cycles) | ||
| 9. revision controller | ||
|
|
||
| Now we do a couple neat things: | ||
| 1. Export all content from the fake clients to produce resource manifests that will be created bootkube style against | ||
| the kube-apiserver. | ||
| Someone will have grown a dependency and we know for sure that the next operator will require input from the previous one. | ||
| 2. Wire up the fake clients to our installer command. | ||
| In theory, this command will create an exact copy of the "normal" kube-apiserver static pod that we create. | ||
|
|
||
| This gives leaves supporting only one shape of static pods, which makes support of these static pods much easier. |
There was a problem hiding this comment.
Is this in theory supportable so during a 4.y release cycle we could define static versions that people can templatize (minimally) with things like on disk certs? I.e. would this "shape" to be roughly supportable with limited flexibility to change, without having to change the existing operator dramatically?
There was a problem hiding this comment.
I think our first attempt would be to get people to run this installer with an input that looks exactly like what they would use in a "real" cluster. So we would accept a manifest containing their serving cert and a manifest containing their apiserver.config.openshift.io that says how to use it.
This would allow us...
- to have one set of code managing the user input
- having a single external interface to the world instead of promising the shape of a static pod
- allow the cluster-admin to test/confirm his changes in a real cluster and take those settings as input to producing a single-node cluster.
If we start trying to allow injection of disk certs, our on-disk static pods become an API we need to support.
|
|
||
| ## Open Questions [optional] | ||
|
|
||
| 1. Do we even have a use-case for spending the time building this thought experiment? |
There was a problem hiding this comment.
The ILT is working on collecting use cases for single node clusters. Progress has been slow on my side with other commitments but we'll have this put together soon. In the absence of this, people keep wanting to take a single physical server and create a 3 node cluster w/ three VMs on the single node. The overhead of that is insane IMO. Regardless, single node clusters are on the roadmap so please spend the cycles on this.
|
/cc @arithx @LorbusChris |
| We can initialize our control loops using fake clients and wire listeners to synthetically update indexers backing fake | ||
| listers. | ||
| This is like we do for unit tests, only wired into the update reactors for the client. | ||
| If we separate the reactive bits of the control loops, the informer watch triggers adn the like, from the data input bits |
| This is like we do for unit tests, only wired into the update reactors for the client. | ||
| If we separate the reactive bits of the control loops, the informer watch triggers adn the like, from the data input bits | ||
| (I think this is possible), we can have very high fidelity. | ||
| In the kube-apiserver, the ordering would like this for instance: |
| A while back, Seth Jennings had a cool idea for trying to create a single node cluster using ignition. | ||
| The cluster would be non-configurable after "creation", non-upgradable, non-HA. | ||
| The cluster would only have etcd, kube-apiserver, kube-controller-manager, kube-scheduler. | ||
| This is a description of how we could generate supportable static pods. |
There was a problem hiding this comment.
What would be the outcome? Not really understanding the 'static pod' and 'supportability' in this context. The summary and motivation lack a justification and a why ...
|
|
||
| ## Motivation | ||
|
|
||
| Documenting a thought experiment about single node clusters. |
There was a problem hiding this comment.
Single node kubernetes clusters. Not OpenShift clusters.
There was a problem hiding this comment.
Right, I'd be curious to know why we care about having a k8s cluster which is somewhat similar to OpenShift cluster but that much. Is the goal to provide a k8s cluster which control plane is managed similarly how the OpenShift control plane or something else that I'm missing here.
There was a problem hiding this comment.
Will there be a way to start core operators of OpenShift?
|
I will be referencing this from a broader discussion document that covers an approach to minimal edge clusters using basic control plane, no-operator deployments. |
There was a problem hiding this comment.
@deads2k What is the benefit of creating a single node cluster that isn't OpenShift, doesn't have operators and isn't configurable? To me this feels like a different product but maybe I just missed the use case.
I like the idea of bootstrapping via static pods and ignition, but to make a real cluster if possible. That would cost more time though.
|
|
||
| ### Restrictions | ||
| Some things become impractical once we cannot reconfigure the kube-apiserver, they include... | ||
| 1. short lifespan of kcm and ksch client certificates - we can no longer rotate these |
There was a problem hiding this comment.
some would be rotated after expiry as the recovery cert rotation is embeded
|
Issues go stale after 90d of inactivity. Mark the issue as fresh by commenting If this issue is safe to close now please do so with /lifecycle stale |
openshift-ci-robot
added
the
lifecycle/stale
|
/remove-lifecycle stale |
openshift-ci-robot
removed
the
lifecycle/stale
|
Issues go stale after 90d of inactivity. Mark the issue as fresh by commenting If this issue is safe to close now please do so with /lifecycle stale |
openshift-ci-robot
added
the
lifecycle/stale
|
/remove-lifecycle stale |
openshift-ci-robot
removed
the
lifecycle/stale
|
Issues go stale after 90d of inactivity. Mark the issue as fresh by commenting If this issue is safe to close now please do so with /lifecycle stale |
openshift-ci
bot
added
the
lifecycle/stale
|
Stale issues rot after 30d of inactivity. Mark the issue as fresh by commenting If this issue is safe to close now please do so with /lifecycle rotten |
openshift-ci
bot
added
lifecycle/rotten
|
Rotten issues close after 30d of inactivity. Reopen the issue by commenting /close |
|
@openshift-bot: Closed this PR. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
12 participants
/hold
This is only a thought experiment. I started down the path of seeing if I could quickly build this, but the smart way to build requires a few foundational refactors and testing of fake client libraries and fake indexer techniques that haven't been proven.
Let's talk about
@smarterclayton @mfojtik @sttts @soltysh @hexfusion