From 44794bbef18f66a2de28175a3ea752c1a036efb9 Mon Sep 17 00:00:00 2001 From: Abhinav Dahiya Date: Tue, 14 Aug 2018 12:08:54 -0700 Subject: [PATCH] Documentation/design: drop docs regarding phases --- Documentation/design/initphase.md | 44 --------- Documentation/design/launchphase.md | 90 ------------------ Documentation/design/preparephase.md | 93 ------------------ Documentation/design/renderphase.md | 136 --------------------------- Documentation/design/workflow.md | 3 - Documentation/design/workflow.svg | 1 - 6 files changed, 367 deletions(-) delete mode 100644 Documentation/design/initphase.md delete mode 100644 Documentation/design/launchphase.md delete mode 100644 Documentation/design/preparephase.md delete mode 100644 Documentation/design/renderphase.md delete mode 100644 Documentation/design/workflow.md delete mode 100644 Documentation/design/workflow.svg diff --git a/Documentation/design/initphase.md b/Documentation/design/initphase.md deleted file mode 100644 index 17960910ebf..00000000000 --- a/Documentation/design/initphase.md +++ /dev/null @@ -1,44 +0,0 @@ -# Init Phase - -## Goals - -1. Generates `InstallConfig` by prompting user for required parameters, and populating the rest of configuration with default values. -2. Make useful decisions on default values for most options. -3. Init phase should never prompt the user for more than 10 options unless there is a very strong reason. - -## Non Goals - -1. Supporting all the configuration option through init is not a goal. Init is just a helper for humans. - -## Overview - -The output of the `init` command is a config file that describes a default cluster for the given provider, and is valid/immediately usable. - -The `init` step is not required, this is merely a helper to generate a valid `InstallConfig`. A user could use examples from documentation, roll their own by hand, or re-use existing configs from source control. - -## Detailed Design - -### Prompts on CLI vs flags vs Environment variables - -1. Flags don’t work well here because we want the fast-path to be single command. -2. Init is only valuable for humans and environment variables are similarly tedious. - -Init phase should use prompts on CLI for top-level configurations that are required to generate `InstallConfig`. - -### Remember user's answers - -In case where users runs `installer init` and stops after platform prompt for some reason. Should the init command ask for platform again when it is rerun. - -The number of prompts from users need to be very few and remembering user's answer is unnecessary optimization. - -### Idempotency - -Re-running the `installer init` prompts the user for options again and writes the `install-config.yaml` to disk, overwriting the old `install-config.yaml` file in the directory. - -### Output - -Init phase generates `install-config.yaml` file that contains `InstallConfig` object in the current working directory or directory specified by `--asset-dir` flag. - -## TODOs - -1. Add examples for user prompts for each platform and the corresponding `install-config.yaml`. diff --git a/Documentation/design/launchphase.md b/Documentation/design/launchphase.md deleted file mode 100644 index 89d47d9c80c..00000000000 --- a/Documentation/design/launchphase.md +++ /dev/null @@ -1,90 +0,0 @@ -# Launch - -## Goals - -1. Validate credentials for cloud. -2. Create a functional cluster. -3. Validate the cluster is ready to be used by an user. - -## Overview - -The launch phase creates a cluster based on the assets generated in the prepare phase. The launch performs operations, - -1. Pre-flight checks: Verify that credentials being passed to the installer are sufficient to bring up the cluster. -2. Launch all the platform specific resources required for the cluster. For example in AWS this includes, DNS in Route53, VPCs, ELBs, IAM roles, Security groups etc. -3. Bootstrap the cluster. -4. Wait for the cluster to be ready for use by an user. - -## Detailed design - -### Idempotency - -Launch is **NOT** idempotent. Re-running the `installer launch` command should exit with a failure. - -### Pre flight checks - -### Platform Specific checks - -1. AWS - - The launch phase performs following checks - - * Credentials are sufficient to create all the resources. - * If VPC / Route53 zones were supplied, ensure they are valid. - * TODO: add more - -2. Libvirt - - The launch phase performs following checks - - * Ensure QEMU URI is reachable - * OS image path is valid. - * TODO: add more - -### Launch platform-specific resources - -Use terraform to create resources. - -### Bootstrapping cluster - -**[Various options are discussed here](https://docs.google.com/document/d/17sTJ1mdWtPTFkaHLENX2aeEYw1-o4aDS2NbWnZRgOlY/edit#heading=h.r9how0eg6txs)** *This link might be private* - -The goal is to eliminate external coordination steps and keep the "special" steps constrained to a single throw-away node. - -a. Launch a bootstrap node from `bootstrap.ign` that was generated in prepare step. The user-data of the bootstrap node either contains the `bootstrap.ign` or points to a remote location that contains the `bootstrap.ign` file. - -b. Launch 3 master nodes with ign endpoint as `api.example.com` - -c. Launch ELB or equivalent resource that fronts bootstrap node, and masters - -d. ALIAS `api.example.com` → ELB - -e. Start bootstrap MachineConfig Server to serve ignition for the masters. Also start the `etcd-signer-server` for serving TLS assets for the etcd members. - -f. When etcd on masters has formed quorum, stop local MachineConfig Server and the `etcd-signer-server`. - -g. Run bootkube to launch self-hosted control-plane. - - * Bootkube starts `boostrap-*` static pods on boostrap node to create a temporary control plane. - * Then bootkube uses the temporary control plane to bootstrap the self-hosted control plane. - * When bootkube validates that the self-hosted plane is ready, it shuts down. - * When bootkube shuts down, it tears down bootstrap-apiserver (this triggers fail on ELB healthcheck, and bootstrap node is removed from backend pool). - -h. Destroy bootstrap node and destroy the remote location if ign for bootstrap node was served from remote endpoint as it stores secrets. - -### Bootstrapping etcd - -Etcd is co-located with master nodes. Therefore the bootstrap MachineConfig Server serves the ignition file with the etcd static pods. But the etcd nodes need TLS assets to communicate with each other. - -1. The boostrap node runs a [`etcd-signer-server`](https://github.com/coreos/kubecsr/tree/master/cmd/kube-etcd-signer-server) docker container which mimics the kube-apiserver's `CertificateSigningRequest` endpoint. -2. etcd systemd-service has a `PreStartHook` defined that runs a [`etcd-client`](https://github.com/coreos/kubecsr/tree/master/cmd/kube-client-agent). The `etcd-client` reaches out to the API server endpoint, currently being served by `etcd-signer-server`, for certificates. -3. After the `PreStartHook` succeeds, each etcd member has all the TLS assets for creating the etcd cluster. - -### Verify cluster is ready - -The launch step needs to exit only when the cluster is ready for use by an user. - -TODO: need more info to decide when cluster is up. - -a. Is installer done when control-plane is ready. -b. Installer is ready when all the second-level operators report `Done` condition. \ No newline at end of file diff --git a/Documentation/design/preparephase.md b/Documentation/design/preparephase.md deleted file mode 100644 index 7a45215ff34..00000000000 --- a/Documentation/design/preparephase.md +++ /dev/null @@ -1,93 +0,0 @@ -# Prepare - -## Goal - -1. Generation of the final Ignition configurations for the master and bootstrap nodes. - -2. Prepare needs to validate that the assets required to bootstrap the cluster are present and correct. - -## Overview - -The prepare step is responsible for taking all of the generated assets, including any user-customizations, and generating the final asset state that will be used by the infrastructure provisioning tool. - -This step is necessary because when a user modifies assets after the `render` phase - we must consume them into the final cluster states that will be applied at installation. - -In most cases, prepare should only be responsible for generating a bootstrap node ignition config, a master node ignition stub, and provisioning tool specific files (e.g. tfvars). All other asset generation should be the result of the `render` step - -## Detailed Design - -### Idempotency - -Prepare should be able to be run multiple times, with the output simply replacing any previously generated state. This is to allow a user to modify their source assets, and easily re-generate the configuration that will be used by the installer. - -### Validation - -Prepare should validate that assets required for bootstrap are present. For example, - -1. etcd-cert-signer-server on bootstrap node. -2. machineconfig server on bootstrap node. -3. bootkube service for bootstrap node. -4. kubelet service file for bootstrap node. and maybe more.. - -### Extra assets added by user - -The user can customize the assets generated by `render` step. The user can also add new assets that need to be installed in the cluster. Prepare step should allow addition of assets by user. - -The type of additions that are allowed are: - -1. Assets like systemd-service files for bootstrap node. -2. Kubernetes objects to be installed in the cluster. - -The type of additions not supported are: - -1. Assets like systemd-service files for master and worker nodes. - -For example, - -If the directory before running prepare looks like: - -``` -./install-config.yaml -./auth/kubeconfig-admin -./auth/kubeconfig-bootstrap -./manifests/kube-core-operator.yaml -./manifests/kube-core-config.yaml -./manifests/kube-core-operator-sa.yaml -./manifests/network-operator.yaml -./manifests/network-config.yaml -./manifests/network-operator-sa.yaml -... -./tls/root-ca.crt -./tls/root-ca.key -./tls/kube-ca.crt -./tls/kube-ca.key -``` - -The user can edit the `./manifests/network-config.yaml` file to update network related configuration. - -### Output - -The Prepare step creates ignition file for bootstrap, master and worker node namely, `bootstrap.ign`, `master.ign` and `worker.ign` in the current working directory or the directory specified by `--assets-dir` flag. - -The directory after prepare would look like - -``` -./install-config.yaml -./auth/kubeconfig-admin -./auth/kubeconfig-bootstrap -./ign/bootstrap.ign -./ign/master.ign -./ign/worker.ign -./manifests/kube-core-operator.yaml -./manifests/kube-core-config.yaml -./manifests/kube-core-operator-sa.yaml -... -./tls/root-ca.crt -./tls/root-ca.key -./tls/kube-ca.crt -./tls/kube-ca.key -``` - -#### TODO - -1. The prepare might have to create some `tf files` for launching the cluster with terraform. \ No newline at end of file diff --git a/Documentation/design/renderphase.md b/Documentation/design/renderphase.md deleted file mode 100644 index 2d349ec49c9..00000000000 --- a/Documentation/design/renderphase.md +++ /dev/null @@ -1,136 +0,0 @@ -# Render - -## Goals - -1. Render is not required to be run separately. This is exposed as a means of allowing users a deeper entry point for install-time customization. - -2. This renders all the assets necessary to bring up a cluster on to disk. This allows users to customize assets without over-parameterizing the `InstallConfig`. - -3. Allow user to commit the assets that describe a cluster. - -## Non-goals - -1. Render is incapable of allowing users to customize cluster objects beyond those already supported by the object's operator. - -## Overview - -Render has 3 steps: - -1. Validate `InstallConfig` object. If `install-config.yaml` file is not present fallback to init phase to generate `InstallConfig` object. - -2. Generate all the required TLS assets for the cluster. - -3. Use `operator render self` interface to generate the cluster objects for all the operators. - -Render puts all the assets on disk for user customization or next phase of the installer. - -## Detailed design - -### Idempotency - -Re-running the `installer render` renders the manifests to disk again, overwriting the older manifests in the directory. - -### Input - -Render phase accepts a valid `InstallConfig` object that exists at `install-config.yaml` in the current directory or the directory specified by `--assets-dir` flag. - -### InstallConfig Validation - -Render needs to perform 3 kinds of validation: - -1. Validate configuration options are known. e.g. networking provider is known. - -2. Platform configuration validation. This also includes validating all node pools belong to the specified platform. - -3. Ensure `master` node pool exists and has valid configuration. - -### TLS asset generation - -Generate all the TLS assets required for the cluster and their corresponding Kubernetes secrets. All this needs to be go code rather than terraform templates. - -Some of the TLS assets that need to be generated are: -1. root-ca -2. kube-ca -3. aggregator-ca -4. etcd-ca -5. apiserver -6. ingress and more - -### Rendering operator manifests - -1. Historically, the installer had manifests for all the cluster objects that are required to run the operator. For example, deployment manifest for the operator, service account, roles/rolebindings and operator config etc. So changes to the installer were required to test any change in the operator. This meant that the first indication if the operator had broken install was after the operator changes were merged in its repository and then operator was updated in the installer repository. - -2. Also, all the cluster objects required by the operator to functions had to be managed by the installer team. - -Therefore, to allow operator teams to iterate and control all aspects of an operator, the installer needs a process to delegate the `InstallConfig` object to each operator in render phase to generate iteself and all other cluster objects necessary for it to function correctly. - -#### Prerequisites - -1. Render cannot assume container runtime on host. -2. Render needs to be successfully on (at least) Linux and Darwin systems. - -#### Solutions - -1. Vendor operators - - One way to achieve rendering is to vendor operator's code into installer and hook operator defined render function in installer's Render phase. - - Pros: - - * No extra work for cross-platform compile for various operators. - * Single simple go binary for installer. - - Cons: - - * On operator repository, CI needs to update the toml file and run re-vendor to run install test in their PR. - -2. Git submodules - - Installer repo uses git submodules to import operators and then asserts **build and render interfaces** between installer and operator to hook operator into installer's Render phase. - - Installer uses a build interface to build all the operators as binaries at build time and then packages them along side the installer binary. The render step uses the shipped operator binary to render manifests using the render interface. - - Pros: - - * Installer repo is smaller. - * CI is much easier to work as it doesn't have to deal with go vendoring and only git submodules to replace/update the operator submodules during testing. - - Cons: - - * All operator repositories must be capable of building cross-platform binaries. Previously the operators have only built Linux binaries. - -**TODO** Add the preferred solution or other approaches. - -### Output - -The render phase renders all the assets into following directories in the current working directory or the directory specified by `--asset-dir` flag. - -#### auth - -This directory has assets that can be used to authenticate with the cluster for various purposes. For example, `kubeconfig` with admin privileges will exist in this directory. - -#### manifests - -This directory consists of all the manifests rendered by each operator and the secrets generated from TLS assets. - -### tls - -This directory consists of all the TLS assets generated by the installer. - -**IMPORTANT:** Some TLS assets for example, the `root-ca.key` will be written to disk and should be not be stored in insecure/public repositories. - -### Example render output - -``` -./install-config.yaml -./auth/kubeconfig-admin -./auth/kubeconfig-bootstrap -./manifests/kube-core-operator.yaml -./manifests/kube-core-config.yaml -./manifests/kube-core-operator-sa.yaml -... -./tls/root-ca.crt -./tls/root-ca.key -./tls/kube-ca.crt -./tls/kube-ca.key -``` diff --git a/Documentation/design/workflow.md b/Documentation/design/workflow.md deleted file mode 100644 index 221b65965d1..00000000000 --- a/Documentation/design/workflow.md +++ /dev/null @@ -1,3 +0,0 @@ -# High-level Install Workflow Diagram - -![High-level Install Workflow diagram](./workflow.svg) diff --git a/Documentation/design/workflow.svg b/Documentation/design/workflow.svg deleted file mode 100644 index e78cdc2dfec..00000000000 --- a/Documentation/design/workflow.svg +++ /dev/null @@ -1 +0,0 @@ - \ No newline at end of file