Skip to content

Commit 39a926a

Browse files
cuppettwking
cuppett
authored and
wking
committed
Adding initial user doc/guide & materials for UPI AWS installation
Moved the encrypted-AMI section from "Create Cluster" to "Running Cluster", because it has more value there folks inspecting their account and what has happened after the fact. Since we call out the running instances (by count with a picture) they may wonder "where did this unique AMI come from it is running"? It goes along with some of the other explanations we chase with as well. The Create Cluster section just has the whole IPI output, but no explanation of all wizardry under the covers, calling this particular wrinkle out there seemed a bit awkward to me.
1 parent 710f9d3 commit 39a926a

File tree

10 files changed

+2059
-9
lines changed

10 files changed

+2059
-9
lines changed

‎docs/user/aws/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ This document is a guide for preparing a new AWS account for use with OpenShift.
44
will help prepare an account to create a single cluster and provide insight for adjustments which may be
55
needed for additional clusters.
66

7-
Follow along with the steps and links below to configure your AWS cluster:
7+
Follow along with the steps and links below to configure your AWS account and provision an OpenShift cluster:
88

99
1. [Route53](route53.md)
1010
2. [Limits](limits.md)

‎docs/user/aws/images/install_upi_vpc.png‎

91.3 KB
Loading

‎docs/user/aws/install.md‎

Lines changed: 16 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,15 @@
11
# Cluster Installation
22

3-
At this point, you are ready to perform the OpenShift installation outlined [here][cloud-install] and begin at
4-
Step 3: Download the Installer.
3+
At this point, you are ready to perform the OpenShift installation. You have two choices for installing your cluster on
4+
AWS, Installer-Provided Infrastructure (IPI) or User-Provided Infrastructure (UPI). See below for an example of an
5+
IPI install.
6+
7+
To see a guided example of a UPI install, please see [Install: User-Provided Infrastructure](install_upi.md)
58

6-
## Example
9+
## Example: Installer-Provided Infrastructure (IPI)
10+
11+
The steps for performing an IPI-based install are outlined [here][cloud-install]. Following this guide you may begin at
12+
Step 3: Download the Installer.
713

814
### Create Configuration
915

@@ -33,11 +39,6 @@ INFO Access the OpenShift web-console here: https://console-openshift-console.ap
3339
INFO Login to the console with user: kubeadmin, password: XXXX
3440
```
3541

36-
This creates an encrypted AMI for the bootstrap and control-plane machines.
37-
The encrypted AMI is [copied][encrypted-copy] from the AMI configured in the control-plane machine-API provider spec, which is RHCOS by default.
38-
The encryption uses the default EBS key for your target account and region (`aws kms describe-key --key-id alias/aws/ebs`).
39-
The encrypted AMI is deregistered by `destroy cluster`.
40-
4142
### Running Cluster
4243

4344
In Route53, there will be a new, private hosted zone (for internal lookups):
@@ -48,6 +49,13 @@ In EC2, there will be 6 running instances:
4849

4950
![EC2 instances after install](images/install_nodes.png)
5051

52+
The installation creates an encrypted AMI for the bootstrap and control-plane machines.
53+
The encrypted AMI is [copied][encrypted-copy] from the AMI configured in the control-plane machine-API provider spec,
54+
which is RHCOS by default.
55+
The encryption uses the default EBS key for your target account and region
56+
(`aws kms describe-key --key-id alias/aws/ebs`).
57+
The encrypted AMI is deregistered by `destroy cluster`.
58+
5159
The relationship of the EC2 instances, elastic load balancers (ELBs) and Route53 hosted zones is as depicted:
5260

5361
![Architecture relationship of ELBs and instances](images/install_nodes_elbs.png)

‎docs/user/aws/install_upi.md‎

Lines changed: 277 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,277 @@
1+
# Install: User Provided Infrastructure (UPI)
2+
3+
The steps for performing a UPI-based install are outlined here. Several [CloudFormation][cloudformation] templates are
4+
provided to assist in completing these steps or to help model your own. You are also free to create the required
5+
resources through other methods; the CloudFormation templates are just an example.
6+
7+
## Create Ignition Configs
8+
9+
The machines will be started manually. Therefore, it is required to generate the bootstrap and machine Ignition configs
10+
and store them for later steps.
11+
12+
```console
13+
$ openshift-install-linux-amd64 create ignition-configs
14+
? SSH Public Key /home/user_id/.ssh/id_rsa.pub
15+
? Platform aws
16+
? Region us-east-1
17+
? Base Domain example.com
18+
? Cluster Name openshift
19+
? Pull Secret [? for help]
20+
```
21+
22+
After running the command, several files will be available in the directory.
23+
24+
```console
25+
$ tree
26+
.
27+
├── auth
28+
│   └── kubeconfig
29+
├── bootstrap.ign
30+
├── master.ign
31+
├── metadata.json
32+
└── worker.ign
33+
```
34+
35+
## Create/Identify the VPC to be Used
36+
37+
You may create a VPC with various desirable characteristics for your situation (VPN, route tables, etc.). The
38+
VPC configuration and a CloudFormation template is provided [here](../../../upi/aws/cloudformation/01_vpc.yaml).
39+
40+
A created VPC via the template or manually should approximate a setup similar to this:
41+
42+
![Example VPC Architecture Diagram](images/install_upi_vpc.png)
43+
44+
## Create DNS entries and Load Balancers for Control Plane Components
45+
46+
The DNS and load balancer configuration within a CloudFormation template is provided
47+
[here](../../../upi/aws/cloudformation/02_cluster_infra.yaml). It uses a public hosted zone and creates a private hosted
48+
zone similar to the IPI installation method. It also creates load balancers and listeners the same way as the IPI
49+
installation method. This template can be run multiple times within a single VPC and in combination with the VPC
50+
template provided above.
51+
52+
### Optional: Manually Create Load Balancer Configuration
53+
54+
It is needed to create a TCP load balancer for ports 6443 (the Kubernetes API and its extensions) and 22623 (Ignition
55+
configurations for new machines). The targets will be the master nodes. Port 6443 must be accessible to both clients
56+
external to the cluster and nodes within the cluster. Port 22623 must be accessible to nodes within the cluster.
57+
58+
### Optional: Manually Create Route53 Hosted Zones & Records
59+
60+
For the cluster name identified earlier in [Create Ignition Configs](#create-ignition-configs), you must create a DNS
61+
entry which resolves to your created load balancer. The entry `api.$clustername.$domain` should point to the load balancer.
62+
63+
## Create Security Groups and IAM Roles
64+
65+
The security group and IAM configuration within a CloudFormation template is provided
66+
[here](../../../upi/aws/cloudformation/03_cluster_security.yaml). Run this template to get the minimal and permanent
67+
set of security groups and IAM roles needed for an operational cluster. It can also be inspected for the current
68+
set of required rules to facilitate manual creation.
69+
70+
## Launch Temporary Bootstrap Resource
71+
72+
The bootstrap launch and other necessary, temporary security group plus IAM configuration and a CloudFormation
73+
template is provided [here](../../../upi/aws/cloudformation/04_cluster_bootstrap.yaml). Upload your generated `bootstrap.ign`
74+
file to an S3 bucket in your account and run this template to get a bootstrap node along with a predictable clean up of
75+
the resources when complete. It can also be inspected for the set of required attributes via manual creation.
76+
77+
## Launch Permanent Master Nodes
78+
79+
The master launch and other necessary DNS entries for etcd are provided within a CloudFormation
80+
template [here](../../../upi/aws/cloudformation/05_cluster_master_nodes.yaml). Run this template to get three master
81+
nodes. It can also be inspected for the set of required attributes needed for manual creation of the nodes, DNS entries
82+
and load balancer configuration.
83+
84+
## Monitor for `bootstrap-complete` and Initialization
85+
86+
```console
87+
$ bin/openshift-install user-provided-infrastructure bootstrap-complete
88+
INFO Waiting up to 30m0s for the Kubernetes API at https://api.test.example.com:6443...
89+
INFO API v1.12.4+c53f462 up
90+
INFO Waiting up to 30m0s for the bootstrap-complete event...
91+
```
92+
93+
## Destroy Bootstrap Resources
94+
95+
At this point, you should delete the bootstrap resources. If using the CloudFormation template, you would [delete the
96+
stack][delete-stack] created for the bootstrap to clean up all the temporary resources.
97+
98+
## Cleanup Machine API Resources
99+
100+
By querying the Machine API, you'll notice the cluster is attempting to reconcile the predefined
101+
Machine and MachineSet definitions. We will begin to correct that here. In this step, we delete
102+
the pre-defined master nodes. Our masters are not controlled by the Machine API.
103+
104+
### Example: Deleting Master Machine Definitions
105+
106+
```console
107+
$ export KUBECONFIG=auth/kubeconfig
108+
$ oc get machines --namespace openshift-machine-api
109+
NAME INSTANCE STATE TYPE REGION ZONE AGE
110+
test-tkh7l-master-0 m4.xlarge us-east-2 us-east-2a 9m22s
111+
test-tkh7l-master-1 m4.xlarge us-east-2 us-east-2b 9m22s
112+
test-tkh7l-master-2 m4.xlarge us-east-2 us-east-2c 9m21s
113+
test-tkh7l-worker-us-east-2a-qjcxq m4.large us-east-2 us-east-2a 8m6s
114+
test-tkh7l-worker-us-east-2b-nq8zs m4.large us-east-2 us-east-2b 8m6s
115+
test-tkh7l-worker-us-east-2c-ww6c6 m4.large us-east-2 us-east-2c 8m7s
116+
$ oc delete machine --namespace openshift-machine-api test-tkh7l-master-0
117+
machine.machine.openshift.io "test-tkh7l-master-0" deleted
118+
$ oc delete machine --namespace openshift-machine-api test-tkh7l-master-1
119+
machine.machine.openshift.io "test-tkh7l-master-1" deleted
120+
$ oc delete machine --namespace openshift-machine-api test-tkh7l-master-2
121+
machine.machine.openshift.io "test-tkh7l-master-2" deleted
122+
```
123+
124+
## Launch Additional Worker Nodes
125+
126+
To launch workers, you are able to launch individual EC2 instances discretely or by automated processes outside the
127+
cluster (e.g. Auto Scaling Groups). However, you are also able to take advantage of the built in cluster scaling mechanisms
128+
and the machine API in OCP.
129+
130+
### Option 1: Dynamic Compute using Machine API
131+
132+
By default, MachineSets are created and will have failed to launch. We can correct the desired subnet filter,
133+
target security group, RHEL CoreOS AMI and EC2 instance profile.
134+
135+
```console
136+
$ oc get machinesets --namespace openshift-machine-api
137+
NAME DESIRED CURRENT READY AVAILABLE AGE
138+
test-tkh7l-worker-us-east-2a 1 1 11m
139+
test-tkh7l-worker-us-east-2b 1 1 11m
140+
test-tkh7l-worker-us-east-2c 1 1 11m
141+
```
142+
143+
```console
144+
$ oc get machineset --namespace openshift-machine-api test-tkh7l-worker-us-east-2a -o yaml
145+
apiVersion: machine.openshift.io/v1beta1
146+
kind: MachineSet
147+
metadata:
148+
creationTimestamp: 2019-03-14T14:03:03Z
149+
generation: 1
150+
labels:
151+
machine.openshift.io/cluster-api-cluster: test-tkh7l
152+
machine.openshift.io/cluster-api-machine-role: worker
153+
machine.openshift.io/cluster-api-machine-type: worker
154+
name: test-tkh7l-worker-us-east-2a
155+
namespace: openshift-machine-api
156+
resourceVersion: "2350"
157+
selfLink: /apis/machine.openshift.io/v1beta1/namespaces/openshift-machine-api/machinesets/test-tkh7l-worker-us-east-2a
158+
uid: e2a6c8a6-4661-11e9-a9b0-0296069fd3a2
159+
spec:
160+
replicas: 1
161+
selector:
162+
matchLabels:
163+
machine.openshift.io/cluster-api-cluster: test-tkh7l
164+
machine.openshift.io/cluster-api-machineset: test-tkh7l-worker-us-east-2a
165+
template:
166+
metadata:
167+
creationTimestamp: null
168+
labels:
169+
machine.openshift.io/cluster-api-cluster: test-tkh7l
170+
machine.openshift.io/cluster-api-machine-role: worker
171+
machine.openshift.io/cluster-api-machine-type: worker
172+
machine.openshift.io/cluster-api-machineset: test-tkh7l-worker-us-east-2a
173+
spec:
174+
metadata:
175+
creationTimestamp: null
176+
providerSpec:
177+
value:
178+
ami:
179+
id: ami-0eecbb884c8b35b1e
180+
apiVersion: awsproviderconfig.openshift.io/v1beta1
181+
blockDevices:
182+
- ebs:
183+
iops: 0
184+
volumeSize: 120
185+
volumeType: gp2
186+
credentialsSecret:
187+
name: aws-cloud-credentials
188+
deviceIndex: 0
189+
iamInstanceProfile:
190+
id: test-tkh7l-worker-profile
191+
instanceType: m4.large
192+
kind: AWSMachineProviderConfig
193+
metadata:
194+
creationTimestamp: null
195+
placement:
196+
availabilityZone: us-east-2a
197+
region: us-east-2
198+
publicIp: null
199+
securityGroups:
200+
- filters:
201+
- name: tag:Name
202+
values:
203+
- test-tkh7l-worker-sg
204+
subnet:
205+
filters:
206+
- name: tag:Name
207+
values:
208+
- test-tkh7l-private-us-east-2a
209+
tags:
210+
- name: kubernetes.io/cluster/test-tkh7l
211+
value: owned
212+
userDataSecret:
213+
name: worker-user-data
214+
versions:
215+
kubelet: ""
216+
status:
217+
fullyLabeledReplicas: 1
218+
observedGeneration: 1
219+
replicas: 1
220+
```
221+
222+
At this point, you'd edit the YAML to update the relevant values to match your UPI installation.
223+
224+
```console
225+
$ oc edit machineset --namespace openshift-machine-api test-tkh7l-worker-us-east-2a
226+
machineset.machine.openshift.io/test-tkh7l-worker-us-east-2a edited
227+
```
228+
229+
Once the Machine API has a chance to reconcile and begin launching hosts with the correct attributes, you
230+
should start to see new output in your EC2 console and oc commands.
231+
232+
```console
233+
$ oc get machines --namespace openshift-machine-api
234+
NAME INSTANCE STATE TYPE REGION ZONE AGE
235+
test-tkh7l-worker-us-east-2a-hxlqn i-0e7f3a52b2919471e pending m4.4xlarge us-east-2 us-east-2a 3s
236+
```
237+
238+
### Option 2: Manually Launching Worker Instances
239+
240+
The worker launch is provided within a CloudFormation template [here](../../../upi/aws/cloudformation/06_cluster_worker_node.yaml).
241+
You can launch a CloudFormation stack to manage each individual worker. A similar launch configuration could be used by
242+
outside automation or AWS auto scaling groups.
243+
244+
## Configure Router for UPI DNS
245+
246+
TODO: Identify changes needed to Router or Ingress for DNS `*.apps` registration or LoadBalancer creation.
247+
248+
## Monitor for Cluster Completion
249+
250+
```console
251+
$ bin/openshift-install user-provided-infrastructure finish
252+
INFO Waiting up to 30m0s for the cluster to initialize...
253+
```
254+
255+
Also, you can observe the running state of your cluster pods:
256+
257+
```console
258+
$ oc get pods --all-namespaces
259+
NAMESPACE NAME READY STATUS RESTARTS AGE
260+
kube-system etcd-member-ip-10-0-3-111.us-east-2.compute.internal 1/1 Running 0 35m
261+
kube-system etcd-member-ip-10-0-3-239.us-east-2.compute.internal 1/1 Running 0 37m
262+
kube-system etcd-member-ip-10-0-3-24.us-east-2.compute.internal 1/1 Running 0 35m
263+
openshift-apiserver-operator openshift-apiserver-operator-6d6674f4f4-h7t2t 1/1 Running 1 37m
264+
openshift-apiserver apiserver-fm48r 1/1 Running 0 30m
265+
openshift-apiserver apiserver-fxkvv 1/1 Running 0 29m
266+
openshift-apiserver apiserver-q85nm 1/1 Running 0 29m
267+
...
268+
openshift-service-ca-operator openshift-service-ca-operator-66ff6dc6cd-9r257 1/1 Running 0 37m
269+
openshift-service-ca apiservice-cabundle-injector-695b6bcbc-cl5hm 1/1 Running 0 35m
270+
openshift-service-ca configmap-cabundle-injector-8498544d7-25qn6 1/1 Running 0 35m
271+
openshift-service-ca service-serving-cert-signer-6445fc9c6-wqdqn 1/1 Running 0 35m
272+
openshift-service-catalog-apiserver-operator openshift-service-catalog-apiserver-operator-549f44668b-b5q2w 1/1 Running 0 32m
273+
openshift-service-catalog-controller-manager-operator openshift-service-catalog-controller-manager-operator-b78cr2lnm 1/1 Running 0 31m
274+
```
275+
276+
[cloudformation]: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/Welcome.html
277+
[delete-stack]: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-console-delete-stack.html

0 commit comments

Comments
 (0)