coreos · enxebre · Jan 4, 2018 · Dec 18, 2017
diff --git a/.gitignore b/.gitignore
@@ -14,3 +14,4 @@ bin/
 bin_test/
 matchbox/
 modules/update-payload/generated/
+/contrib/govcloud/vpn.conf
diff --git a/Documentation/dev/govcloud/README.md b/Documentation/dev/govcloud/README.md
@@ -0,0 +1,97 @@
+# Install Tectonic on AWS GovCloud Platform with Terraform
+
+Use this guide to manually install a Tectonic cluster on a AWS GovCloud account.
+
+## Prerequsities
+
+- **Terraform:** >= v0.10.7
+- **Tectonic Account:** Register for a [Tectonic Account](https://coreos.com/tectonic), which is free for up to 10 nodes. You must provide the cluster license and pull secret during installation.
+- **AWS GovCloud:** Obtain credentials for [GovCloud](http://docs.aws.amazon.com/govcloud-us/latest/UserGuide/govcloud-differences.html)    
+- **DNS:** The Tectonic Installer assumes that a PowerDNS server instance is running and reachable from the VPC where the cluster is running.
+
+See [contrib/govcloud](../../../contrib/govcloud) for an example of a prebuilt VPC with restricted VPN access and a PowerDNS server.
+
+## Getting Started
+
+First, clone the Tectonic Installer repository:
+
+```
+$ git clone https://github.com/coreos/tectonic-installer.git
+$ cd tectonic-installer
+```
+
+Initialise Terraform:
+
+```
+$ terraform init platforms/govcloud
+``` 
+
+Configure your AWS GovCloud credentials.
+
+```
+$ export AWS_ACCESS_KEY_ID=my-id
+$ export AWS_SECRET_ACCESS_KEY=secret-key
+```
+
+## Customize the deployment
+
+Customizations to the base installation live in examples/terraform.tfvars.govcloud. Export a variable that will be your cluster identifier:
+
+```
+$ export CLUSTER=my-cluster
+```
+
+Create a build directory to hold your customizations and copy the example file into it:
+
+```
+$ mkdir -p build/${CLUSTER}
+$ cp examples/terraform.tfvars.govcloud build/${CLUSTER}/terraform.tfvars
+```
+
+Edit the parameters with your VPC details:
+```
+tectonic_govcloud_external_vpc_id
+tectonic_govcloud_external_master_subnet_ids
+tectonic_govcloud_external_worker_subnet_ids
+tectonic_govcloud_dns_server_ip
+
+```
+
+## Deploy the cluster
+
+If you are following the [contrib/govcloud](../../../contrib/govcloud) example and deploying from an external machine, connect to the VPN now. 
+Add the `tectonic_govcloud_dns_server_ip` to your local DNS resolver.
+
+Test out the plan before deploying everything:
+
+```
+$ terraform plan -var-file=build/${CLUSTER}/terraform.tfvars platforms/govcloud
+```
+
+Next, deploy the cluster:
+
+```
+$ terraform apply -var-file=build/${CLUSTER}/terraform.tfvars platforms/govcloud
+```
+
+This should run for a little bit, and when complete, your Tectonic cluster should be ready.
+
+### Access the cluster
+
+The Tectonic Console should be up and running after the containers have downloaded. You can access it at the DNS name configured in your variables file prefixed by the cluster name, i.e ```https://cluster_name.tectonic_base_domain```.
+
+Inside of the /generated folder you should find any credentials, including the CA if generated, and a kubeconfig. You can use this to control the cluster with kubectl:
+
+```
+$ export KUBECONFIG=generated/auth/kubeconfig
+$ kubectl cluster-info
+```
+### Delete the cluster
+
+```
+$ terraform destroy -var-file=build/${CLUSTER}/terraform.tfvars platforms/govcloud
+```
+
+## Known issues and workarounds
+
+At the moment because of the [AWS user data limit](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html#instancedata-add-user-data) and [Ignition not supporting the S3 protocol for replacing the content](https://github.com/coreos/bugs/issues/2216), the Ignition config for the nodes is stored in a public bucket and it has to be removed manually.
diff --git a/Documentation/variables/govcloud.md b/Documentation/variables/govcloud.md
@@ -0,0 +1,44 @@
+<!-- DO NOT EDIT. THIS FILE IS GENERATED BY THE MAKEFILE. -->
+# Terraform variables
+This document gives an overview of variables used in the GovCloud AWS platform of the Tectonic SDK.
+
+## Inputs
+
+| Name | Description | Type | Default |
+|------|-------------|:----:|:-----:|
+| tectonic_autoscaling_group_extra_tags | (optional) Extra AWS tags to be applied to created autoscaling group resources. This is a list of maps having the keys `key`, `value` and `propagate_at_launch`.<br><br>Example: `[ { key = "foo", value = "bar", propagate_at_launch = true } ]` | list | `<list>` |
+| tectonic_dns_name | (optional) DNS prefix used to construct the console and API server endpoints. | string | `` |
+| tectonic_govcloud_assets_s3_bucket_name | (optional) Unique name under which the Amazon S3 bucket will be created. Bucket name must start with a lower case name and is limited to 63 characters. The Tectonic Installer uses the bucket to store tectonic assets and kubeconfig. If name is not provided the installer will construct the name using "tectonic_cluster_name", current AWS region and "tectonic_base_domain" | string | `` |
+| tectonic_govcloud_config_version | (internal) This declares the version of the AWS configuration variables. It has no impact on generated assets but declares the version contract of the configuration. | string | `1.0` |
+| tectonic_govcloud_dns_server_ip |  | string | - |
+| tectonic_govcloud_etcd_ec2_type | Instance size for the etcd node(s). Example: `t2.medium`. Read the [etcd recommended hardware](https://coreos.com/etcd/docs/latest/op-guide/hardware.html) guide for best performance | string | `t2.medium` |
+| tectonic_govcloud_etcd_extra_sg_ids | (optional) List of additional security group IDs for etcd nodes.<br><br>Example: `["sg-51530134", "sg-b253d7cc"]` | list | `<list>` |
+| tectonic_govcloud_etcd_root_volume_iops | The amount of provisioned IOPS for the root block device of etcd nodes. Ignored if the volume type is not io1. | string | `100` |
+| tectonic_govcloud_etcd_root_volume_size | The size of the volume in gigabytes for the root block device of etcd nodes. | string | `30` |
+| tectonic_govcloud_etcd_root_volume_type | The type of volume for the root block device of etcd nodes. | string | `gp2` |
+| tectonic_govcloud_external_master_subnet_ids | (optional) List of subnet IDs within an existing VPC to deploy master nodes into. Required to use an existing VPC and the list must match the AZ count.<br><br>Example: `["subnet-111111", "subnet-222222", "subnet-333333"]` | list | `<list>` |
+| tectonic_govcloud_external_private_zone | (optional) If set, the given Route53 zone ID will be used as the internal (private) zone. This zone will be used to create etcd DNS records as well as internal API and internal Ingress records. If set, no additional private zone will be created.<br><br>Example: `"Z1ILINNUJGTAO1"` | string | `` |
+| tectonic_govcloud_external_vpc_id | (optional) ID of an existing VPC to launch nodes into. If unset a new VPC is created.<br><br>Example: `vpc-123456` | string | `` |
+| tectonic_govcloud_external_worker_subnet_ids | (optional) List of subnet IDs within an existing VPC to deploy worker nodes into. Required to use an existing VPC and the list must match the AZ count.<br><br>Example: `["subnet-111111", "subnet-222222", "subnet-333333"]` | list | `<list>` |
+| tectonic_govcloud_extra_tags | (optional) Extra AWS tags to be applied to created resources. | map | `<map>` |
+| tectonic_govcloud_master_custom_subnets | (optional) This configures master availability zones and their corresponding subnet CIDRs directly.<br><br>Example: `{ eu-west-1a = "10.0.0.0/20", eu-west-1b = "10.0.16.0/20" }` | map | `<map>` |
+| tectonic_govcloud_master_ec2_type | Instance size for the master node(s). Example: `t2.medium`. | string | `t2.medium` |
+| tectonic_govcloud_master_extra_sg_ids | (optional) List of additional security group IDs for master nodes.<br><br>Example: `["sg-51530134", "sg-b253d7cc"]` | list | `<list>` |
+| tectonic_govcloud_master_iam_role_name | (optional) Name of IAM role to use for the instance profiles of master nodes. The name is also the last part of a role's ARN.<br><br>Example:  * Role ARN  = arn:aws:iam::123456789012:role/tectonic-installer  * Role Name = tectonic-installer | string | `` |
+| tectonic_govcloud_master_root_volume_iops | The amount of provisioned IOPS for the root block device of master nodes. Ignored if the volume type is not io1. | string | `100` |
+| tectonic_govcloud_master_root_volume_size | The size of the volume in gigabytes for the root block device of master nodes. | string | `30` |
+| tectonic_govcloud_master_root_volume_type | The type of volume for the root block device of master nodes. | string | `gp2` |
+| tectonic_govcloud_private_endpoints | (optional) If set to true, create private-facing ingress resources (ELB, A-records). If set to false, no private-facing ingress resources will be provisioned and all DNS records will be created in the public Route53 zone. | string | `true` |
+| tectonic_govcloud_profile | (optional) This declares the AWS credentials profile to use. | string | `default` |
+| tectonic_govcloud_public_endpoints | (optional) If set to true, create public-facing ingress resources (ELB, A-records). If set to false, no public-facing ingress resources will be created. | string | `false` |
+| tectonic_govcloud_ssh_key | Name of an SSH key located within the AWS region. Example: coreos-user. | string | - |
+| tectonic_govcloud_vpc_cidr_block | Block of IP addresses used by the VPC. This should not overlap with any other networks, such as a private datacenter connected via Direct Connect. | string | `10.0.0.0/16` |
+| tectonic_govcloud_worker_custom_subnets | (optional) This configures worker availability zones and their corresponding subnet CIDRs directly.<br><br>Example: `{ eu-west-1a = "10.0.64.0/20", eu-west-1b = "10.0.80.0/20" }` | map | `<map>` |
+| tectonic_govcloud_worker_ec2_type | Instance size for the worker node(s). Example: `t2.medium`. | string | `t2.medium` |
+| tectonic_govcloud_worker_extra_sg_ids | (optional) List of additional security group IDs for worker nodes.<br><br>Example: `["sg-51530134", "sg-b253d7cc"]` | list | `<list>` |
+| tectonic_govcloud_worker_iam_role_name | (optional) Name of IAM role to use for the instance profiles of worker nodes. The name is also the last part of a role's ARN.<br><br>Example:  * Role ARN  = arn:aws:iam::123456789012:role/tectonic-installer  * Role Name = tectonic-installer | string | `` |
+| tectonic_govcloud_worker_load_balancers | (optional) List of ELBs to attach all worker instances to. This is useful for exposing NodePort services via load-balancers managed separately from the cluster.<br><br>Example:  * `["ingress-nginx"]` | list | `<list>` |
+| tectonic_govcloud_worker_root_volume_iops | The amount of provisioned IOPS for the root block device of worker nodes. Ignored if the volume type is not io1. | string | `100` |
+| tectonic_govcloud_worker_root_volume_size | The size of the volume in gigabytes for the root block device of worker nodes. | string | `30` |
+| tectonic_govcloud_worker_root_volume_type | The type of volume for the root block device of worker nodes. | string | `gp2` |
+
diff --git a/Jenkinsfile b/Jenkinsfile
@@ -6,7 +6,7 @@
 3. CoreOS does not ship with `make`, so Docker builds still have to use small scripts.
 */
 
-creds = [
+commonCreds = [
   file(credentialsId: 'tectonic-license', variable: 'TF_VAR_tectonic_license_path'),
   file(credentialsId: 'tectonic-pull', variable: 'TF_VAR_tectonic_pull_secret_path'),
   file(credentialsId: 'GCP-APPLICATION', variable: 'GOOGLE_APPLICATION_CREDENTIALS'),
@@ -15,10 +15,6 @@ creds = [
     passwordVariable: 'LOG_ANALYZER_PASSWORD',
     usernameVariable: 'LOG_ANALYZER_USER'
   ),
-  [
-    $class: 'AmazonWebServicesCredentialsBinding',
-    credentialsId: 'tectonic-jenkins-installer'
-  ],
   [
     $class: 'AzureCredentialsBinding',
     credentialsId: 'azure-tectonic-test-service-principal',
@@ -34,6 +30,23 @@ creds = [
   ]
 ]
 
+creds = commonCreds.collect()
+creds.push(
+  [
+    $class: 'AmazonWebServicesCredentialsBinding',
+    credentialsId: 'tectonic-jenkins-installer'
+  ],
+)
+
+govcloudCreds = commonCreds.collect()
+govcloudCreds.push(
+    usernamePassword(
+      credentialsId: 'tectonic-jenkins-installer-govcloud',
+      passwordVariable: 'AWS_SECRET_ACCESS_KEY',
+      usernameVariable: 'AWS_ACCESS_KEY_ID'
+    )
+)
+
 quayCreds = [
   usernamePassword(
     credentialsId: 'quay-robot',
@@ -94,6 +107,11 @@ pipeline {
       defaultValue: true,
       description: ''
     )
+    booleanParam(
+      name: 'PLATFORM/GOVCLOUD',
+      defaultValue: true,
+      description: ''
+    )
     booleanParam(
       name: 'PLATFORM/AZURE',
       defaultValue: true,
@@ -283,6 +301,9 @@ pipeline {
             [file: 'ca_spec.rb', args: ''],
             [file: 'custom_tls_spec.rb', args: '']
           ]
+          def govcloud = [
+            [file: 'vpc_internal_spec.rb', args: '--device=/dev/net/tun --cap-add=NET_ADMIN -u root']
+          ]
           def azure = [
             [file: 'basic_spec.rb', args: ''],
             [file: 'private_external_spec.rb', args: '--device=/dev/net/tun --cap-add=NET_ADMIN -u root'],
@@ -310,28 +331,35 @@ pipeline {
           if (params."PLATFORM/AWS") {
             aws.each { build ->
               filepath = 'spec/aws/' + build.file
-              builds['aws/' + build.file] = runRSpecTest(filepath, build.args)
+              builds['aws/' + build.file] = runRSpecTest(filepath, build.args, creds)
+            }
+          }
+
+          if (params."PLATFORM/GOVCLOUD") {
+            govcloud.each { build ->
+              filepath = 'spec/govcloud/' + build.file
+              builds['govcloud/' + build.file] = runRSpecTest(filepath, build.args, govcloudCreds)
             }
           }
 
           if (params."PLATFORM/AZURE") {
             azure.each { build ->
               filepath = 'spec/azure/' + build.file
-              builds['azure/' + build.file] = runRSpecTest(filepath, build.args)
+              builds['azure/' + build.file] = runRSpecTest(filepath, build.args, creds)
             }
           }
 
           if (params."PLATFORM/GCP") {
             gcp.each { build ->
               filepath = 'spec/gcp/' + build.file
-              builds['gcp/' + build.file] = runRSpecTest(filepath, build.args)
+              builds['gcp/' + build.file] = runRSpecTest(filepath, build.args, creds)
             }
           }
 
           if (params."PLATFORM/BARE_METAL") {
             metal.each { build ->
               filepath = 'spec/metal/' + build.file
-              builds['metal/' + build.file] = runRSpecTestBareMetal(filepath)
+              builds['metal/' + build.file] = runRSpecTestBareMetal(filepath, creds)
             }
           }
           parallel builds
@@ -398,15 +426,15 @@ def forcefullyCleanWorkspace() {
   }
 }
 
-def runRSpecTest(testFilePath, dockerArgs) {
+def runRSpecTest(testFilePath, dockerArgs, credentials) {
   return {
     node('worker && ec2') {
       def err = null
       try {
         timeout(time: 5, unit: 'HOURS') {
           forcefullyCleanWorkspace()
           ansiColor('xterm') {
-            withCredentials(creds + quayCreds) {
+            withCredentials(credentials + quayCreds) {
               withDockerContainer(
                 image: tectonicSmokeTestEnvImage,
                 args: '-u root -v /var/run/docker.sock:/var/run/docker.sock ' + dockerArgs

diff --git a/Makefile b/Makefile
@@ -76,6 +76,10 @@ docs:
 			'This document gives an overview of variables used in the AWS platform of the Tectonic SDK.', \
 			platforms/aws/variables.tf)
 
+	$(call terraform-docs, Documentation/variables/govcloud.md, \
+			'This document gives an overview of variables used in the GovCloud AWS platform of the Tectonic SDK.', \
+			platforms/govcloud/variables.tf)
+
 	$(call terraform-docs, Documentation/variables/azure.md, \
 			'This document gives an overview of variables used in the Azure platform of the Tectonic SDK.', \
 			platforms/azure/variables.tf)
@@ -102,6 +106,10 @@ examples:
 			config.tf, \
 			platforms/aws/variables.tf)
 
+	$(call terraform-examples, examples/terraform.tfvars.govcloud, \
+			config.tf, \
+			platforms/govcloud/variables.tf)
+
 	$(call terraform-examples, \
 			examples/terraform.tfvars.azure, \
 			config.tf, \

diff --git a/contrib/govcloud/README.md b/contrib/govcloud/README.md
@@ -0,0 +1,77 @@
+# Internal Cluster
+
+This directory contains Terraform configuration that provisions a VPC with a VPN connection and a PowerDNS server in AWS GovCloud. This setup is designed to emulate a customer-like deployment in order to end-to-end test deploying Tectonic as a private "Internal Cluster" to an "Existing VPC.
+
+This Terraform configuration provisions the following AWS resources by default:
+* 1 VPC with name configured by `TF_VAR_vpc_name`
+* 4 subnets in the VPC with count configured by `TF_VAR_subnet_count`
+* 1 public subnet containing an internet gateway and NAT gateway
+* 1 private Route 53 zone for `tectonic.dev.coreos.systems` configured by `TF_VAR_base_domain`
+* 1 t2.micro EC2 Container Linux instance in the public subnet running docker containers for OpenVPN, PowerDNS and Nginx for serving OpenVPN client configuration.
+* 1 VPN gateway and VPN connection
+
+## Usage
+
+### Install Terraform
+
+[Download the Terraform binary](https://www.terraform.io/downloads.html) and install it.
+
+
+### Configure Credentials
+
+Any existing credentials available in the `~/.aws/credentials` file will automatically be used. Otherwise, make the AWS credentials available by exporting the following environment variables:
+
+```
+export AWS_ACCESS_KEY_ID=<aws-key-id>
+export AWS_SECRET_ACCESS_KEY=<aws-key-secret>
+```
+
+### Additional Variables
+
+Terraform will prompt for any unset required variables. These variables can be manually entered at every run, exported as environment variables, or configured with a [terraform.tfvars](https://www.terraform.io/docs/configuration/variables.html#variable-files) file that will be ignored by git and used for every run. Simply create a `terraform.tfvars` file and set any required variables or overrides
+
+### Running
+
+Validate the configuration and plan the run with:
+
+```
+terraform plan
+```
+
+Provision the infrastructure with:
+
+```
+terraform apply
+```
+
+### Connect to the VPN
+
+Once the infrastructure is ready, Terraform will output an `ovpn_url` variable containing the URL of the OpenVPN Access Server. In order to connect to the VPN, take the following steps:
+
+1. Navigate to `ovpn_url` and login to the Access Server with the username `openvpn` and the password provided when running Terraform.
+2. Download the OpenVPN configuration file from the Access Server.
+3. Follow the instructions for the appropriate OS to setup a VPN connection using the configuration file.
+4. When establishing the VPN connection, use the same credentials used when connecting to the Access Server. If prompted, do not provide a private key password.
+
+### Tectonic Installation
+
+Once all the infrastructure is provisioned and the VPN connection is available, a Tectonic cluster can be installed in the VPC. When running the Tectonic installer, be sure to:
+
+* Install Tectonic in the provisioned VPC by selecting the "Existing VPC" option and selecting the appropriate VPC ID in the GUI or by setting the `TF_VAR_tectonic_aws_external_vpc_id` environment variable.
+
+
+### Tear Down
+
+To tear down the infrastructure or to restart the process, run:
+
+```
+terraform destroy
+```
+
+### Troubleshooting
+
+If `terraform apply` fails, Terraform will not automatically roll back the created resources. Before attempting to create the infrastructure again, the resources must be destroyed manually by running:
+
+```
+terraform destroy
+```