diff --git a/.gitignore b/.gitignore index 51af9db8c5..f49f14676f 100644 --- a/.gitignore +++ b/.gitignore @@ -14,3 +14,4 @@ bin/ bin_test/ matchbox/ modules/update-payload/generated/ +/contrib/govcloud/vpn.conf diff --git a/Documentation/dev/govcloud/README.md b/Documentation/dev/govcloud/README.md new file mode 100644 index 0000000000..963b37a1cd --- /dev/null +++ b/Documentation/dev/govcloud/README.md @@ -0,0 +1,97 @@ +# Install Tectonic on AWS GovCloud Platform with Terraform + +Use this guide to manually install a Tectonic cluster on a AWS GovCloud account. + +## Prerequsities + +- **Terraform:** >= v0.10.7 +- **Tectonic Account:** Register for a [Tectonic Account](https://coreos.com/tectonic), which is free for up to 10 nodes. You must provide the cluster license and pull secret during installation. +- **AWS GovCloud:** Obtain credentials for [GovCloud](http://docs.aws.amazon.com/govcloud-us/latest/UserGuide/govcloud-differences.html) +- **DNS:** The Tectonic Installer assumes that a PowerDNS server instance is running and reachable from the VPC where the cluster is running. + +See [contrib/govcloud](../../../contrib/govcloud) for an example of a prebuilt VPC with restricted VPN access and a PowerDNS server. + +## Getting Started + +First, clone the Tectonic Installer repository: + +``` +$ git clone https://github.com/coreos/tectonic-installer.git +$ cd tectonic-installer +``` + +Initialise Terraform: + +``` +$ terraform init platforms/govcloud +``` + +Configure your AWS GovCloud credentials. + +``` +$ export AWS_ACCESS_KEY_ID=my-id +$ export AWS_SECRET_ACCESS_KEY=secret-key +``` + +## Customize the deployment + +Customizations to the base installation live in examples/terraform.tfvars.govcloud. Export a variable that will be your cluster identifier: + +``` +$ export CLUSTER=my-cluster +``` + +Create a build directory to hold your customizations and copy the example file into it: + +``` +$ mkdir -p build/${CLUSTER} +$ cp examples/terraform.tfvars.govcloud build/${CLUSTER}/terraform.tfvars +``` + +Edit the parameters with your VPC details: +``` +tectonic_govcloud_external_vpc_id +tectonic_govcloud_external_master_subnet_ids +tectonic_govcloud_external_worker_subnet_ids +tectonic_govcloud_dns_server_ip + +``` + +## Deploy the cluster + +If you are following the [contrib/govcloud](../../../contrib/govcloud) example and deploying from an external machine, connect to the VPN now. +Add the `tectonic_govcloud_dns_server_ip` to your local DNS resolver. + +Test out the plan before deploying everything: + +``` +$ terraform plan -var-file=build/${CLUSTER}/terraform.tfvars platforms/govcloud +``` + +Next, deploy the cluster: + +``` +$ terraform apply -var-file=build/${CLUSTER}/terraform.tfvars platforms/govcloud +``` + +This should run for a little bit, and when complete, your Tectonic cluster should be ready. + +### Access the cluster + +The Tectonic Console should be up and running after the containers have downloaded. You can access it at the DNS name configured in your variables file prefixed by the cluster name, i.e ```https://cluster_name.tectonic_base_domain```. + +Inside of the /generated folder you should find any credentials, including the CA if generated, and a kubeconfig. You can use this to control the cluster with kubectl: + +``` +$ export KUBECONFIG=generated/auth/kubeconfig +$ kubectl cluster-info +``` +### Delete the cluster + +``` +$ terraform destroy -var-file=build/${CLUSTER}/terraform.tfvars platforms/govcloud +``` + +## Known issues and workarounds + +At the moment because of the [AWS user data limit](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html#instancedata-add-user-data) and [Ignition not supporting the S3 protocol for replacing the content](https://github.com/coreos/bugs/issues/2216), the Ignition config for the nodes is stored in a public bucket and it has to be removed manually. diff --git a/Documentation/variables/govcloud.md b/Documentation/variables/govcloud.md new file mode 100644 index 0000000000..01c49f8ad2 --- /dev/null +++ b/Documentation/variables/govcloud.md @@ -0,0 +1,44 @@ + +# Terraform variables +This document gives an overview of variables used in the GovCloud AWS platform of the Tectonic SDK. + +## Inputs + +| Name | Description | Type | Default | +|------|-------------|:----:|:-----:| +| tectonic_autoscaling_group_extra_tags | (optional) Extra AWS tags to be applied to created autoscaling group resources. This is a list of maps having the keys `key`, `value` and `propagate_at_launch`.

Example: `[ { key = "foo", value = "bar", propagate_at_launch = true } ]` | list | `` | +| tectonic_dns_name | (optional) DNS prefix used to construct the console and API server endpoints. | string | `` | +| tectonic_govcloud_assets_s3_bucket_name | (optional) Unique name under which the Amazon S3 bucket will be created. Bucket name must start with a lower case name and is limited to 63 characters. The Tectonic Installer uses the bucket to store tectonic assets and kubeconfig. If name is not provided the installer will construct the name using "tectonic_cluster_name", current AWS region and "tectonic_base_domain" | string | `` | +| tectonic_govcloud_config_version | (internal) This declares the version of the AWS configuration variables. It has no impact on generated assets but declares the version contract of the configuration. | string | `1.0` | +| tectonic_govcloud_dns_server_ip | | string | - | +| tectonic_govcloud_etcd_ec2_type | Instance size for the etcd node(s). Example: `t2.medium`. Read the [etcd recommended hardware](https://coreos.com/etcd/docs/latest/op-guide/hardware.html) guide for best performance | string | `t2.medium` | +| tectonic_govcloud_etcd_extra_sg_ids | (optional) List of additional security group IDs for etcd nodes.

Example: `["sg-51530134", "sg-b253d7cc"]` | list | `` | +| tectonic_govcloud_etcd_root_volume_iops | The amount of provisioned IOPS for the root block device of etcd nodes. Ignored if the volume type is not io1. | string | `100` | +| tectonic_govcloud_etcd_root_volume_size | The size of the volume in gigabytes for the root block device of etcd nodes. | string | `30` | +| tectonic_govcloud_etcd_root_volume_type | The type of volume for the root block device of etcd nodes. | string | `gp2` | +| tectonic_govcloud_external_master_subnet_ids | (optional) List of subnet IDs within an existing VPC to deploy master nodes into. Required to use an existing VPC and the list must match the AZ count.

Example: `["subnet-111111", "subnet-222222", "subnet-333333"]` | list | `` | +| tectonic_govcloud_external_private_zone | (optional) If set, the given Route53 zone ID will be used as the internal (private) zone. This zone will be used to create etcd DNS records as well as internal API and internal Ingress records. If set, no additional private zone will be created.

Example: `"Z1ILINNUJGTAO1"` | string | `` | +| tectonic_govcloud_external_vpc_id | (optional) ID of an existing VPC to launch nodes into. If unset a new VPC is created.

Example: `vpc-123456` | string | `` | +| tectonic_govcloud_external_worker_subnet_ids | (optional) List of subnet IDs within an existing VPC to deploy worker nodes into. Required to use an existing VPC and the list must match the AZ count.

Example: `["subnet-111111", "subnet-222222", "subnet-333333"]` | list | `` | +| tectonic_govcloud_extra_tags | (optional) Extra AWS tags to be applied to created resources. | map | `` | +| tectonic_govcloud_master_custom_subnets | (optional) This configures master availability zones and their corresponding subnet CIDRs directly.

Example: `{ eu-west-1a = "10.0.0.0/20", eu-west-1b = "10.0.16.0/20" }` | map | `` | +| tectonic_govcloud_master_ec2_type | Instance size for the master node(s). Example: `t2.medium`. | string | `t2.medium` | +| tectonic_govcloud_master_extra_sg_ids | (optional) List of additional security group IDs for master nodes.

Example: `["sg-51530134", "sg-b253d7cc"]` | list | `` | +| tectonic_govcloud_master_iam_role_name | (optional) Name of IAM role to use for the instance profiles of master nodes. The name is also the last part of a role's ARN.

Example: * Role ARN = arn:aws:iam::123456789012:role/tectonic-installer * Role Name = tectonic-installer | string | `` | +| tectonic_govcloud_master_root_volume_iops | The amount of provisioned IOPS for the root block device of master nodes. Ignored if the volume type is not io1. | string | `100` | +| tectonic_govcloud_master_root_volume_size | The size of the volume in gigabytes for the root block device of master nodes. | string | `30` | +| tectonic_govcloud_master_root_volume_type | The type of volume for the root block device of master nodes. | string | `gp2` | +| tectonic_govcloud_private_endpoints | (optional) If set to true, create private-facing ingress resources (ELB, A-records). If set to false, no private-facing ingress resources will be provisioned and all DNS records will be created in the public Route53 zone. | string | `true` | +| tectonic_govcloud_profile | (optional) This declares the AWS credentials profile to use. | string | `default` | +| tectonic_govcloud_public_endpoints | (optional) If set to true, create public-facing ingress resources (ELB, A-records). If set to false, no public-facing ingress resources will be created. | string | `false` | +| tectonic_govcloud_ssh_key | Name of an SSH key located within the AWS region. Example: coreos-user. | string | - | +| tectonic_govcloud_vpc_cidr_block | Block of IP addresses used by the VPC. This should not overlap with any other networks, such as a private datacenter connected via Direct Connect. | string | `10.0.0.0/16` | +| tectonic_govcloud_worker_custom_subnets | (optional) This configures worker availability zones and their corresponding subnet CIDRs directly.

Example: `{ eu-west-1a = "10.0.64.0/20", eu-west-1b = "10.0.80.0/20" }` | map | `` | +| tectonic_govcloud_worker_ec2_type | Instance size for the worker node(s). Example: `t2.medium`. | string | `t2.medium` | +| tectonic_govcloud_worker_extra_sg_ids | (optional) List of additional security group IDs for worker nodes.

Example: `["sg-51530134", "sg-b253d7cc"]` | list | `` | +| tectonic_govcloud_worker_iam_role_name | (optional) Name of IAM role to use for the instance profiles of worker nodes. The name is also the last part of a role's ARN.

Example: * Role ARN = arn:aws:iam::123456789012:role/tectonic-installer * Role Name = tectonic-installer | string | `` | +| tectonic_govcloud_worker_load_balancers | (optional) List of ELBs to attach all worker instances to. This is useful for exposing NodePort services via load-balancers managed separately from the cluster.

Example: * `["ingress-nginx"]` | list | `` | +| tectonic_govcloud_worker_root_volume_iops | The amount of provisioned IOPS for the root block device of worker nodes. Ignored if the volume type is not io1. | string | `100` | +| tectonic_govcloud_worker_root_volume_size | The size of the volume in gigabytes for the root block device of worker nodes. | string | `30` | +| tectonic_govcloud_worker_root_volume_type | The type of volume for the root block device of worker nodes. | string | `gp2` | + diff --git a/Jenkinsfile b/Jenkinsfile index c8ebd40d4f..bd19c9d619 100644 --- a/Jenkinsfile +++ b/Jenkinsfile @@ -6,7 +6,7 @@ 3. CoreOS does not ship with `make`, so Docker builds still have to use small scripts. */ -creds = [ +commonCreds = [ file(credentialsId: 'tectonic-license', variable: 'TF_VAR_tectonic_license_path'), file(credentialsId: 'tectonic-pull', variable: 'TF_VAR_tectonic_pull_secret_path'), file(credentialsId: 'GCP-APPLICATION', variable: 'GOOGLE_APPLICATION_CREDENTIALS'), @@ -15,10 +15,6 @@ creds = [ passwordVariable: 'LOG_ANALYZER_PASSWORD', usernameVariable: 'LOG_ANALYZER_USER' ), - [ - $class: 'AmazonWebServicesCredentialsBinding', - credentialsId: 'tectonic-jenkins-installer' - ], [ $class: 'AzureCredentialsBinding', credentialsId: 'azure-tectonic-test-service-principal', @@ -34,6 +30,23 @@ creds = [ ] ] +creds = commonCreds.collect() +creds.push( + [ + $class: 'AmazonWebServicesCredentialsBinding', + credentialsId: 'tectonic-jenkins-installer' + ], +) + +govcloudCreds = commonCreds.collect() +govcloudCreds.push( + usernamePassword( + credentialsId: 'tectonic-jenkins-installer-govcloud', + passwordVariable: 'AWS_SECRET_ACCESS_KEY', + usernameVariable: 'AWS_ACCESS_KEY_ID' + ) +) + quayCreds = [ usernamePassword( credentialsId: 'quay-robot', @@ -94,6 +107,11 @@ pipeline { defaultValue: true, description: '' ) + booleanParam( + name: 'PLATFORM/GOVCLOUD', + defaultValue: true, + description: '' + ) booleanParam( name: 'PLATFORM/AZURE', defaultValue: true, @@ -283,6 +301,9 @@ pipeline { [file: 'ca_spec.rb', args: ''], [file: 'custom_tls_spec.rb', args: ''] ] + def govcloud = [ + [file: 'vpc_internal_spec.rb', args: '--device=/dev/net/tun --cap-add=NET_ADMIN -u root'] + ] def azure = [ [file: 'basic_spec.rb', args: ''], [file: 'private_external_spec.rb', args: '--device=/dev/net/tun --cap-add=NET_ADMIN -u root'], @@ -310,28 +331,35 @@ pipeline { if (params."PLATFORM/AWS") { aws.each { build -> filepath = 'spec/aws/' + build.file - builds['aws/' + build.file] = runRSpecTest(filepath, build.args) + builds['aws/' + build.file] = runRSpecTest(filepath, build.args, creds) + } + } + + if (params."PLATFORM/GOVCLOUD") { + govcloud.each { build -> + filepath = 'spec/govcloud/' + build.file + builds['govcloud/' + build.file] = runRSpecTest(filepath, build.args, govcloudCreds) } } if (params."PLATFORM/AZURE") { azure.each { build -> filepath = 'spec/azure/' + build.file - builds['azure/' + build.file] = runRSpecTest(filepath, build.args) + builds['azure/' + build.file] = runRSpecTest(filepath, build.args, creds) } } if (params."PLATFORM/GCP") { gcp.each { build -> filepath = 'spec/gcp/' + build.file - builds['gcp/' + build.file] = runRSpecTest(filepath, build.args) + builds['gcp/' + build.file] = runRSpecTest(filepath, build.args, creds) } } if (params."PLATFORM/BARE_METAL") { metal.each { build -> filepath = 'spec/metal/' + build.file - builds['metal/' + build.file] = runRSpecTestBareMetal(filepath) + builds['metal/' + build.file] = runRSpecTestBareMetal(filepath, creds) } } parallel builds @@ -398,7 +426,7 @@ def forcefullyCleanWorkspace() { } } -def runRSpecTest(testFilePath, dockerArgs) { +def runRSpecTest(testFilePath, dockerArgs, credentials) { return { node('worker && ec2') { def err = null @@ -406,7 +434,7 @@ def runRSpecTest(testFilePath, dockerArgs) { timeout(time: 5, unit: 'HOURS') { forcefullyCleanWorkspace() ansiColor('xterm') { - withCredentials(creds + quayCreds) { + withCredentials(credentials + quayCreds) { withDockerContainer( image: tectonicSmokeTestEnvImage, args: '-u root -v /var/run/docker.sock:/var/run/docker.sock ' + dockerArgs diff --git a/Makefile b/Makefile index eaccdf5a68..e47f848e41 100644 --- a/Makefile +++ b/Makefile @@ -76,6 +76,10 @@ docs: 'This document gives an overview of variables used in the AWS platform of the Tectonic SDK.', \ platforms/aws/variables.tf) + $(call terraform-docs, Documentation/variables/govcloud.md, \ + 'This document gives an overview of variables used in the GovCloud AWS platform of the Tectonic SDK.', \ + platforms/govcloud/variables.tf) + $(call terraform-docs, Documentation/variables/azure.md, \ 'This document gives an overview of variables used in the Azure platform of the Tectonic SDK.', \ platforms/azure/variables.tf) @@ -102,6 +106,10 @@ examples: config.tf, \ platforms/aws/variables.tf) + $(call terraform-examples, examples/terraform.tfvars.govcloud, \ + config.tf, \ + platforms/govcloud/variables.tf) + $(call terraform-examples, \ examples/terraform.tfvars.azure, \ config.tf, \ diff --git a/contrib/govcloud/README.md b/contrib/govcloud/README.md new file mode 100644 index 0000000000..e9104bd0cf --- /dev/null +++ b/contrib/govcloud/README.md @@ -0,0 +1,77 @@ +# Internal Cluster + +This directory contains Terraform configuration that provisions a VPC with a VPN connection and a PowerDNS server in AWS GovCloud. This setup is designed to emulate a customer-like deployment in order to end-to-end test deploying Tectonic as a private "Internal Cluster" to an "Existing VPC. + +This Terraform configuration provisions the following AWS resources by default: +* 1 VPC with name configured by `TF_VAR_vpc_name` +* 4 subnets in the VPC with count configured by `TF_VAR_subnet_count` +* 1 public subnet containing an internet gateway and NAT gateway +* 1 private Route 53 zone for `tectonic.dev.coreos.systems` configured by `TF_VAR_base_domain` +* 1 t2.micro EC2 Container Linux instance in the public subnet running docker containers for OpenVPN, PowerDNS and Nginx for serving OpenVPN client configuration. +* 1 VPN gateway and VPN connection + +## Usage + +### Install Terraform + +[Download the Terraform binary](https://www.terraform.io/downloads.html) and install it. + + +### Configure Credentials + +Any existing credentials available in the `~/.aws/credentials` file will automatically be used. Otherwise, make the AWS credentials available by exporting the following environment variables: + +``` +export AWS_ACCESS_KEY_ID= +export AWS_SECRET_ACCESS_KEY= +``` + +### Additional Variables + +Terraform will prompt for any unset required variables. These variables can be manually entered at every run, exported as environment variables, or configured with a [terraform.tfvars](https://www.terraform.io/docs/configuration/variables.html#variable-files) file that will be ignored by git and used for every run. Simply create a `terraform.tfvars` file and set any required variables or overrides + +### Running + +Validate the configuration and plan the run with: + +``` +terraform plan +``` + +Provision the infrastructure with: + +``` +terraform apply +``` + +### Connect to the VPN + +Once the infrastructure is ready, Terraform will output an `ovpn_url` variable containing the URL of the OpenVPN Access Server. In order to connect to the VPN, take the following steps: + +1. Navigate to `ovpn_url` and login to the Access Server with the username `openvpn` and the password provided when running Terraform. +2. Download the OpenVPN configuration file from the Access Server. +3. Follow the instructions for the appropriate OS to setup a VPN connection using the configuration file. +4. When establishing the VPN connection, use the same credentials used when connecting to the Access Server. If prompted, do not provide a private key password. + +### Tectonic Installation + +Once all the infrastructure is provisioned and the VPN connection is available, a Tectonic cluster can be installed in the VPC. When running the Tectonic installer, be sure to: + +* Install Tectonic in the provisioned VPC by selecting the "Existing VPC" option and selecting the appropriate VPC ID in the GUI or by setting the `TF_VAR_tectonic_aws_external_vpc_id` environment variable. + + +### Tear Down + +To tear down the infrastructure or to restart the process, run: + +``` +terraform destroy +``` + +### Troubleshooting + +If `terraform apply` fails, Terraform will not automatically roll back the created resources. Before attempting to create the infrastructure again, the resources must be destroyed manually by running: + +``` +terraform destroy +``` diff --git a/contrib/govcloud/main.tf b/contrib/govcloud/main.tf new file mode 100644 index 0000000000..f883566559 --- /dev/null +++ b/contrib/govcloud/main.tf @@ -0,0 +1,150 @@ +provider "aws" { + region = "${var.vpc_aws_region}" +} + +# Declare the data source +data "aws_availability_zones" "available" {} + +resource "aws_vpc" "vpc" { + cidr_block = "${var.vpc_cidr}" + enable_dns_support = true + enable_dns_hostnames = true + + tags { + Name = "${var.vpc_name}" + } +} + +data "aws_ami" "coreos_ami" { + most_recent = true + + filter { + name = "name" + values = ["CoreOS-stable-*"] + } + + filter { + name = "architecture" + values = ["x86_64"] + } + + filter { + name = "virtualization-type" + values = ["hvm"] + } + + filter { + name = "owner-id" + values = ["190570271432"] + } +} + +resource "aws_instance" "bastion" { + # 1st available AZ + availability_zone = "${data.aws_availability_zones.available.names[0]}" + ami = "${data.aws_ami.coreos_ami.image_id}" + instance_type = "t2.micro" + subnet_id = "${aws_subnet.pub_subnet_generic.id}" + vpc_security_group_ids = ["${compact(concat(list(aws_security_group.powerdns.id), list(aws_security_group.vpn_sg.id)))}"] + source_dest_check = false + key_name = "${var.ssh_key}" + user_data = "${data.template_file.services.rendered}" + + depends_on = ["aws_eip.ovpn_eip"] + + tags { + Name = "${var.vpc_name}-server" + } +} + +data "template_file" "services" { + template = "${file("${path.module}/services.sh")}" + + vars { + ip = "${aws_eip.ovpn_eip.public_ip}" + dns_zone = "${var.base_domain}" + private_cidr = "${cidrsubnet(var.vpc_cidr, 6, 25)}" + username = "${var.nginx_username}" + password = "${var.nginx_password}" + } +} + +# IGW +resource "aws_internet_gateway" "igw" { + vpc_id = "${aws_vpc.vpc.id}" + + tags { + Name = "${var.vpc_name}-igw" + } +} + +# General purpose public subnet. used for OVPN access and IGW/NAT attachment. +resource "aws_subnet" "pub_subnet_generic" { + vpc_id = "${aws_vpc.vpc.id}" + + # 1st available AZ + availability_zone = "${data.aws_availability_zones.available.names[0]}" + cidr_block = "10.0.255.0/24" + map_public_ip_on_launch = true + + tags { + Name = "${var.vpc_name}-vpn" + } +} + +resource "aws_route_table_association" "pub_subnet_generic" { + subnet_id = "${aws_subnet.pub_subnet_generic.id}" + route_table_id = "${aws_route_table.pub_rt.id}" +} + +# public subnet route table +resource "aws_route_table" "pub_rt" { + vpc_id = "${aws_vpc.vpc.id}" + + route { + cidr_block = "0.0.0.0/0" + gateway_id = "${aws_internet_gateway.igw.id}" + } + + tags { + Name = "${var.vpc_name}-public" + } +} + +# private subnets +resource "aws_subnet" "priv_subnet" { + count = "${var.subnet_count}" + vpc_id = "${aws_vpc.vpc.id}" + availability_zone = "${element(data.aws_availability_zones.available.names, count.index)}" + cidr_block = "${cidrsubnet(var.vpc_cidr, 8, count.index + 100)}" + + tags { + Name = "${var.vpc_name}-${count.index}" + } +} + +resource "aws_route_table_association" "priv_subnet" { + count = "${var.subnet_count}" + subnet_id = "${aws_subnet.priv_subnet.*.id[count.index]}" + route_table_id = "${aws_route_table.priv_rt.id}" +} + +# private subnet route table +resource "aws_route_table" "priv_rt" { + vpc_id = "${aws_vpc.vpc.id}" + propagating_vgws = ["${aws_vpn_gateway.vpg.id}"] + + route { + cidr_block = "0.0.0.0/0" + instance_id = "${aws_instance.bastion.id}" + } + + route { + cidr_block = "${var.local_network_cidr}" + gateway_id = "${aws_vpn_gateway.vpg.id}" + } + + tags { + Name = "${var.vpc_name}-private" + } +} diff --git a/contrib/govcloud/powerdns.tf b/contrib/govcloud/powerdns.tf new file mode 100644 index 0000000000..d651a67daf --- /dev/null +++ b/contrib/govcloud/powerdns.tf @@ -0,0 +1,74 @@ +# Security Group +resource "aws_security_group" "powerdns" { + vpc_id = "${aws_vpc.vpc.id}" +} + +resource "aws_security_group_rule" "powerdns_egress" { + type = "egress" + security_group_id = "${aws_security_group.powerdns.id}" + + from_port = 0 + to_port = 0 + protocol = "-1" + cidr_blocks = ["0.0.0.0/0"] +} + +resource "aws_security_group_rule" "powerdns_ingress_ssh" { + type = "ingress" + security_group_id = "${aws_security_group.powerdns.id}" + + protocol = "tcp" + cidr_blocks = ["0.0.0.0/0"] + from_port = 22 + to_port = 22 +} + +resource "aws_security_group_rule" "powerdns_ingress_http" { + type = "ingress" + security_group_id = "${aws_security_group.powerdns.id}" + + protocol = "tcp" + cidr_blocks = ["0.0.0.0/0"] + from_port = 80 + to_port = 80 +} + +resource "aws_security_group_rule" "powerdns_ingress_https" { + type = "ingress" + security_group_id = "${aws_security_group.powerdns.id}" + + protocol = "tcp" + cidr_blocks = ["0.0.0.0/0"] + from_port = 443 + to_port = 443 +} + +resource "aws_security_group_rule" "powerdns_ingress_api" { + type = "ingress" + security_group_id = "${aws_security_group.powerdns.id}" + + protocol = "tcp" + cidr_blocks = ["0.0.0.0/0"] + from_port = 8081 + to_port = 8081 +} + +resource "aws_security_group_rule" "powerdns_ingress_dns" { + type = "ingress" + security_group_id = "${aws_security_group.powerdns.id}" + + protocol = "tcp" + cidr_blocks = ["0.0.0.0/0"] + from_port = 53 + to_port = 53 +} + +resource "aws_security_group_rule" "powerdns_ingress_dns_udp" { + type = "ingress" + security_group_id = "${aws_security_group.powerdns.id}" + + protocol = "udp" + cidr_blocks = ["0.0.0.0/0"] + from_port = 53 + to_port = 53 +} diff --git a/contrib/govcloud/services.sh b/contrib/govcloud/services.sh new file mode 100644 index 0000000000..8cb3f6b9ac --- /dev/null +++ b/contrib/govcloud/services.sh @@ -0,0 +1,66 @@ +#!/bin/bash +# TODO: Convert to Ignition + Systemd + +# OpenVPN https://github.com/kylemanna/docker-openvpn +OVPN_DATA=basic-data +CLIENT=CI +IMG=kylemanna/openvpn:2.4 +SERV_IP=${ip} +CLIENT_DIR=/home/core/vpn-config +mkdir -p $CLIENT_DIR +docker run -v $OVPN_DATA:/etc/openvpn --rm $IMG ovpn_genconfig -u udp://$SERV_IP +docker run -v $OVPN_DATA:/etc/openvpn --rm -e "EASYRSA_BATCH=1" -e "EASYRSA_REQ_CN=CI Test CA" $IMG ovpn_initpki nopass +docker run -v $OVPN_DATA:/etc/openvpn --rm $IMG easyrsa build-client-full $CLIENT nopass +docker run -v $OVPN_DATA:/etc/openvpn --rm $IMG ovpn_getclient $CLIENT | tee $CLIENT_DIR/config.ovpn +docker run --restart always --name "ovpn-test" -v $OVPN_DATA:/etc/openvpn -d -p 1194:1194/udp --cap-add=NET_ADMIN $IMG + +# Web server for OpenVPN client Config +NGINX_DIR=/home/core/nginx-config +mkdir -p $NGINX_DIR +echo " +server { + listen 80; + server_name localhost; + auth_basic 'Administrators Area'; + auth_basic_user_file /etc/nginx/conf.d/htpasswd; + location / { + root /usr/share/nginx/html; + index config.ovpn; + } +} +" > $NGINX_DIR/default.conf + +printf "${username}:$(openssl passwd -crypt ${password})\n" >> $NGINX_DIR/htpasswd +docker run -d --restart always \ +--name nginx-app \ +--net host \ +-v $CLIENT_DIR:/usr/share/nginx/html/ \ +-v $NGINX_DIR:/etc/nginx/conf.d/ \ +nginx:1.13.7-alpine + +# PowerDNS +docker run -d --name=mysql --net=host -e MYSQL_ROOT_PASSWORD=powerdns mysql +sleep 5 +docker run --net=host \ +--name pdns-master -d \ +--restart always \ +-e PDNS_RECURSOR=10.0.0.2 \ +-e PDNS_SOA=10.0.0.2 \ +-e PDNS_ALLOW_AXFR_IPS=127.0.0.1 \ +-e PDNS_DISTRIBUTOR_THREADS=3 \ +-e PDNS_CACHE_TTL=20 \ +-e PDNS_RECURSIVE_CACHE_TTL=10 \ +-e DB_ENV_MYSQL_ROOT_PASSWORD=powerdns \ +-e MYSQL_HOST=127.0.0.1 \ +-e MYSQL_PORT="3306" \ +-e PDNS_ZONE=${dns_zone} \ +quay.io/nicholas_lane/pdns:4.0-1 +sleep 20 +docker exec pdns-master pdnsutil create-zone ${dns_zone} + +# Nat Gateway routing +echo 1 > /proc/sys/net/ipv4/ip_forward +iptables -t nat -A POSTROUTING -s ${private_cidr} -j MASQUERADE + +# Disable automatic updates +systemctl stop update-engine diff --git a/contrib/govcloud/vars.tf b/contrib/govcloud/vars.tf new file mode 100644 index 0000000000..1f2c01ffe9 --- /dev/null +++ b/contrib/govcloud/vars.tf @@ -0,0 +1,67 @@ +# placeholders for access_key / secret_key +# should be fed through env var or variable file +# https://www.terraform.io/docs/configuration/variables.html + +variable vpc_name { + description = "The name of the VPC to identify created resources." + default = "bastion" +} + +variable base_domain { + default = "tectonic-ci.de" + description = "The base domain for this cluster's FQDN" +} + +variable vpc_aws_region { + description = "The target AWS region for the cluster" + default = "us-gov-west-1" +} + +variable vpc_cidr { + default = "10.0.0.0/16" + description = "The CIDR range used for your entire VPC" +} + +variable subnet_count { + default = 4 + description = "Number of private subnets to pre-create" +} + +variable local_network_cidr { + default = "10.7.0.0/16" + description = "IP range in the network your laptop is on (dosn't actually matter unless your instances need to connect to the local network your laptop is on)" +} + +variable "ssh_key" { + default = "" +} + +variable "nginx_username" { + description = "Used for retrieving the OpenVPN client config file." + default = "user" +} + +variable "nginx_password" { + description = "Used for retrieving the OpenVPN client config file." + default = "password" +} + +output "ovpn_url" { + value = "http://${aws_eip.ovpn_eip.public_ip}" +} + +output "base_domain" { + value = "${var.base_domain}" +} + +output "vpc_id" { + value = "${aws_vpc.vpc.id}" +} + +output "vpc_dns" { + value = "${aws_instance.bastion.private_ip}" +} + +output "subnets" { + value = "${aws_subnet.priv_subnet.*.id}" +} diff --git a/contrib/govcloud/vpn.tf b/contrib/govcloud/vpn.tf new file mode 100644 index 0000000000..16238c7ce3 --- /dev/null +++ b/contrib/govcloud/vpn.tf @@ -0,0 +1,108 @@ +resource "aws_vpn_gateway" "vpg" { + vpc_id = "${aws_vpc.vpc.id}" + + # 1st available AZ + availability_zone = "${data.aws_availability_zones.available.names[0]}" + + tags { + Name = "${var.vpc_name}-vpg" + } +} + +resource "aws_eip" "ovpn_eip" { + vpc = true +} + +resource "aws_eip_association" "vpn_eip_assoc" { + instance_id = "${aws_instance.bastion.id}" + allocation_id = "${aws_eip.ovpn_eip.id}" +} + +resource "aws_security_group" "vpn_sg" { + name = "ovpn-server-sg" + description = "Allow all inbound traffic" + vpc_id = "${aws_vpc.vpc.id}" + + ingress { + from_port = 0 + to_port = 65535 + protocol = "tcp" + cidr_blocks = ["${cidrsubnet(var.vpc_cidr, 6, 25)}"] + } + + ingress { + from_port = 943 + to_port = 943 + protocol = "tcp" + cidr_blocks = ["0.0.0.0/0"] + } + + ingress { + from_port = 22 + to_port = 22 + protocol = "tcp" + cidr_blocks = ["0.0.0.0/0"] + } + + ingress { + from_port = 80 + to_port = 80 + protocol = "tcp" + cidr_blocks = ["0.0.0.0/0"] + } + + ingress { + from_port = 443 + to_port = 443 + protocol = "tcp" + cidr_blocks = ["0.0.0.0/0"] + } + + ingress { + from_port = 943 + to_port = 943 + protocol = "tcp" + cidr_blocks = ["0.0.0.0/0"] + } + + ingress { + from_port = 1194 + to_port = 1194 + protocol = "udp" + cidr_blocks = ["0.0.0.0/0"] + } + + egress { + from_port = 0 + to_port = 0 + + # all + protocol = "-1" + cidr_blocks = ["0.0.0.0/0"] + } + + tags { + Name = "${var.vpc_name}-ovpn-sg" + } +} + +resource "aws_customer_gateway" "customer_gateway" { + bgp_asn = 65000 + ip_address = "${aws_eip.ovpn_eip.public_ip}" + type = "ipsec.1" + + tags { + Name = "${var.vpc_name}-customer-gateway" + } +} + +resource "aws_vpn_connection" "main" { + vpn_gateway_id = "${aws_vpn_gateway.vpg.id}" + customer_gateway_id = "${aws_customer_gateway.customer_gateway.id}" + type = "ipsec.1" + static_routes_only = true + + tags { + Name = "${var.vpc_name}-vpn-conn" + } +} diff --git a/contrib/internal-cluster/README.md b/contrib/internal-cluster/README.md index 700f5a5c41..2c4da575a8 100644 --- a/contrib/internal-cluster/README.md +++ b/contrib/internal-cluster/README.md @@ -59,7 +59,7 @@ Terraform does not support changing SOA TTLs in Route 53. As a result, in order ### Tectonic Installation -Once all the infrastructure is provisioned and the VPN connection is available, a Tectonic clusyou can be installed the VPC. When running the Tectonic installer, be sure to: +Once all the infrastructure is provisioned and the VPN connection is available, a Tectonic cluster can be installed in the VPC. When running the Tectonic installer, be sure to: * Select the provisioned private DNS Zone using the GUI or by setting the `TF_VAR_tectonic_aws_external_private_zone` environment variable. * Install Tectonic in the provisioned VPC by selecting the "Existing VPC" option and selecting the appropriate VPC ID in the GUI or by setting the `TF_VAR_tectonic_aws_external_vpc_id` environment variable. diff --git a/examples/terraform.tfvars.govcloud b/examples/terraform.tfvars.govcloud new file mode 100644 index 0000000000..126a3ed41d --- /dev/null +++ b/examples/terraform.tfvars.govcloud @@ -0,0 +1,317 @@ + +// (optional) Extra AWS tags to be applied to created autoscaling group resources. +// This is a list of maps having the keys `key`, `value` and `propagate_at_launch`. +// +// Example: `[ { key = "foo", value = "bar", propagate_at_launch = true } ]` +// tectonic_autoscaling_group_extra_tags = "" + +// The base DNS domain of the cluster. It must NOT contain a trailing period. Some +// DNS providers will automatically add this if necessary. +// +// Example: `openstack.dev.coreos.systems`. +// +// Note: This field MUST be set manually prior to creating the cluster. +// This applies only to cloud platforms. +// +// [Azure-specific NOTE] +// To use Azure-provided DNS, `tectonic_base_domain` should be set to `""` +// If using DNS records, ensure that `tectonic_base_domain` is set to a properly configured external DNS zone. +// Instructions for configuring delegated domains for Azure DNS can be found here: https://docs.microsoft.com/en-us/azure/dns/dns-delegate-domain-azure-dns +tectonic_base_domain = "" + +// (optional) The content of the PEM-encoded CA certificate, used to generate Tectonic Console's server certificate. +// If left blank, a CA certificate will be automatically generated. +// tectonic_ca_cert = "" + +// (optional) The content of the PEM-encoded CA key, used to generate Tectonic Console's server certificate. +// This field is mandatory if `tectonic_ca_cert` is set. +// tectonic_ca_key = "" + +// (optional) The algorithm used to generate tectonic_ca_key. +// The default value is currently recommended. +// This field is mandatory if `tectonic_ca_cert` is set. +// tectonic_ca_key_alg = "RSA" + +// (optional) This declares the IP range to assign Kubernetes pod IPs in CIDR notation. +// tectonic_cluster_cidr = "10.2.0.0/16" + +// The name of the cluster. +// If used in a cloud-environment, this will be prepended to `tectonic_base_domain` resulting in the URL to the Tectonic console. +// +// Note: This field MUST be set manually prior to creating the cluster. +// Warning: Special characters in the name like '.' may cause errors on OpenStack platforms due to resource name constraints. +tectonic_cluster_name = "" + +// (optional) The Container Linux update channel. +// +// Examples: `stable`, `beta`, `alpha` +// tectonic_container_linux_channel = "stable" + +// The Container Linux version to use. Set to `latest` to select the latest available version for the selected update channel. +// +// Examples: `latest`, `1465.6.0` +tectonic_container_linux_version = "latest" + +// (optional) A list of PEM encoded CA files that will be installed in /etc/ssl/certs on etcd, master, and worker nodes. +// tectonic_custom_ca_pem_list = "" + +// (optional) This only applies if you use the modules/dns/ddns module. +// +// Specifies the RFC2136 Dynamic DNS server key algorithm. +// tectonic_ddns_key_algorithm = "" + +// (optional) This only applies if you use the modules/dns/ddns module. +// +// Specifies the RFC2136 Dynamic DNS server key name. +// tectonic_ddns_key_name = "" + +// (optional) This only applies if you use the modules/dns/ddns module. +// +// Specifies the RFC2136 Dynamic DNS server key secret. +// tectonic_ddns_key_secret = "" + +// (optional) This only applies if you use the modules/dns/ddns module. +// +// Specifies the RFC2136 Dynamic DNS server IP/host to register IP addresses to. +// tectonic_ddns_server = "" + +// (optional) DNS prefix used to construct the console and API server endpoints. +// tectonic_dns_name = "" + +// (optional) The size in MB of the PersistentVolume used for handling etcd backups. +// tectonic_etcd_backup_size = "512" + +// (optional) The name of an existing Kubernetes StorageClass that will be used for handling etcd backups. +// tectonic_etcd_backup_storage_class = "" + +// (optional) The path of the file containing the CA certificate for TLS communication with etcd. +// +// Note: This works only when used in conjunction with an external etcd cluster. +// If set, the variable `tectonic_etcd_servers` must also be set. +// tectonic_etcd_ca_cert_path = "/dev/null" + +// (optional) The path of the file containing the client certificate for TLS communication with etcd. +// +// Note: This works only when used in conjunction with an external etcd cluster. +// If set, the variables `tectonic_etcd_servers`, `tectonic_etcd_ca_cert_path`, and `tectonic_etcd_client_key_path` must also be set. +// tectonic_etcd_client_cert_path = "/dev/null" + +// (optional) The path of the file containing the client key for TLS communication with etcd. +// +// Note: This works only when used in conjunction with an external etcd cluster. +// If set, the variables `tectonic_etcd_servers`, `tectonic_etcd_ca_cert_path`, and `tectonic_etcd_client_cert_path` must also be set. +// tectonic_etcd_client_key_path = "/dev/null" + +// The number of etcd nodes to be created. +// If set to zero, the count of etcd nodes will be determined automatically. +// +// Note: This is not supported on bare metal. +tectonic_etcd_count = "0" + +// (optional) List of external etcd v3 servers to connect with (hostnames/IPs only). +// Needs to be set if using an external etcd cluster. +// Note: If this variable is defined, the installer will not create self-signed certs. +// To provide a CA certificate to trust the etcd servers, set "tectonic_etcd_ca_cert_path". +// +// Example: `["etcd1", "etcd2", "etcd3"]` +// tectonic_etcd_servers = "" + +// (optional) If set to `true`, all etcd endpoints will be configured to use the "https" scheme. +// +// Note: If `tectonic_experimental` is set to `true` this variable has no effect, because the experimental self-hosted etcd always uses TLS. +// tectonic_etcd_tls_enabled = true + +// (optional) Unique name under which the Amazon S3 bucket will be created. Bucket name must start with a lower case name and is limited to 63 characters. +// The Tectonic Installer uses the bucket to store tectonic assets and kubeconfig. +// If name is not provided the installer will construct the name using "tectonic_cluster_name", current AWS region and "tectonic_base_domain" +// tectonic_govcloud_assets_s3_bucket_name = "" + +// +tectonic_govcloud_dns_server_ip = "" + +// Instance size for the etcd node(s). Example: `t2.medium`. Read the [etcd recommended hardware](https://coreos.com/etcd/docs/latest/op-guide/hardware.html) guide for best performance +tectonic_govcloud_etcd_ec2_type = "t2.medium" + +// (optional) List of additional security group IDs for etcd nodes. +// +// Example: `["sg-51530134", "sg-b253d7cc"]` +// tectonic_govcloud_etcd_extra_sg_ids = "" + +// The amount of provisioned IOPS for the root block device of etcd nodes. +// Ignored if the volume type is not io1. +tectonic_govcloud_etcd_root_volume_iops = "100" + +// The size of the volume in gigabytes for the root block device of etcd nodes. +tectonic_govcloud_etcd_root_volume_size = "30" + +// The type of volume for the root block device of etcd nodes. +tectonic_govcloud_etcd_root_volume_type = "gp2" + +// (optional) List of subnet IDs within an existing VPC to deploy master nodes into. +// Required to use an existing VPC and the list must match the AZ count. +// +// Example: `["subnet-111111", "subnet-222222", "subnet-333333"]` +// tectonic_govcloud_external_master_subnet_ids = "" + +// (optional) If set, the given Route53 zone ID will be used as the internal (private) zone. +// This zone will be used to create etcd DNS records as well as internal API and internal Ingress records. +// If set, no additional private zone will be created. +// +// Example: `"Z1ILINNUJGTAO1"` +// tectonic_govcloud_external_private_zone = "" + +// (optional) ID of an existing VPC to launch nodes into. +// If unset a new VPC is created. +// +// Example: `vpc-123456` +// tectonic_govcloud_external_vpc_id = "" + +// (optional) List of subnet IDs within an existing VPC to deploy worker nodes into. +// Required to use an existing VPC and the list must match the AZ count. +// +// Example: `["subnet-111111", "subnet-222222", "subnet-333333"]` +// tectonic_govcloud_external_worker_subnet_ids = "" + +// (optional) Extra AWS tags to be applied to created resources. +// tectonic_govcloud_extra_tags = "" + +// (optional) This configures master availability zones and their corresponding subnet CIDRs directly. +// +// Example: +// `{ eu-west-1a = "10.0.0.0/20", eu-west-1b = "10.0.16.0/20" }` +// tectonic_govcloud_master_custom_subnets = "" + +// Instance size for the master node(s). Example: `t2.medium`. +tectonic_govcloud_master_ec2_type = "t2.medium" + +// (optional) List of additional security group IDs for master nodes. +// +// Example: `["sg-51530134", "sg-b253d7cc"]` +// tectonic_govcloud_master_extra_sg_ids = "" + +// (optional) Name of IAM role to use for the instance profiles of master nodes. +// The name is also the last part of a role's ARN. +// +// Example: +// * Role ARN = arn:aws:iam::123456789012:role/tectonic-installer +// * Role Name = tectonic-installer +// tectonic_govcloud_master_iam_role_name = "" + +// The amount of provisioned IOPS for the root block device of master nodes. +// Ignored if the volume type is not io1. +tectonic_govcloud_master_root_volume_iops = "100" + +// The size of the volume in gigabytes for the root block device of master nodes. +tectonic_govcloud_master_root_volume_size = "30" + +// The type of volume for the root block device of master nodes. +tectonic_govcloud_master_root_volume_type = "gp2" + +// (optional) If set to true, create private-facing ingress resources (ELB, A-records). +// If set to false, no private-facing ingress resources will be provisioned and all DNS records will be created in the public Route53 zone. +// tectonic_govcloud_private_endpoints = true + +// (optional) This declares the AWS credentials profile to use. +// tectonic_govcloud_profile = "default" + +// (optional) If set to true, create public-facing ingress resources (ELB, A-records). +// If set to false, no public-facing ingress resources will be created. +// tectonic_govcloud_public_endpoints = false + +// Name of an SSH key located within the AWS region. Example: coreos-user. +tectonic_govcloud_ssh_key = "" + +// Block of IP addresses used by the VPC. +// This should not overlap with any other networks, such as a private datacenter connected via Direct Connect. +tectonic_govcloud_vpc_cidr_block = "10.0.0.0/16" + +// (optional) This configures worker availability zones and their corresponding subnet CIDRs directly. +// +// Example: `{ eu-west-1a = "10.0.64.0/20", eu-west-1b = "10.0.80.0/20" }` +// tectonic_govcloud_worker_custom_subnets = "" + +// Instance size for the worker node(s). Example: `t2.medium`. +tectonic_govcloud_worker_ec2_type = "t2.medium" + +// (optional) List of additional security group IDs for worker nodes. +// +// Example: `["sg-51530134", "sg-b253d7cc"]` +// tectonic_govcloud_worker_extra_sg_ids = "" + +// (optional) Name of IAM role to use for the instance profiles of worker nodes. +// The name is also the last part of a role's ARN. +// +// Example: +// * Role ARN = arn:aws:iam::123456789012:role/tectonic-installer +// * Role Name = tectonic-installer +// tectonic_govcloud_worker_iam_role_name = "" + +// (optional) List of ELBs to attach all worker instances to. +// This is useful for exposing NodePort services via load-balancers managed separately from the cluster. +// +// Example: +// * `["ingress-nginx"]` +// tectonic_govcloud_worker_load_balancers = "" + +// The amount of provisioned IOPS for the root block device of worker nodes. +// Ignored if the volume type is not io1. +tectonic_govcloud_worker_root_volume_iops = "100" + +// The size of the volume in gigabytes for the root block device of worker nodes. +tectonic_govcloud_worker_root_volume_size = "30" + +// The type of volume for the root block device of worker nodes. +tectonic_govcloud_worker_root_volume_type = "gp2" + +// The path to the tectonic licence file. +// You can download the Tectonic license file from your Account overview page at [1]. +// +// [1] https://account.coreos.com/overview +// +// Note: This field MUST be set manually prior to creating the cluster unless `tectonic_vanilla_k8s` is set to `true`. +tectonic_license_path = "" + +// The number of master nodes to be created. +// This applies only to cloud platforms. +tectonic_master_count = "1" + +// (optional) Configures the network to be used in Tectonic. One of the following values can be used: +// +// - "flannel": enables overlay networking only. This is implemented by flannel using VXLAN. +// +// - "canal": [ALPHA] enables overlay networking including network policy. Overlay is implemented by flannel using VXLAN. Network policy is implemented by Calico. +// +// - "calico": [ALPHA] enables BGP based networking. Routing and network policy is implemented by Calico. Note this has been tested on baremetal installations only. +// +// - "none": disables the installation of any Pod level networking layer provided by Tectonic. By setting this value, users are expected to deploy their own solution to enable network connectivity for Pods and Services. +// tectonic_networking = "flannel" + +// The path the pull secret file in JSON format. +// This is known to be a "Docker pull secret" as produced by the docker login [1] command. +// A sample JSON content is shown in [2]. +// You can download the pull secret from your Account overview page at [3]. +// +// [1] https://docs.docker.com/engine/reference/commandline/login/ +// +// [2] https://coreos.com/os/docs/latest/registry-authentication.html#manual-registry-auth-setup +// +// [3] https://account.coreos.com/overview +// +// Note: This field MUST be set manually prior to creating the cluster unless `tectonic_vanilla_k8s` is set to `true`. +tectonic_pull_secret_path = "" + +// (optional) This declares the IP range to assign Kubernetes service cluster IPs in CIDR notation. +// The maximum size of this IP range is /12 +// tectonic_service_cidr = "10.3.0.0/16" + +// Validity period of the self-signed certificates (in hours). +// Default is 3 years. +// This setting is ignored if user provided certificates are used. +tectonic_tls_validity_period = "26280" + +// If set to true, a vanilla Kubernetes cluster will be deployed, omitting any Tectonic assets. +tectonic_vanilla_k8s = false + +// The number of worker nodes to be created. +// This applies only to cloud platforms. +tectonic_worker_count = "3" diff --git a/modules/aws/etcd/govcloud.tf b/modules/aws/etcd/govcloud.tf new file mode 100644 index 0000000000..7792a92a0b --- /dev/null +++ b/modules/aws/etcd/govcloud.tf @@ -0,0 +1,33 @@ +data "aws_region" "current" { + current = true +} + +locals { + govcloud_partition = "${data.aws_region.current.name == "us-gov-west-1" ? 1 : 0}" + aws_partition = "${data.aws_region.current.name == "us-gov-west-1" ? 0 : 1}" + ami_owner = "${data.aws_region.current.name == "us-gov-west-1" ? "190570271432" : "595879546273"}" + arn = "${data.aws_region.current.name == "us-gov-west-1" ? "aws-us-gov" : "aws"}" + + s3_object_keys = "${data.aws_region.current.name == "us-gov-west-1" ? + join(" ", aws_s3_bucket_object.ignition_etcd_govcloud.*.key) : + join(" ", aws_s3_bucket_object.ignition_etcd.*.key)}" + + // Ignition does not support s3 protocol for GovCloud https://github.com/coreos/ignition/pull/477 + s3_prefix = "${data.aws_region.current.name == "us-gov-west-1" ? "https://s3-us-gov-west-1.amazonaws.com" : "s3:/" }" + s3_endpoints = "${formatlist("${local.s3_prefix}/%s/%s", var.s3_bucket, split(" ", local.s3_object_keys))}" +} + +// Ignition config +resource "aws_s3_bucket_object" "ignition_etcd_govcloud" { + count = "${local.govcloud_partition == 1 ? (length(var.external_endpoints) == 0 ? var.instance_count : 0) : 0}" + + bucket = "${var.s3_bucket}" + key = "ignition_etcd_${count.index}.json" + content = "${data.ignition_config.etcd.*.rendered[count.index]}" + acl = "public-read" + + server_side_encryption = "AES256" + + // Terraform does not support tags here for GovCloud + // https://github.com/terraform-providers/terraform-provider-aws/pull/2665 +} diff --git a/modules/aws/etcd/ignition.tf b/modules/aws/etcd/ignition.tf index bb29af7749..22c5d75d8e 100644 --- a/modules/aws/etcd/ignition.tf +++ b/modules/aws/etcd/ignition.tf @@ -8,6 +8,7 @@ data "ignition_config" "etcd" { files = [ "${var.ign_etcd_crt_id_list}", + "${var.dns_server_ip != "" ? join("", data.ignition_file.node_resolv.*.id) : ""}", ] } @@ -32,3 +33,20 @@ EOF }, ] } + +// DNS Server resolution +data "template_file" "node_resolv" { + count = "${var.dns_server_ip != "" ? 1 : 0}" + template = "search ${data.aws_region.current.name}.compute.internal\nnameserver ${var.dns_server_ip}" +} + +data "ignition_file" "node_resolv" { + count = "${var.dns_server_ip != "" ? 1 : 0}" + path = "/etc/resolv.conf" + mode = 0644 + filesystem = "root" + + content { + content = "${data.template_file.node_resolv.rendered}" + } +} diff --git a/modules/aws/etcd/ignition_s3.tf b/modules/aws/etcd/ignition_s3.tf index b0b98f1373..7a3abd2330 100644 --- a/modules/aws/etcd/ignition_s3.tf +++ b/modules/aws/etcd/ignition_s3.tf @@ -1,5 +1,5 @@ resource "aws_s3_bucket_object" "ignition_etcd" { - count = "${length(var.external_endpoints) == 0 ? var.instance_count : 0}" + count = "${local.aws_partition == 1 ? (length(var.external_endpoints) == 0 ? var.instance_count : 0) : 0}" bucket = "${var.s3_bucket}" key = "ignition_etcd_${count.index}.json" @@ -19,7 +19,7 @@ data "ignition_config" "s3" { count = "${length(var.external_endpoints) == 0 ? var.instance_count : 0}" replace { - source = "${format("s3://%s/%s", var.s3_bucket, aws_s3_bucket_object.ignition_etcd.*.key[count.index])}" + source = "${local.s3_endpoints[count.index]}" verification = "sha512-${sha512(data.ignition_config.etcd.*.rendered[count.index])}" } } diff --git a/modules/aws/etcd/nodes.tf b/modules/aws/etcd/nodes.tf index b03b4983e4..85c9a22104 100644 --- a/modules/aws/etcd/nodes.tf +++ b/modules/aws/etcd/nodes.tf @@ -16,7 +16,7 @@ data "aws_ami" "coreos_ami" { filter { name = "owner-id" - values = ["595879546273"] + values = ["${local.ami_owner}"] } } @@ -85,7 +85,7 @@ resource "aws_iam_role_policy" "etcd" { "Action" : [ "s3:GetObject" ], - "Resource": "arn:aws:s3:::*", + "Resource": "arn:${local.arn}:s3:::*", "Effect": "Allow" } ] diff --git a/modules/aws/etcd/variables.tf b/modules/aws/etcd/variables.tf index 0facbe7ba8..dc2a506d01 100644 --- a/modules/aws/etcd/variables.tf +++ b/modules/aws/etcd/variables.tf @@ -90,3 +90,8 @@ variable "etcd_iam_role" { default = "" description = "IAM role to use for the instance profiles of etcd nodes." } + +variable "dns_server_ip" { + type = "string" + default = "" +} diff --git a/modules/aws/master-asg/govcloud.tf b/modules/aws/master-asg/govcloud.tf new file mode 100644 index 0000000000..f7bf2b9221 --- /dev/null +++ b/modules/aws/master-asg/govcloud.tf @@ -0,0 +1,30 @@ +data "aws_region" "current" { + current = true +} + +locals { + govcloud_partition = "${data.aws_region.current.name == "us-gov-west-1" ? 1 : 0}" + aws_partition = "${data.aws_region.current.name == "us-gov-west-1" ? 0 : 1}" + ami_owner = "${data.aws_region.current.name == "us-gov-west-1" ? "190570271432" : "595879546273"}" + arn = "${data.aws_region.current.name == "us-gov-west-1" ? "aws-us-gov" : "aws"}" + s3_object_key = "ignition_master.json" + + // Ignition does not support s3 protocol for GovCloud https://github.com/coreos/ignition/pull/477 + s3_endpoint = "${data.aws_region.current.name == "us-gov-west-1" ? + format("https://s3-us-gov-west-1.amazonaws.com/%s/%s", var.s3_bucket, local.s3_object_key) : + format("s3://%s/%s", var.s3_bucket, local.s3_object_key)}" +} + +// Ignition config +resource "aws_s3_bucket_object" "ignition_master_govcloud" { + count = "${local.govcloud_partition}" + bucket = "${var.s3_bucket}" + key = "${local.s3_object_key}" + content = "${data.ignition_config.main.rendered}" + acl = "public-read" + + server_side_encryption = "AES256" + + // Terraform does not support tags here for GovCloud + // https://github.com/terraform-providers/terraform-provider-aws/pull/2665 +} diff --git a/modules/aws/master-asg/ignition.tf b/modules/aws/master-asg/ignition.tf index 632d300ba4..dc12152f13 100644 --- a/modules/aws/master-asg/ignition.tf +++ b/modules/aws/master-asg/ignition.tf @@ -8,6 +8,7 @@ data "ignition_config" "main" { "${var.ign_max_user_watches_id}", "${var.ign_s3_puller_id}", "${var.ign_ca_cert_id_list}", + "${var.dns_server_ip != "" ? join("", data.ignition_file.node_resolv.*.id) : ""}", ] systemd = ["${compact(list( @@ -83,3 +84,20 @@ data "ignition_file" "rm_assets" { content = "${data.template_file.rm_assets.rendered}" } } + +// DNS Server resolution +data "template_file" "node_resolv" { + count = "${var.dns_server_ip != "" ? 1 : 0}" + template = "search ${data.aws_region.current.name}.compute.internal\nnameserver ${var.dns_server_ip}" +} + +data "ignition_file" "node_resolv" { + count = "${var.dns_server_ip != "" ? 1 : 0}" + path = "/etc/resolv.conf" + mode = 0644 + filesystem = "root" + + content { + content = "${data.template_file.node_resolv.rendered}" + } +} diff --git a/modules/aws/master-asg/ignition_s3.tf b/modules/aws/master-asg/ignition_s3.tf index a6e5a6ca00..87f8df1768 100644 --- a/modules/aws/master-asg/ignition_s3.tf +++ b/modules/aws/master-asg/ignition_s3.tf @@ -1,6 +1,7 @@ resource "aws_s3_bucket_object" "ignition_master" { + count = "${local.aws_partition}" bucket = "${var.s3_bucket}" - key = "ignition_master.json" + key = "${local.s3_object_key}" content = "${data.ignition_config.main.rendered}" acl = "private" @@ -15,7 +16,7 @@ resource "aws_s3_bucket_object" "ignition_master" { data "ignition_config" "s3" { replace { - source = "${format("s3://%s/%s", var.s3_bucket, aws_s3_bucket_object.ignition_master.key)}" + source = "${local.s3_endpoint}" verification = "sha512-${sha512(data.ignition_config.main.rendered)}" } } diff --git a/modules/aws/master-asg/master.tf b/modules/aws/master-asg/master.tf index fa3fbab130..764f685336 100644 --- a/modules/aws/master-asg/master.tf +++ b/modules/aws/master-asg/master.tf @@ -16,7 +16,7 @@ data "aws_ami" "coreos_ami" { filter { name = "owner-id" - values = ["595879546273"] + values = ["${local.ami_owner}"] } } @@ -155,7 +155,7 @@ resource "aws_iam_role_policy" "master_policy" { "s3:ListBucket", "s3:PutObject" ], - "Resource": "arn:aws:s3:::*", + "Resource": "arn:${local.arn}:s3:::*", "Effect": "Allow" }, { diff --git a/modules/aws/master-asg/variables.tf b/modules/aws/master-asg/variables.tf index f269fd425b..6e7157b26d 100644 --- a/modules/aws/master-asg/variables.tf +++ b/modules/aws/master-asg/variables.tf @@ -142,3 +142,8 @@ variable "ign_rm_assets_path_unit_id" { variable "s3_bucket" { type = "string" } + +variable "dns_server_ip" { + type = "string" + default = "" +} diff --git a/modules/aws/worker-asg/govcloud.tf b/modules/aws/worker-asg/govcloud.tf new file mode 100644 index 0000000000..fce4660153 --- /dev/null +++ b/modules/aws/worker-asg/govcloud.tf @@ -0,0 +1,30 @@ +data "aws_region" "current" { + current = true +} + +locals { + govcloud_partition = "${data.aws_region.current.name == "us-gov-west-1" ? 1 : 0}" + aws_partition = "${data.aws_region.current.name == "us-gov-west-1" ? 0 : 1}" + ami_owner = "${data.aws_region.current.name == "us-gov-west-1" ? "190570271432" : "595879546273"}" + arn = "${data.aws_region.current.name == "us-gov-west-1" ? "aws-us-gov" : "aws"}" + s3_object_key = "ignition_worker.json" + + // Ignition does not support s3 protocol for GovCloud https://github.com/coreos/ignition/pull/477 + s3_endpoint = "${data.aws_region.current.name == "us-gov-west-1" ? + format("https://s3-us-gov-west-1.amazonaws.com/%s/%s", var.s3_bucket, local.s3_object_key) : + format("s3://%s/%s", var.s3_bucket, local.s3_object_key)}" +} + +// Ignition config +resource "aws_s3_bucket_object" "ignition_worker_govcloud" { + count = "${local.govcloud_partition}" + bucket = "${var.s3_bucket}" + key = "${local.s3_object_key}" + content = "${data.ignition_config.main.rendered}" + acl = "public-read" + + server_side_encryption = "AES256" + + // Terraform does not support tags here for GovCloud + // https://github.com/terraform-providers/terraform-provider-aws/pull/2665 +} diff --git a/modules/aws/worker-asg/ignition.tf b/modules/aws/worker-asg/ignition.tf index 7c0dcca517..98265189d2 100644 --- a/modules/aws/worker-asg/ignition.tf +++ b/modules/aws/worker-asg/ignition.tf @@ -5,6 +5,7 @@ data "ignition_config" "main" { "${var.ign_max_user_watches_id}", "${var.ign_s3_puller_id}", "${var.ign_ca_cert_id_list}", + "${var.dns_server_ip != "" ? join("", data.ignition_file.node_resolv.*.id) : ""}", ] systemd = [ @@ -15,3 +16,20 @@ data "ignition_config" "main" { "${var.ign_update_ca_certificates_dropin_id}", ] } + +// DNS Server resolution +data "template_file" "node_resolv" { + count = "${var.dns_server_ip != "" ? 1 : 0}" + template = "search ${data.aws_region.current.name}.compute.internal\nnameserver ${var.dns_server_ip}" +} + +data "ignition_file" "node_resolv" { + count = "${var.dns_server_ip != "" ? 1 : 0}" + path = "/etc/resolv.conf" + mode = 0644 + filesystem = "root" + + content { + content = "${data.template_file.node_resolv.rendered}" + } +} diff --git a/modules/aws/worker-asg/ignition_s3.tf b/modules/aws/worker-asg/ignition_s3.tf index 9766c6ba5b..b448e95cf8 100644 --- a/modules/aws/worker-asg/ignition_s3.tf +++ b/modules/aws/worker-asg/ignition_s3.tf @@ -1,6 +1,7 @@ resource "aws_s3_bucket_object" "ignition_worker" { + count = "${local.aws_partition}" bucket = "${var.s3_bucket}" - key = "ignition_worker.json" + key = "${local.s3_object_key}" content = "${data.ignition_config.main.rendered}" acl = "private" @@ -15,7 +16,7 @@ resource "aws_s3_bucket_object" "ignition_worker" { data "ignition_config" "s3" { replace { - source = "${format("s3://%s/%s", var.s3_bucket, aws_s3_bucket_object.ignition_worker.key)}" + source = "${local.s3_endpoint}" verification = "sha512-${sha512(data.ignition_config.main.rendered)}" } } diff --git a/modules/aws/worker-asg/variables.tf b/modules/aws/worker-asg/variables.tf index bdbddfc0ff..df89b45391 100644 --- a/modules/aws/worker-asg/variables.tf +++ b/modules/aws/worker-asg/variables.tf @@ -86,3 +86,8 @@ variable "ign_s3_puller_id" { variable "s3_bucket" { type = "string" } + +variable "dns_server_ip" { + type = "string" + default = "" +} diff --git a/modules/aws/worker-asg/worker.tf b/modules/aws/worker-asg/worker.tf index 72e46cd3c7..7d8c8cce62 100644 --- a/modules/aws/worker-asg/worker.tf +++ b/modules/aws/worker-asg/worker.tf @@ -16,7 +16,7 @@ data "aws_ami" "coreos_ami" { filter { name = "owner-id" - values = ["595879546273"] + values = ["${local.ami_owner}"] } } @@ -166,7 +166,7 @@ resource "aws_iam_role_policy" "worker_policy" { "Action" : [ "s3:GetObject" ], - "Resource": "arn:aws:s3:::*", + "Resource": "arn:${local.arn}:s3:::*", "Effect": "Allow" }, { diff --git a/modules/dns/powerdns/outputs.tf b/modules/dns/powerdns/outputs.tf new file mode 100644 index 0000000000..380e379c13 --- /dev/null +++ b/modules/dns/powerdns/outputs.tf @@ -0,0 +1,26 @@ +output "etcd_a_nodes" { + value = "${powerdns_record.etcd.*.name}" +} + +# We have to do this join() & split() 'trick' because the ternary operator can't output lists. +# https://github.com/hashicorp/terraform/issues/11566#issuecomment-289417805 +output "etcd_endpoints" { + value = ["${split(",", length(var.external_endpoints) == 0 ? join(",", powerdns_record.etcd.*.name) : join(",", var.external_endpoints))}"] +} + +output "ingress_external_fqdn" { + # Remove "." at the end of each name + value = "${replace(replace(join(";", powerdns_record.ingress_public.*.name), "/[.];/", " "), "/[.]$/", "")}" +} + +output "ingress_internal_fqdn" { + value = "${replace(replace(join(";", powerdns_record.ingress_private.*.name), "/[.];/", " "), "/[.]$/", "")}" +} + +output "api_external_fqdn" { + value = "${replace(replace(join(";", powerdns_record.api_external.*.name), "/[.];/", " "), "/[.]$/", "")}" +} + +output "api_internal_fqdn" { + value = "${replace(replace(join(";", powerdns_record.api_internal.*.name), "/[.];/", " "), "/[.]$/", "")}" +} diff --git a/modules/dns/powerdns/records.tf b/modules/dns/powerdns/records.tf new file mode 100644 index 0000000000..b52f8b3b33 --- /dev/null +++ b/modules/dns/powerdns/records.tf @@ -0,0 +1,54 @@ +provider "powerdns" { + version = "0.1.0" + api_key = "tectonicgov" + server_url = "${var.nameserver_url}" +} + +resource "powerdns_record" "api_internal" { + count = "${var.tectonic_private_endpoints}" + zone = "${var.base_domain}" + name = "${var.custom_dns_name == "" ? var.cluster_name : var.custom_dns_name}-api.${var.base_domain}." + ttl = 60 + type = "ALIAS" + + records = ["${var.api_internal_elb_dns_name}."] +} + +resource "powerdns_record" "api_external" { + count = "${var.tectonic_public_endpoints}" + zone = "${var.base_domain}" + name = "${var.custom_dns_name == "" ? var.cluster_name : var.custom_dns_name}-api.${var.base_domain}." + type = "ALIAS" + ttl = 60 + records = ["${var.api_external_elb_dns_name}."] +} + +resource "powerdns_record" "ingress_public" { + count = "${var.tectonic_public_endpoints}" + zone = "${var.base_domain}" + name = "${var.custom_dns_name == "" ? var.cluster_name : var.custom_dns_name}.${var.base_domain}." + type = "ALIAS" + ttl = 60 + records = ["${var.console_elb_dns_name}."] +} + +resource "powerdns_record" "ingress_private" { + count = "${var.tectonic_private_endpoints}" + zone = "${var.base_domain}" + name = "${var.custom_dns_name == "" ? var.cluster_name : var.custom_dns_name}.${var.base_domain}." + type = "ALIAS" + ttl = 60 + + records = [ + "${var.console_elb_dns_name}.", + ] +} + +resource "powerdns_record" "etcd" { + count = "${var.etcd_count}" + zone = "${var.base_domain}" + name = "${var.cluster_name}-etcd-${count.index}.${var.base_domain}." + type = "A" + ttl = 60 + records = ["${var.etcd_ip_addresses[count.index]}"] +} diff --git a/modules/dns/powerdns/variables.tf b/modules/dns/powerdns/variables.tf new file mode 100644 index 0000000000..d472ea2632 --- /dev/null +++ b/modules/dns/powerdns/variables.tf @@ -0,0 +1,182 @@ +variable "etcd_tls_enabled" { + description = "Indicates whether TLS is used for etcd" + type = "string" + default = "1" +} + +variable "cluster_name" { + description = "The name of the cluster" + type = "string" +} + +variable "base_domain" { + description = "The base domain used in records" + type = "string" +} + +variable "master_count" { + description = "The number of masters" + type = "string" +} + +variable "worker_count" { + description = "The number of workers" + type = "string" + default = "0" +} + +variable "etcd_count" { + description = "The number of etcd nodes" + type = "string" +} + +variable "etcd_ip_addresses" { + description = "List of string IPs for etcd nodes" + type = "list" +} + +variable "master_ip_addresses" { + description = "List of string IPs for masters" + type = "list" + default = [] +} + +variable "worker_ip_addresses" { + description = "List of string IPs for workers" + type = "list" + default = [] +} + +variable "cluster_id" { + description = "Cluster ID" + type = "string" +} + +variable "external_endpoints" { + type = "list" + + description = < 0 ? + join("|", keys(var.tectonic_govcloud_master_custom_subnets)) : + join("|", data.aws_availability_zones.azs.names) + )}" + worker_azs = "${ split("|", "${length(keys(var.tectonic_govcloud_worker_custom_subnets))}" > 0 ? + join("|", keys(var.tectonic_govcloud_worker_custom_subnets)) : + join("|", data.aws_availability_zones.azs.names) + )}" +} + +module "etcd" { + source = "../../modules/aws/etcd" + + base_domain = "${var.tectonic_base_domain}" + cluster_id = "${module.tectonic.cluster_id}" + cluster_name = "${var.tectonic_cluster_name}" + container_image = "${var.tectonic_container_images["etcd"]}" + container_linux_channel = "${var.tectonic_container_linux_channel}" + container_linux_version = "${module.container_linux.version}" + ec2_type = "${var.tectonic_govcloud_etcd_ec2_type}" + external_endpoints = "${compact(var.tectonic_etcd_servers)}" + extra_tags = "${var.tectonic_govcloud_extra_tags}" + ign_etcd_crt_id_list = "${module.ignition_masters.etcd_crt_id_list}" + ign_etcd_dropin_id_list = "${module.ignition_masters.etcd_dropin_id_list}" + instance_count = "${length(data.template_file.etcd_hostname_list.*.id)}" + root_volume_iops = "${var.tectonic_govcloud_etcd_root_volume_iops}" + root_volume_size = "${var.tectonic_govcloud_etcd_root_volume_size}" + root_volume_type = "${var.tectonic_govcloud_etcd_root_volume_type}" + s3_bucket = "${aws_s3_bucket.tectonic.bucket}" + sg_ids = "${concat(var.tectonic_govcloud_etcd_extra_sg_ids, list(module.vpc.etcd_sg_id))}" + ssh_key = "${var.tectonic_govcloud_ssh_key}" + subnets = "${module.vpc.worker_subnet_ids}" + tls_enabled = "${var.tectonic_etcd_tls_enabled}" + dns_server_ip = "${var.tectonic_govcloud_dns_server_ip}" +} + +module "ignition_masters" { + source = "../../modules/ignition" + + assets_location = "${aws_s3_bucket_object.tectonic_assets.bucket}/${aws_s3_bucket_object.tectonic_assets.key}" + base_domain = "${var.tectonic_base_domain}" + bootstrap_upgrade_cl = "${var.tectonic_bootstrap_upgrade_cl}" + cloud_provider = "aws" + cluster_name = "${var.tectonic_cluster_name}" + container_images = "${var.tectonic_container_images}" + custom_ca_cert_pem_list = "${var.tectonic_custom_ca_pem_list}" + etcd_advertise_name_list = "${data.template_file.etcd_hostname_list.*.rendered}" + etcd_ca_cert_pem = "${module.etcd_certs.etcd_ca_crt_pem}" + etcd_client_crt_pem = "${module.etcd_certs.etcd_client_crt_pem}" + etcd_client_key_pem = "${module.etcd_certs.etcd_client_key_pem}" + etcd_count = "${length(data.template_file.etcd_hostname_list.*.id)}" + etcd_initial_cluster_list = "${data.template_file.etcd_hostname_list.*.rendered}" + etcd_peer_crt_pem = "${module.etcd_certs.etcd_peer_crt_pem}" + etcd_peer_key_pem = "${module.etcd_certs.etcd_peer_key_pem}" + etcd_server_crt_pem = "${module.etcd_certs.etcd_server_crt_pem}" + etcd_server_key_pem = "${module.etcd_certs.etcd_server_key_pem}" + etcd_tls_enabled = "${var.tectonic_etcd_tls_enabled}" + image_re = "${var.tectonic_image_re}" + ingress_ca_cert_pem = "${module.ingress_certs.ca_cert_pem}" + kube_ca_cert_pem = "${module.kube_certs.ca_cert_pem}" + kube_dns_service_ip = "${module.bootkube.kube_dns_service_ip}" + kubeconfig_fetch_cmd = "/opt/s3-puller.sh ${aws_s3_bucket_object.kubeconfig.bucket}/${aws_s3_bucket_object.kubeconfig.key} /etc/kubernetes/kubeconfig" + kubelet_cni_bin_dir = "${var.tectonic_networking == "calico" || var.tectonic_networking == "canal" ? "/var/lib/cni/bin" : "" }" + kubelet_debug_config = "${var.tectonic_kubelet_debug_config}" + kubelet_node_label = "node-role.kubernetes.io/master" + kubelet_node_taints = "node-role.kubernetes.io/master=:NoSchedule" + tectonic_vanilla_k8s = "${var.tectonic_vanilla_k8s}" +} + +module "masters" { + source = "../../modules/aws/master-asg" + + assets_s3_location = "${aws_s3_bucket_object.tectonic_assets.bucket}/${aws_s3_bucket_object.tectonic_assets.key}" + autoscaling_group_extra_tags = "${var.tectonic_autoscaling_group_extra_tags}" + aws_lbs = "${module.vpc.aws_lbs}" + base_domain = "${var.tectonic_base_domain}" + cluster_id = "${module.tectonic.cluster_id}" + cluster_name = "${var.tectonic_cluster_name}" + container_images = "${var.tectonic_container_images}" + container_linux_channel = "${var.tectonic_container_linux_channel}" + container_linux_version = "${module.container_linux.version}" + ec2_type = "${var.tectonic_govcloud_master_ec2_type}" + extra_tags = "${var.tectonic_govcloud_extra_tags}" + ign_bootkube_path_unit_id = "${module.bootkube.systemd_path_unit_id}" + ign_bootkube_service_id = "${module.bootkube.systemd_service_id}" + ign_ca_cert_id_list = "${module.ignition_masters.ca_cert_id_list}" + ign_docker_dropin_id = "${module.ignition_masters.docker_dropin_id}" + ign_init_assets_service_id = "${module.ignition_masters.init_assets_service_id}" + ign_installer_kubelet_env_id = "${module.ignition_masters.installer_kubelet_env_id}" + ign_installer_runtime_mappings_id = "${module.ignition_masters.installer_runtime_mappings_id}" + ign_k8s_node_bootstrap_service_id = "${module.ignition_masters.k8s_node_bootstrap_service_id}" + ign_kubelet_service_id = "${module.ignition_masters.kubelet_service_id}" + ign_locksmithd_service_id = "${module.ignition_masters.locksmithd_service_id}" + ign_max_user_watches_id = "${module.ignition_masters.max_user_watches_id}" + ign_rm_assets_path_unit_id = "${module.ignition_masters.rm_assets_path_unit_id}" + ign_rm_assets_service_id = "${module.ignition_masters.rm_assets_service_id}" + ign_s3_puller_id = "${module.ignition_masters.s3_puller_id}" + ign_tectonic_path_unit_id = "${var.tectonic_vanilla_k8s ? "" : module.tectonic.systemd_path_unit_id}" + ign_tectonic_service_id = "${module.tectonic.systemd_service_id}" + ign_update_ca_certificates_dropin_id = "${module.ignition_masters.update_ca_certificates_dropin_id}" + image_re = "${var.tectonic_image_re}" + instance_count = "${var.tectonic_master_count}" + master_iam_role = "${var.tectonic_govcloud_master_iam_role_name}" + master_sg_ids = "${concat(var.tectonic_govcloud_master_extra_sg_ids, list(module.vpc.master_sg_id))}" + private_endpoints = "${var.tectonic_govcloud_private_endpoints}" + public_endpoints = "${var.tectonic_govcloud_public_endpoints}" + root_volume_iops = "${var.tectonic_govcloud_master_root_volume_iops}" + root_volume_size = "${var.tectonic_govcloud_master_root_volume_size}" + root_volume_type = "${var.tectonic_govcloud_master_root_volume_type}" + s3_bucket = "${aws_s3_bucket.tectonic.bucket}" + ssh_key = "${var.tectonic_govcloud_ssh_key}" + subnet_ids = "${module.vpc.master_subnet_ids}" + dns_server_ip = "${var.tectonic_govcloud_dns_server_ip}" +} + +module "ignition_workers" { + source = "../../modules/ignition" + + bootstrap_upgrade_cl = "${var.tectonic_bootstrap_upgrade_cl}" + cloud_provider = "aws" + container_images = "${var.tectonic_container_images}" + custom_ca_cert_pem_list = "${var.tectonic_custom_ca_pem_list}" + etcd_ca_cert_pem = "${module.etcd_certs.etcd_ca_crt_pem}" + image_re = "${var.tectonic_image_re}" + ingress_ca_cert_pem = "${module.ingress_certs.ca_cert_pem}" + kube_ca_cert_pem = "${module.kube_certs.ca_cert_pem}" + kube_dns_service_ip = "${module.bootkube.kube_dns_service_ip}" + kubeconfig_fetch_cmd = "/opt/s3-puller.sh ${aws_s3_bucket_object.kubeconfig.bucket}/${aws_s3_bucket_object.kubeconfig.key} /etc/kubernetes/kubeconfig" + kubelet_cni_bin_dir = "${var.tectonic_networking == "calico" || var.tectonic_networking == "canal" ? "/var/lib/cni/bin" : "" }" + kubelet_debug_config = "${var.tectonic_kubelet_debug_config}" + kubelet_node_label = "node-role.kubernetes.io/node" + kubelet_node_taints = "" + tectonic_vanilla_k8s = "${var.tectonic_vanilla_k8s}" +} + +module "workers" { + source = "../../modules/aws/worker-asg" + + autoscaling_group_extra_tags = "${var.tectonic_autoscaling_group_extra_tags}" + cluster_id = "${module.tectonic.cluster_id}" + cluster_name = "${var.tectonic_cluster_name}" + container_linux_channel = "${var.tectonic_container_linux_channel}" + container_linux_version = "${module.container_linux.version}" + ec2_type = "${var.tectonic_govcloud_worker_ec2_type}" + extra_tags = "${var.tectonic_govcloud_extra_tags}" + ign_ca_cert_id_list = "${module.ignition_masters.ca_cert_id_list}" + ign_docker_dropin_id = "${module.ignition_workers.docker_dropin_id}" + ign_installer_kubelet_env_id = "${module.ignition_workers.installer_kubelet_env_id}" + ign_installer_runtime_mappings_id = "${module.ignition_workers.installer_runtime_mappings_id}" + ign_k8s_node_bootstrap_service_id = "${module.ignition_workers.k8s_node_bootstrap_service_id}" + ign_kubelet_service_id = "${module.ignition_workers.kubelet_service_id}" + ign_locksmithd_service_id = "${module.ignition_workers.locksmithd_service_id}" + ign_max_user_watches_id = "${module.ignition_workers.max_user_watches_id}" + ign_s3_puller_id = "${module.ignition_workers.s3_puller_id}" + ign_update_ca_certificates_dropin_id = "${module.ignition_workers.update_ca_certificates_dropin_id}" + instance_count = "${var.tectonic_worker_count}" + load_balancers = "${var.tectonic_govcloud_worker_load_balancers}" + root_volume_iops = "${var.tectonic_govcloud_worker_root_volume_iops}" + root_volume_size = "${var.tectonic_govcloud_worker_root_volume_size}" + root_volume_type = "${var.tectonic_govcloud_worker_root_volume_type}" + s3_bucket = "${aws_s3_bucket.tectonic.bucket}" + sg_ids = "${concat(var.tectonic_govcloud_worker_extra_sg_ids, list(module.vpc.worker_sg_id))}" + ssh_key = "${var.tectonic_govcloud_ssh_key}" + subnet_ids = "${module.vpc.worker_subnet_ids}" + vpc_id = "${module.vpc.vpc_id}" + worker_iam_role = "${var.tectonic_govcloud_worker_iam_role_name}" + dns_server_ip = "${var.tectonic_govcloud_dns_server_ip}" +} + +module "dns" { + source = "../../modules/dns/powerdns" + nameserver_url = "http://${var.tectonic_govcloud_dns_server_ip}:8081" + api_external_elb_dns_name = "${module.vpc.aws_api_external_dns_name}" + api_external_elb_zone_id = "${module.vpc.aws_elb_api_external_zone_id}" + api_internal_elb_dns_name = "${module.vpc.aws_api_internal_dns_name}" + api_internal_elb_zone_id = "${module.vpc.aws_elb_api_internal_zone_id}" + api_ip_addresses = "${module.vpc.aws_lbs}" + base_domain = "${var.tectonic_base_domain}" + cluster_id = "${module.tectonic.cluster_id}" + cluster_name = "${var.tectonic_cluster_name}" + console_elb_dns_name = "${module.vpc.aws_console_dns_name}" + console_elb_zone_id = "${module.vpc.aws_elb_onsole_zone_id}" + custom_dns_name = "${var.tectonic_dns_name}" + elb_alias_enabled = true + etcd_count = "${length(data.template_file.etcd_hostname_list.*.id)}" + etcd_ip_addresses = "${module.etcd.ip_addresses}" + external_endpoints = ["${compact(var.tectonic_etcd_servers)}"] + master_count = "${var.tectonic_master_count}" + tectonic_external_private_zone = "${var.tectonic_govcloud_external_private_zone}" + tectonic_external_vpc_id = "${module.vpc.vpc_id}" + tectonic_extra_tags = "${var.tectonic_govcloud_extra_tags}" + tectonic_private_endpoints = "${var.tectonic_govcloud_private_endpoints}" + tectonic_public_endpoints = "${var.tectonic_govcloud_public_endpoints}" + tectonic_vanilla_k8s = "${var.tectonic_vanilla_k8s}" +} diff --git a/platforms/govcloud/s3.tf b/platforms/govcloud/s3.tf new file mode 100644 index 0000000000..014e01aeb9 --- /dev/null +++ b/platforms/govcloud/s3.tf @@ -0,0 +1,53 @@ +data "aws_region" "current" { + current = true +} + +resource "aws_s3_bucket" "tectonic" { + # Buckets must start with a lower case name and are limited to 63 characters, + # so we prepend the letter 'a' and use the md5 hex digest for the case of a long domain + # leaving 29 chars for the cluster name. + bucket = "${var.tectonic_govcloud_assets_s3_bucket_name == "" ? format("%s%s-%s", "a", var.tectonic_cluster_name, md5(format("%s-%s", data.aws_region.current.name , var.tectonic_base_domain))) : var.tectonic_govcloud_assets_s3_bucket_name }" + + acl = "private" + + tags = "${merge(map( + "Name", "${var.tectonic_cluster_name}-tectonic", + "KubernetesCluster", "${var.tectonic_cluster_name}", + "tectonicClusterID", "${module.tectonic.cluster_id}" + ), var.tectonic_govcloud_extra_tags)}" + + lifecycle { + ignore_changes = ["*"] + } +} + +# Bootkube / Tectonic assets +resource "aws_s3_bucket_object" "tectonic_assets" { + bucket = "${aws_s3_bucket.tectonic.bucket}" + key = "assets.zip" + source = "${data.archive_file.assets.output_path}" + acl = "private" + + # To be on par with the current Tectonic installer, we only do server-side + # encryption, using AES256. Eventually, we should start using KMS-based + # client-side encryption. + server_side_encryption = "AES256" + + lifecycle { + ignore_changes = ["*"] + } +} + +# kubeconfig +resource "aws_s3_bucket_object" "kubeconfig" { + bucket = "${aws_s3_bucket.tectonic.bucket}" + key = "kubeconfig" + content = "${module.bootkube.kubeconfig}" + acl = "private" + + # The current Tectonic installer stores bits of the kubeconfig in KMS. As we + # do not support KMS yet, we at least offload it to S3 for now. Eventually, + # we should consider using KMS-based client-side encryption, or uploading it + # to KMS. + server_side_encryption = "AES256" +} diff --git a/platforms/govcloud/tectonic.tf b/platforms/govcloud/tectonic.tf new file mode 100644 index 0000000000..b1dc5a97fc --- /dev/null +++ b/platforms/govcloud/tectonic.tf @@ -0,0 +1,164 @@ +data "template_file" "etcd_hostname_list" { + count = "${var.tectonic_etcd_count > 0 ? var.tectonic_etcd_count : length(data.aws_availability_zones.azs.names) == 5 ? 5 : 3}" + template = "${var.tectonic_cluster_name}-etcd-${count.index}.${var.tectonic_base_domain}" +} + +module "bootkube" { + source = "../../modules/bootkube" + cloud_provider = "aws" + + cluster_name = "${var.tectonic_cluster_name}" + + kube_apiserver_url = "https://${var.tectonic_govcloud_private_endpoints ? module.dns.api_internal_fqdn : module.dns.api_external_fqdn}:443" + oidc_issuer_url = "https://${var.tectonic_govcloud_private_endpoints ? module.dns.ingress_internal_fqdn : module.dns.ingress_external_fqdn}/identity" + + # Platform-independent variables wiring, do not modify. + container_images = "${var.tectonic_container_images}" + versions = "${var.tectonic_versions}" + self_hosted_etcd = "" + + service_cidr = "${var.tectonic_service_cidr}" + cluster_cidr = "${var.tectonic_cluster_cidr}" + + advertise_address = "0.0.0.0" + anonymous_auth = "false" + + oidc_username_claim = "email" + oidc_groups_claim = "groups" + oidc_client_id = "tectonic-kubectl" + oidc_ca_cert = "${module.ingress_certs.ca_cert_pem}" + + apiserver_cert_pem = "${module.kube_certs.apiserver_cert_pem}" + apiserver_key_pem = "${module.kube_certs.apiserver_key_pem}" + etcd_ca_cert_pem = "${module.etcd_certs.etcd_ca_crt_pem}" + etcd_client_cert_pem = "${module.etcd_certs.etcd_client_crt_pem}" + etcd_client_key_pem = "${module.etcd_certs.etcd_client_key_pem}" + etcd_peer_cert_pem = "${module.etcd_certs.etcd_peer_crt_pem}" + etcd_peer_key_pem = "${module.etcd_certs.etcd_peer_key_pem}" + etcd_server_cert_pem = "${module.etcd_certs.etcd_server_crt_pem}" + etcd_server_key_pem = "${module.etcd_certs.etcd_server_key_pem}" + kube_ca_cert_pem = "${module.kube_certs.ca_cert_pem}" + kubelet_cert_pem = "${module.kube_certs.kubelet_cert_pem}" + kubelet_key_pem = "${module.kube_certs.kubelet_key_pem}" + + etcd_backup_size = "${var.tectonic_etcd_backup_size}" + etcd_backup_storage_class = "${var.tectonic_etcd_backup_storage_class}" + etcd_endpoints = "${module.dns.etcd_endpoints}" + master_count = "${var.tectonic_master_count}" + + # The default behavior of Kubernetes's controller manager is to mark a node + # as Unhealthy after 40s without an update from the node's kubelet. However, + # AWS ELB's Route53 records have a fixed TTL of 60s. Therefore, when an ELB's + # node disappears (e.g. scaled down or crashed), kubelet might fail to report + # for a period of time that exceed the default grace period of 40s and the + # node might become Unhealthy. While the eviction process won't start until + # the pod_eviction_timeout is reached, 5min by default, certain operators + # might already have taken action. This is the case for the etcd operator as + # of v0.3.3, which removes the likely-healthy etcd pods from the the + # cluster, potentially leading to a loss-of-quorum as generally all kubelets + # are affected simultaneously. + # + # To cope with this issue, we increase the grace period, and reduce the + # pod eviction time-out accordingly so pods still get evicted after an total + # time of 340s after the first post-status failure. + # + # Ref: https://github.com/kubernetes/kubernetes/issues/41916 + # Ref: https://github.com/kubernetes-incubator/kube-aws/issues/598 + node_monitor_grace_period = "2m" + + pod_eviction_timeout = "220s" + + cloud_config_path = "" +} + +module "tectonic" { + source = "../../modules/tectonic" + platform = "aws" + + cluster_name = "${var.tectonic_cluster_name}" + + base_address = "${var.tectonic_govcloud_private_endpoints ? module.dns.ingress_internal_fqdn : module.dns.ingress_external_fqdn}" + kube_apiserver_url = "https://${var.tectonic_govcloud_private_endpoints ? module.dns.api_internal_fqdn : module.dns.api_external_fqdn}:443" + service_cidr = "${var.tectonic_service_cidr}" + + # Platform-independent variables wiring, do not modify. + container_images = "${var.tectonic_container_images}" + container_base_images = "${var.tectonic_container_base_images}" + versions = "${var.tectonic_versions}" + + license_path = "${var.tectonic_vanilla_k8s ? "/dev/null" : pathexpand(var.tectonic_license_path)}" + pull_secret_path = "${var.tectonic_vanilla_k8s ? "/dev/null" : pathexpand(var.tectonic_pull_secret_path)}" + + admin_email = "${var.tectonic_admin_email}" + admin_password = "${var.tectonic_admin_password}" + + update_channel = "${var.tectonic_update_channel}" + update_app_id = "${var.tectonic_update_app_id}" + update_server = "${var.tectonic_update_server}" + + ca_generated = "${var.tectonic_ca_cert == "" ? false : true}" + ca_cert = "${module.kube_certs.ca_cert_pem}" + + ingress_ca_cert_pem = "${module.ingress_certs.ca_cert_pem}" + ingress_cert_pem = "${module.ingress_certs.cert_pem}" + ingress_key_pem = "${module.ingress_certs.key_pem}" + + identity_client_cert_pem = "${module.identity_certs.client_cert_pem}" + identity_client_key_pem = "${module.identity_certs.client_key_pem}" + identity_server_cert_pem = "${module.identity_certs.server_cert_pem}" + identity_server_key_pem = "${module.identity_certs.server_key_pem}" + + console_client_id = "tectonic-console" + kubectl_client_id = "tectonic-kubectl" + ingress_kind = "NodePort" + self_hosted_etcd = "" + master_count = "${var.tectonic_master_count}" + stats_url = "${var.tectonic_stats_url}" + + image_re = "${var.tectonic_image_re}" + + tectonic_networking = "${var.tectonic_networking}" + calico_mtu = "1480" + cluster_cidr = "${var.tectonic_cluster_cidr}" +} + +module "flannel_vxlan" { + source = "../../modules/net/flannel_vxlan" + + cluster_cidr = "${var.tectonic_cluster_cidr}" + enabled = "${var.tectonic_networking == "flannel"}" + container_images = "${var.tectonic_container_images}" +} + +module "calico" { + source = "../../modules/net/calico" + + container_images = "${var.tectonic_container_images}" + cluster_cidr = "${var.tectonic_cluster_cidr}" + enabled = "${var.tectonic_networking == "calico"}" +} + +module "canal" { + source = "../../modules/net/canal" + + container_images = "${var.tectonic_container_images}" + cluster_cidr = "${var.tectonic_cluster_cidr}" + enabled = "${var.tectonic_networking == "canal"}" +} + +data "archive_file" "assets" { + type = "zip" + source_dir = "./generated/" + + # Because the archive_file provider is a data source, depends_on can't be + # used to guarantee that the tectonic/bootkube modules have generated + # all the assets on disk before trying to archive them. Instead, we use their + # ID outputs, that are only computed once the assets have actually been + # written to disk. We re-hash the IDs (or dedicated module outputs, like module.bootkube.content_hash) + # to make the filename shorter, since there is no security nor collision risk anyways. + # + # Additionally, data sources do not support managing any lifecycle whatsoever, + # and therefore, the archive is never deleted. To avoid cluttering the module + # folder, we write it in the Terraform managed hidden folder `.terraform`. + output_path = "./.terraform/generated_${sha1("${module.etcd_certs.id} ${module.tectonic.id} ${module.bootkube.id} ${module.flannel_vxlan.id} ${module.calico.id} ${module.canal.id}")}.zip" +} diff --git a/platforms/govcloud/tls.tf b/platforms/govcloud/tls.tf new file mode 100644 index 0000000000..6dbab31b1c --- /dev/null +++ b/platforms/govcloud/tls.tf @@ -0,0 +1,40 @@ +module "kube_certs" { + source = "../../modules/tls/kube/self-signed" + + ca_cert_pem = "${var.tectonic_ca_cert}" + ca_key_alg = "${var.tectonic_ca_key_alg}" + ca_key_pem = "${var.tectonic_ca_key}" + kube_apiserver_url = "https://${var.tectonic_govcloud_private_endpoints ? module.dns.api_internal_fqdn : module.dns.api_external_fqdn}:443" + service_cidr = "${var.tectonic_service_cidr}" + validity_period = "${var.tectonic_tls_validity_period}" +} + +module "etcd_certs" { + source = "../../modules/tls/etcd/signed" + + etcd_ca_cert_path = "${var.tectonic_etcd_ca_cert_path}" + etcd_cert_dns_names = "${data.template_file.etcd_hostname_list.*.rendered}" + etcd_client_cert_path = "${var.tectonic_etcd_client_cert_path}" + etcd_client_key_path = "${var.tectonic_etcd_client_key_path}" + self_signed = "${length(compact(var.tectonic_etcd_servers)) == 0 ? "true" : "false"}" + service_cidr = "${var.tectonic_service_cidr}" +} + +module "ingress_certs" { + source = "../../modules/tls/ingress/self-signed" + + base_address = "${var.tectonic_govcloud_private_endpoints ? module.dns.ingress_internal_fqdn : module.dns.ingress_external_fqdn}" + ca_cert_pem = "${module.kube_certs.ca_cert_pem}" + ca_key_alg = "${module.kube_certs.ca_key_alg}" + ca_key_pem = "${module.kube_certs.ca_key_pem}" + validity_period = "${var.tectonic_tls_validity_period}" +} + +module "identity_certs" { + source = "../../modules/tls/identity/self-signed" + + ca_cert_pem = "${module.kube_certs.ca_cert_pem}" + ca_key_alg = "${module.kube_certs.ca_key_alg}" + ca_key_pem = "${module.kube_certs.ca_key_pem}" + validity_period = "${var.tectonic_tls_validity_period}" +} diff --git a/platforms/govcloud/variables.tf b/platforms/govcloud/variables.tf new file mode 100644 index 0000000000..788fb9683e --- /dev/null +++ b/platforms/govcloud/variables.tf @@ -0,0 +1,325 @@ +variable "tectonic_govcloud_config_version" { + description = < e + puts e + end +end diff --git a/tests/rspec/lib/govcloud_vpc.rb b/tests/rspec/lib/govcloud_vpc.rb new file mode 100644 index 0000000000..54526f6a1b --- /dev/null +++ b/tests/rspec/lib/govcloud_vpc.rb @@ -0,0 +1,174 @@ +# frozen_string_literal: true + +require 'json' + +# GovcloudVPC represents an AWS virtual private cloud +class GovcloudVPC + attr_reader :vpn_url + attr_reader :ovpn_password + attr_reader :name + attr_reader :vpc_dns + attr_reader :vpc_id + attr_reader :private_zone_id + attr_reader :master_subnet_ids + attr_reader :worker_subnet_ids + attr_reader :vpn_connection + + def initialize(name) + @name = name + @ovpn_password = + `tr -cd '[:alnum:]' < /dev/urandom | head -c 32 ; echo`.chomp + end + + def env_variables + { + 'TF_VAR_vpc_aws_region' => 'us-gov-west-1', + 'TF_VAR_vpc_name' => @name, + 'TF_VAR_base_domain' => 'tectonic-ci.de', + 'TF_VAR_nginx_username' => 'openvpn', + 'TF_VAR_nginx_password' => @ovpn_password + } + end + + def export_tfvars + vars = { + 'TF_VAR_tectonic_govcloud_external_vpc_id' => @vpc_id, + 'TF_VAR_tectonic_govcloud_external_master_subnet_ids' => @master_subnet_ids, + 'TF_VAR_tectonic_govcloud_external_worker_subnet_ids' => @worker_subnet_ids, + 'TF_VAR_tectonic_govcloud_dns_server_ip' => @vpc_dns + } + vars.each do |key, value| + ENV[key] = value + end + end + + def create + Dir.chdir('../../contrib/govcloud') do + succeeded = system(env_variables, 'terraform init') + raise 'could not init Terraform to create VPC' unless succeeded + succeeded = system(env_variables, 'terraform apply') + raise 'could not create vpc with Terraform' unless succeeded + + parse_terraform_output + wait_for_vpn_access_server + wait_for_dns_server + + @vpn_connection = GovcloudVPNConnection.new(@ovpn_password, @vpn_url) + @vpn_connection.start + end + + set_nameserver + export_tfvars + end + + def parse_terraform_output + tf_out = JSON.parse(`terraform output -json`) + @vpn_url = tf_out['ovpn_url']['value'] + @vpc_dns = tf_out['vpc_dns']['value'] + @vpc_id = tf_out['vpc_id']['value'] + parse_subnets(tf_out) + end + + def parse_subnets(tf_out) + subnets = tf_out['subnets']['value'] + @master_subnet_ids = + "[\"#{subnets[0]}\", \"#{subnets[1]}\"]" + @worker_subnet_ids = + "[\"#{subnets[2]}\", \"#{subnets[3]}\"]" + end + + def destroy + @vpn_connection.stop + rescue + raise 'could not disconnect from vpn' + ensure + terraform_destroy + recover_etc_resolv + end + + def terraform_destroy + Dir.chdir('../../contrib/govcloud') do + 3.times do + return if system(env_variables, 'terraform destroy -force') + end + end + + raise 'could not destroy vpc with Terraform' + end + + def wait_for_vpn_access_server + 90.times do + succeeded = system("curl -k -L --silent -u 'openvpn:#{@ovpn_password}' #{@vpn_url} > /dev/null") + return if succeeded + sleep(5) + puts 'waiting for vpn access server' + end + raise 'waiting for vpn access server timed out' + end + + def wait_for_dns_server + 90.times do + succeeded = system("curl -k -L --silent #{@vpn_url}:8081 > /dev/null") + return if succeeded + sleep(5) + puts 'waiting for dns server' + end + raise 'waiting for dns server timed out' + end + + def set_nameserver + # Use AWS VPC DNS rather than host's. + FileUtils.cp '/etc/resolv.conf', '/etc/resolv.conf.bak' + IO.write('/etc/resolv.conf', "search us-gov-west-1.compute.internal\nnameserver #{@vpc_dns}\nnameserver 8.8.8.8\n") + system('cat /etc/resolv.conf') + end + + def recover_etc_resolv + FileUtils.cp '/etc/resolv.conf.bak', '/etc/resolv.conf' + end +end + +# VPNConnection represents a VPN connection via the VPN server in an AWS VPC +class GovcloudVPNConnection + attr_reader :vpn_url + attr_reader :ovpn_password + attr_reader :vpn_conf + + def initialize(ovpn_password, vpn_url) + @ovpn_password = ovpn_password + @vpn_url = vpn_url + end + + def curl_vpn_config + cmd = 'curl -k -L ' \ + "-u 'openvpn:#{@ovpn_password}' " \ + '--silent ' \ + '--fail ' \ + "#{@vpn_url}" + + @vpn_conf = `#{cmd}`.chomp + IO.write('vpn.conf', @vpn_conf) + end + + def start + curl_vpn_config + succeeded = system('openvpn --config vpn.conf --daemon') + raise 'could not start vpn' unless succeeded + + wait_for_network + puts 'Connection established.' + end + + def stop + system('pkill openvpn || true') + wait_for_network + end + + def wait_for_network + 90.times do + succeeded = system('ping -c 1 8.8.8.8 > /dev/null') + return if succeeded + end + raise 'waiting for network timed out' + end +end diff --git a/tests/rspec/lib/tfvars_file.rb b/tests/rspec/lib/tfvars_file.rb index 2efb1ba0d3..97883a4725 100644 --- a/tests/rspec/lib/tfvars_file.rb +++ b/tests/rspec/lib/tfvars_file.rb @@ -2,7 +2,7 @@ require 'json' -PLATFORMS = %w[aws azure metal vmware gcp].freeze +PLATFORMS = %w[govcloud aws azure metal vmware gcp].freeze # TFVarsFile represents a Terraform configuration file describing a Tectonic # cluster configuration diff --git a/tests/rspec/spec/govcloud/vpc_internal_spec.rb b/tests/rspec/spec/govcloud/vpc_internal_spec.rb new file mode 100644 index 0000000000..b2cc4f01f2 --- /dev/null +++ b/tests/rspec/spec/govcloud/vpc_internal_spec.rb @@ -0,0 +1,29 @@ +# frozen_string_literal: true + +require 'shared_examples/k8s' +require 'govcloud_vpc' +require 'aws_region' +require 'jenkins' +require 'aws_iam' + +RSpec.describe 'govcloud-vpc' do + include_examples('withBuildFolderSetup', '../smoke/govcloud/vars/govcloud-vpc-internal.tfvars.json') + + before(:all) do + @ssh_key = ENV['TF_VAR_tectonic_govcloud_ssh_key'] || AwsSupport.create_aws_key_pairs('us-gov-west-1') + ENV['TF_VAR_tectonic_govcloud_ssh_key'] = @ssh_key + ENV['TF_VAR_ssh_key'] = @ssh_key + + # AWSIAM.assume_role if Jenkins.environment? + @vpc = GovcloudVPC.new('test-vpc-govcloud') + @vpc.create + end + + context 'with a cluster' do + include_examples('withRunningClusterExistingBuildFolder') + end + + after(:all) do + @vpc.destroy + end +end diff --git a/tests/smoke/govcloud/vars/govcloud-vpc-internal.tfvars.json b/tests/smoke/govcloud/vars/govcloud-vpc-internal.tfvars.json new file mode 100644 index 0000000000..39372e20dc --- /dev/null +++ b/tests/smoke/govcloud/vars/govcloud-vpc-internal.tfvars.json @@ -0,0 +1,15 @@ +{ + "tectonic_govcloud_az_count": "2", + "tectonic_govcloud_etcd_ec2_type": "m4.large", + "tectonic_govcloud_master_ec2_type": "m4.large", + "tectonic_govcloud_vpc_cidr_block": "10.0.0.0/16", + "tectonic_govcloud_worker_ec2_type": "m4.large", + "tectonic_ca_cert": "", + "tectonic_ca_key": "", + "tectonic_container_linux_channel": "stable", + "tectonic_etcd_count": "1", + "tectonic_etcd_servers": [""], + "tectonic_master_count": "1", + "tectonic_stats_url": "https://stats-collector-staging.tectonic.com", + "tectonic_worker_count": "2" +}