Merge of 4.1 development

.flake8

@@ -3,3 +3,5 @@
 exclude=.tox,inventory
 max_line_length = 120
 ignore = E501,T003
+per-file-ignores =
+    setup.py : T001

.gitignore

@@ -24,3 +24,13 @@ multi_ec2.yaml
 *.egg-info
 .eggs
 cover/
+test/libvirt/install-config.yml
+test/libvirt/.openshift_install_state.json
+test/libvirt/.openshift_install.log
+test/libvirt/*.ign
+test/libvirt/install-config-ansible.yml
+test/libvirt/terraform/terraform.tfvars
+test/libvirt/terraform/.terraform
+*.tfstate
+*.tfstate.backup
+test/libvirt/inventory.txt

.travis.yml

@@ -14,7 +14,7 @@ python:
 
 install:
   - pip install --upgrade pip
-  - pip install tox-travis coveralls
+  - pip install tox-travis
 
 script:
   - tox
@@ -32,6 +32,3 @@ after_failure:
   - pip_debug_log=/home/travis/.cache/pip/log/debug.log
   - echo Outputting pip debug log from $pip_debug_log
   - cat $pip_debug_log
-
-after_success:
-  - coveralls

README.md

@@ -9,20 +9,18 @@ Master branch is closed! A major refactor is ongoing in devel-40.
 Changes for 3.x should be made directly to the latest release branch they're
 relevant to and backported from there.
 
+WARNING
+=======
+
+This branch is under heavy development.  If you are interested in deploying a
+working cluster, please utilize a release branch.
 
 # OpenShift Ansible
 
 This repository contains [Ansible](https://www.ansible.com/) roles and
 playbooks to install, upgrade, and manage
 [OpenShift](https://www.openshift.com/) clusters.
 
-**Note**: the Ansible playbooks in this repository require an RPM
-package that provides `docker`. Currently, the RPMs from
-[dockerproject.org](https://dockerproject.org/) do not provide this
-requirement, though they may in the future. This limitation is being
-tracked by
-[#2720](https://github.com/openshift/openshift-ansible/issues/2720).
-
 ## Getting the correct version
 When choosing an openshift release, ensure that the necessary origin packages
 are available in your distribution's repository.  By default, openshift-ansible
@@ -69,7 +67,7 @@ Install base dependencies:
 
 Requirements:
 
-- Ansible >= 2.6.5, Ansible 2.7 is not yet supported and known to fail
+- Ansible >= 2.7.8
 - Jinja >= 2.7
 - pyOpenSSL
 - python-lxml
@@ -82,17 +80,6 @@ Fedora:
 dnf install -y ansible pyOpenSSL python-cryptography python-lxml
 ```
 
-Additional requirements:
-
-Logging:
-
-- java-1.8.0-openjdk-headless
-- patch
-
-Metrics:
-
-- httpd-tools
-
 ## Simple all-in-one localhost Installation
 This assumes that you've installed the base dependencies and you're running on
 Fedora or RHEL
@@ -102,62 +89,85 @@ cd openshift-ansible
 sudo ansible-playbook -i inventory/hosts.localhost playbooks/prerequisites.yml
 sudo ansible-playbook -i inventory/hosts.localhost playbooks/deploy_cluster.yml
 ```
-## Node Group Definition and Mapping
-In 3.10 and newer all members of the [nodes] inventory group must be assigned an
-`openshift_node_group_name`. This value is used to select the configmap that
-configures each node. By default there are three configmaps created; one for
-each node group defined in `openshift_node_groups` and they're named
-`node-config-master` `node-config-infra` `node-config-compute`. It's important
-to note that the configmap is also the authoritative definition of node labels,
-the old `openshift_node_labels` value is effectively ignored.
-
-There are also two configmaps that label nodes into multiple roles, these are
-not recommended for production clusters, however they're named
-`node-config-all-in-one` and `node-config-master-infra` if you'd like to use
-them to deploy non production clusters.
-
-The default set of node groups is defined in
-[roles/openshift_facts/defaults/main.yml] like so
 
-```
-openshift_node_groups:
-  - name: node-config-master
-    labels:
-      - 'node-role.kubernetes.io/master=true'
-    edits: []
-  - name: node-config-infra
-    labels:
-      - 'node-role.kubernetes.io/infra=true'
-    edits: []
-  - name: node-config-compute
-    labels:
-      - 'node-role.kubernetes.io/compute=true'
-    edits: []
-  - name: node-config-master-infra
-    labels:
-      - 'node-role.kubernetes.io/infra=true,node-role.kubernetes.io/master=true'
-    edits: []
-  - name: node-config-all-in-one
-    labels:
-      - 'node-role.kubernetes.io/infra=true,node-role.kubernetes.io/master=true,node-role.kubernetes.io/compute=true'
-    edits: []
+# Quickstart
+
+Install the new installer from https://www.github.com/openshift/installer
+
+Construct a proper install-config.yml, and make a copy called
+install-config-ansible.yml.
+
+## Hosts
+You will need the following hosts
+
+### Boostrap host
+This is a special host that is not part of the cluster but is required to be
+available to help the cluster bootstrap itself.  This is not a bastion host,
+it will initially be part of the cluster and should be able to communicate with
+the masters in the cluster.
+
+### Masters
+You need 1 or 3 masters.
+
+### Workers
+You need 0 or more workers.  Note, by default, masters are unschedulable so
+you will need one or more workers if you want to schedule workloads.
+
+## DNS
+4.x installs require specific dns records to be in place, and there is no way
+to complete an install without working DNS.  You are in charge of ensuring the
+following DNS records are resolvable from your cluster, the openshift-ansible
+installer will not make any attempt to do any of this for you.
+
+First, the output of ```hostname``` on each host must be resolvable to other hosts.
+The nodes will communicate with each other based on this value.
+
+install-config.yml value of 'baseDomain' must be a working domain.
+
+### A records
+```sh
+<clustername>-api.<baseDomain> # ex: mycluster-api.example.com
+<clustername>-master-0.<baseDomain> # ex: mycluster-master-0.example.com
+<clustername>-etcd-0.<baseDomain> # ex: mycluster-etcd-0.example.com
+<clustername>-bootstrap.<baseDomain> # ex: mycluster-bootstrap.example.com
 ```
 
-When configuring this in the INI based inventory this must be translated into a
-Python dictionary. Here's an example of a group named `node-config-all-in-one`
-which is suitable for an All-In-One installation with
-kubeletArguments.pods-per-core set to 20
+Note: There should be a master/etcd record for each master host in your cluster
+(either 1 or 3).  etcd hosts must be master hosts, and the records must resolve
+to the same host for each master/etcd record, respectively.
 
+### SRV records
+```sh
+SRV _etcd-client-ssl._tcp.<clustername>.<baseDomain> '1 1 2379 <clustername>-etcd-0.<baseDomain>'
+SRV _etcd-server-ssl._tcp.<clustername>.<baseDomain> '1 1 2380 <clustername>-etcd-0.<baseDomain>'
+...
+SRV _etcd-client-ssl._tcp.<clustername>.<baseDomain> '1 1 2379 <clustername>-etcd-<N-1>.<baseDomain>'
+SRV _etcd-server-ssl._tcp.<clustername>.<baseDomain> '1 1 2380 <clustername>-etcd-<N-1>.<baseDomain>'
+
+# ex: _etcd-client-ssl._tcp.mycluster.example.com '1 1 2379 mycluster-etcd-0.example.com'
 ```
-openshift_node_groups=[{'name': 'node-config-all-in-one', 'labels': ['node-role.kubernetes.io/master=true', 'node-role.kubernetes.io/infra=true', 'node-role.kubernetes.io/compute=true'], 'edits': [{ 'key': 'kubeletArguments.pods-per-core','value': ['20']}]}]
+
+Consult with your DNS provider about the proper way to create SRV records.  In
+any case, there should be a client and server SRV record for each etcd backend,
+and you MUST use the etcd FQDN you created earlier, not the master or any other
+record.
+
+## Inventory
+Check out inventory/40_basic_inventory.ini for an example.
+
+## Generate ignition configs
+Use the openshift-install command to generate ignition configs utilizing the
+install-config.yml you created earlier.  This will consume the install-config.yml
+file, so ensure you have copied the file as mentioned previously.
+
+```sh
+openshift-install create ignition-configs
 ```
 
-For upgrades, the upgrade process will block until you have the required
-configmaps in the openshift-node namespace. Please define
-`openshift_node_groups` as explained above or accept the defaults and run the
-playbooks/openshift-master/openshift_node_group.yml playbook to have them
-created for you automatically.
+## Run playbook
+playbooks/deploy_cluster_40.yml
 
+# Further reading
 
 ## Complete Production Installation Documentation:
 

images/installer/Dockerfile

@@ -4,40 +4,31 @@ MAINTAINER OpenShift Team <dev@lists.openshift.redhat.com>
 
 USER root
 
-# Add origin repo for including the oc client
-COPY images/installer/origin-extra-root /
-
-# install ansible and deps
-RUN INSTALL_PKGS="python-lxml python-dns pyOpenSSL python2-cryptography openssl python2-passlib httpd-tools openssh-clients origin-clients iproute patch" \
- && yum install -y --setopt=tsflags=nodocs $INSTALL_PKGS \
- && EPEL_PKGS="ansible-2.7.4 python2-boto python2-crypto which python2-pip.noarch python2-scandir python2-packaging azure-cli-2.0.46" \
- && yum install -y epel-release \
- && yum install -y --setopt=tsflags=nodocs $EPEL_PKGS \
- && if [ "$(uname -m)" == "x86_64" ]; then yum install -y https://sdodson.fedorapeople.org/google-cloud-sdk-183.0.0-3.el7.x86_64.rpm ; fi \
- && yum install -y java-1.8.0-openjdk-headless \
- && rpm -V $INSTALL_PKGS $EPEL_PKGS $EPEL_TESTING_PKGS \
- && pip install 'apache-libcloud~=2.2.1' 'SecretStorage<3' 'ansible[azure]' 'boto3==1.4.6' \
- && yum clean all
-
 LABEL name="openshift/origin-ansible" \
       summary="OpenShift's installation and configuration tool" \
-      description="A containerized openshift-ansible image to let you run playbooks to install, upgrade, maintain and check an OpenShift cluster" \
+      description="A containerized openshift-ansible image to use in CI - includes necessary packages to test clusters on AWS/GCP/Azure" \
       url="https://github.com/openshift/openshift-ansible" \
       io.k8s.display-name="openshift-ansible" \
-      io.k8s.description="A containerized openshift-ansible image to let you run playbooks to install, upgrade, maintain and check an OpenShift cluster" \
+      io.k8s.description="A containerized openshift-ansible image to use in CI - includes necessary packages to test clusters on AWS/GCP/Azure" \
       io.openshift.expose-services="" \
       io.openshift.tags="openshift,install,upgrade,ansible" \
       atomic.run="once"
 
 ENV USER_UID=1001 \
-    HOME=/opt/app-root/src \
+    HOME=/opt/app-root/src/ \
     WORK_DIR=/usr/share/ansible/openshift-ansible \
     OPTS="-v"
 
 # Add image scripts and files for running as a system container
 COPY images/installer/root /
-# Include playbooks, roles, plugins, etc. from this repo
-COPY . ${WORK_DIR}
+# Add origin repo for including the oc client
+COPY images/installer/origin-extra-root /
+# Install openshift-ansible RPMs
+RUN yum install -y epel-release && \
+    yum-config-manager --enable built > /dev/null && \
+    yum install --setopt=tsflags=nodocs -y \
+      openshift-ansible-test openshift-clients && \
+    yum clean all
 
 RUN /usr/local/bin/user_setup \
  && rm /usr/local/bin/usage.ocp

images/installer/origin-extra-root/etc/yum.repos.d/centos-ansible27.repo

@@ -1,6 +1,6 @@
 
-[centos-ansible26-testing]
-name=CentOS Ansible 2.6 testing repo
+[centos-ansible27-testing]
+name=CentOS Ansible 2.7 testing repo
 baseurl=https://cbs.centos.org/repos/configmanagement7-ansible-27-testing/x86_64/os/
 enabled=1
 gpgcheck=0

images/installer/origin-extra-root/etc/yum.repos.d/openshift-origin-4.0-ci.repo

@@ -0,0 +1,6 @@
+
+[openshift-origin-4.0-ci]
+name=OpenShift Origin 4.0 CI packages
+baseurl=https://rpms.svc.ci.openshift.org/openshift-origin-v4.0/
+enabled=1
+gpgcheck=0