diff --git a/tests/README.md b/tests/README.md index fc3eac7f93..ec7d34e3be 100644 --- a/tests/README.md +++ b/tests/README.md @@ -5,7 +5,6 @@ Our basic set of tests includes: - Code linting -- UI tests - Backend unit tests They are run on **every** PR by default. Successful basic tests are required in @@ -15,30 +14,13 @@ temporarily merged into the target branch of the pull request. ### Actions required - **none** -### FAQ -- *I am not part of the `coreos` or `coreos-inc` GitHub organization. Why are - the tests not being executed on my changes?* - - Pull requests by external contributors need to be checked before they are - tested by our Jenkins setup. Please ask a [maintainer](../MAINTAINERS) to mark - your PR via commenting `ok to test`. - -- *How do I retrigger the tests?* - - Comment with `ok to test` on the PR. - -## Running smoke / k8s-conformance tests on PRs +## Running smoke -In addition to our basic set of tests we have smoke tests and the k8s upstream -conformance tests. These test the Tectonic installer on our supported platforms: -- AWS -- libvirt +In addition to our basic set of tests we have smoke tests which are running on AWS platform only. ### Actions required -- Add the `run-smoke-tests` or/and the `run-conformance-tests` GitHub label -- Add the `platform/` GitHub label for **each** platform you want to run - tests against +- Add the `run-smoke-tests` GitHub label ### FAQ - *I am not able to add labels, what should I do?* @@ -54,11 +36,6 @@ conformance tests. These test the Tectonic installer on our supported platforms: Yes, just add the GitHub labels and comment `ok to test` on the PR. -- *What if I only add the `run-smoke-tests`/`run-conformance-tests` GitHub - label, but no `platform/` label?* - - No smoke / conformance tests will be executed. - - *What if I trigger the tests twice in a small time frame?* Triggering the tests twice in a small time frame results in two test @@ -76,113 +53,27 @@ conformance tests. These test the Tectonic installer on our supported platforms: ## Nightly Runs -We use Jenkins to run our tests and we have two Jenkins jobs to run our `nightly` builds against the master branch. -One job runs all smoke tests for all platforms we have tests for. -The other job runs the conformance tests for one or all platforms we support. Today this job is running -the conformance tests for `AWS` - -Those builds report the status to our internal chat tool. +We use Jenkins to run our tests and our `nightly` builds against the master branch. +Those jobs are executing smoke tests and building docker images. -## Running smoke tests / k8s-conformance tests locally +## Running smoke tests locally ### 1. Expose environment variables -To run the smoke tests / conformance tests locally you need to set the following +To run the smoke tests locally you need to set the following environment variables: ``` bash -CLUSTER -TF_VAR_tectonic_license_path -TF_VAR_tectonic_pull_secret_path -TF_VAR_tectonic_base_domain -TF_VAR_tectonic_admin_email -TF_VAR_tectonic_admin_password -TECTONIC_TESTS_DONT_CLEAN_UP // If you want to keep the cluster alive after the tests -RUN_SMOKE_TESTS=true -RUN_CONFORMANCE_TESTS=true -KUBE_CONFORMANCE_IMAGE=quay.io/coreos/kube-conformance:v1.7.5_coreos.0_golang1.9.1 -COMPONENT_TEST_IMAGES=quay.io/coreos/tectonic-console-tester:v2.7.1, -``` - -And depending on your platform: - -#### AWS -``` bash AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY -TF_VAR_tectonic_aws_ssh_key -TF_VAR_tectonic_aws_region +LICENSE_PATH +PULL_SECRET_PATH ``` -### 2. Launch the tests -Once the environment variables are set, run `make tests/smoke -TEST=aws/basic_spec.rb`, where `TEST=` represents the test spec you want to -run. - - -### 3. Running the rspec tests in a existing cluster -To run the test suite on an existing cluster, use this command: -```sh -PLATFORM=AWS rspec spec/existing_cluster_spec.rb +Optionally you can also set: +```bash +DOMAIN +AWS_REGION ``` - - -## Running Conformance tests in a running cluster - -There are two ways to run the conformance tests in an existing k8s cluster. -One way is to use the docker image that we provide, you can build using the [docker file](../images/kubernetes-e2e) -or use the existing image which is available in [quay.io](https://quay.io/repository/coreos/kube-conformance?tab=tags0). - -The second approach is to use the same process as described in the [CNCF K8s certification](https://github.com/cncf/k8s-conformance/blob/master/instructions.md) - -We will describe both approaches. - -### 1. Running Conformance tests using `quay.io/coreos/kube-conformance` - -To run the conformance testing using the docker image we build you will need to the following: - -* Have a running cluster -* Kube Config file - -Then, you need to execute the following command in your shell: - -```Bash -$ docker run -v :/kubeconfig quay.io/coreos/kube-conformance: -``` - -where: -`` is the version of the Kubernetes we are using, i.e, `v1.8.2_coreos.0` -The first part of the tag `v1.8.2` means the kubernetes version. -The second part `_coreos.0` is related to hyperkube which you can find [here](https://quay.io/repository/coreos/hyperkube?tag=latest&tab=tags) -Usually the version of the conformance image should match with the cluster you have, -i.e. if you have a K8s cluster running version 1.7.5 it is better run the 1.7.5 conformance test image (in this case: `v1.7.5_coreos.0`) - -The CNCF agreed for the versions 1.7.X and 1.8.X you can run the same set of tests that exist for the version 1.8.X. -Above this version (i.e. > 1.9.X) there is not guideline yet, but you might need to run the conformance tests for those versions. - - -### 2. Running Conformance tests using the `CNCF Process` - -If you want to run the conformance tests as the same way the CNCF certification process does, -you will need to do the following: - -* Have a running cluster -* Kube Config file - -```Bash -$ curl -L https://raw.githubusercontent.com/cncf/k8s-conformance/master/sonobuoy-conformance.yaml | kubectl apply -f - -``` - -Watch Sonobuoy's logs with `kubectl logs -f -n sonobuoy sonobuoy` and wait for the line `no-exit was specified, sonobuoy is now blocking`. -When that appears you can copy the results using the following command: - -```Bash -$ kubectl cp sonobuoy/sonobuoy:/tmp/sonobuoy ./results -``` - -#### 2.1. Submitting the results to `CNCF Certification` - -If you need to submit the results to the `CNCF` you can just follow the instructions described [here](https://github.com/cncf/k8s-conformance/blob/master/instructions.md). - -Which is basically run the Conformance tests described in Section 2 and create some files and then open a Pull Request in the [CNCF K8s Conformance](https://github.com/cncf/k8s-conformance). - +### 2. Launch the tests +Once the environment variables are set, run `./tests/run.sh`.