Add the `testsys` cli and implement `cargo make test` for `aws-k8s` #2165

ecpullen · 2022-06-01T17:48:57Z

Issue number:
Closes #2147
Closes #2148

Description of changes:
This pr adds a simplified cli (testsys) that utilizes the bottlerocket-test-system to run sonobuoy tests on aws-k8s variants. The tests can be run using cargo make test.

The testsys cli comes with the ability to:

Check the status of tests and resources
Delete tests and resources from the testsys cluster
Install testsys components to a cluster
Stream the logs from a test or resource
Restart a test that errored
Run aws-k8s sonobuoy tests with cluster and instance creation.

Cargo make targets:

test-tools - Install the testsys cli
setup-test - Install testsys to the testsys cluster (requires KUBECONFIG to be set)
test - Run a testsys test on aws-k8s variants
- Quick testing and certified conformance testing
clean-test - Delete all testsys tests and resources from the testsys cluster
uninstall-test - Uninstall testsys components from the testsys cluster
watch-test - Watch the status of all tests and resources
- Can use cargo make watch-test --arch x86_64 to target a specific arch/variant for tests
log-test - Stream the logs from a testsys test.
- Example use: cargo make test-logs x86-64-aws-k8s-121-test
- --follow can be used to continue the log stream as they become available
testsys - This is a wrapper for all testsys calls

Developer workflow
Using x86_64 and aws-k8s-1.21 to simplify things.

Checkout bottlerocket and make changes
Build bottlerocket: cargo make
Publish amis: cargo make ami

Install testsys to the testsys cluster: cargo make setup-test

[cargo-make] INFO - cargo make 0.35.12
[cargo-make] INFO - Build File: Makefile.toml
[cargo-make] INFO - Task: test-setup
[cargo-make] INFO - Profile: development
[cargo-make] INFO - Running Task: setup
[cargo-make] INFO - Running Task: fetch-sources
[cargo-make] INFO - Running Task: test-tools
[cargo-make] INFO - Running Task: test-setup
[2022-06-16T13:03:05Z INFO  testsys::install] testsys components were successfully installed.
[cargo-make] INFO - Build Done in 1.70 seconds.

Start the tests: cargo make test

  ```
  [cargo-make] INFO - cargo make 0.35.12
  [cargo-make] INFO - Build File: Makefile.toml
  [cargo-make] INFO - Task: test
  [cargo-make] INFO - Profile: development
  [cargo-make] INFO - Running Task: setup
  [cargo-make] INFO - Running Task: fetch-sources
  [cargo-make] INFO - Running Task: test-tools
  [cargo-make] INFO - Running Task: test
  [2022-06-16T13:13:49Z INFO  testsys::run] Successfully added 'x86-64-aws-k8s-121'
  [2022-06-16T13:13:49Z INFO  testsys::run] Successfully added 'x86-64-aws-k8s-121-instances'
  [2022-06-16T13:13:49Z INFO  testsys::run] Successfully added 'x86-64-aws-k8s-121-test'
  [cargo-make] INFO - Build Done in 1.49 seconds.
  ```

<\details>

Check the status of the test: cargo make watch-test

     NAME                                             TYPE          STATE           PASSED   SKIPPED   FAILED
     x86-64-aws-k8s-121                   Resource   completed
     x86-64-aws-k8s-121-instance   Resource   completed
     x86-64-aws-k8s-121-test           Test            passed           1               5772         0

<\details>

Check the logs from a test: cargo make log-test x86-64-aws-k8s-121-test --follow

Clean up tests and resources: cargo make clean-test

[cargo-make] INFO - cargo make 0.35.12
[cargo-make] INFO - Build File: Makefile.toml
[cargo-make] INFO - Task: clean-test
[cargo-make] INFO - Profile: development
[cargo-make] INFO - Running Task: setup
[cargo-make] INFO - Running Task: fetch-sources
[cargo-make] INFO - Running Task: test-tools
[cargo-make] INFO - Running Task: clean-test
[2022-06-16T13:20:45Z INFO  testsys::delete] Delete finished for x86-64-aws-k8s-121-test
[2022-06-16T13:22:05Z INFO  testsys::delete] Delete finished for x86-64-aws-k8s-121-instances
[2022-06-16T13:22:15Z INFO  testsys::delete] Delete finished for x86-64-aws-k8s-121
[2022-06-16T13:22:15Z INFO  testsys::delete] Delete finished
[cargo-make] INFO - Build Done in 101.55 seconds.

<\details>

Uninstall testsys from the cluster: cargo make uninstall-test

[cargo-make] INFO - cargo make 0.35.12
[cargo-make] INFO - Build File: Makefile.toml
[cargo-make] INFO - Task: uninstall-test
[cargo-make] INFO - Profile: development
[cargo-make] INFO - Running Task: setup
[cargo-make] INFO - Running Task: fetch-sources
[cargo-make] INFO - Running Task: test-tools
[cargo-make] INFO - Running Task: uninstall-test
namespace "testsys-bottlerocket-aws" deleted
[cargo-make] INFO - Build Done in 45.74 seconds.

<\details>

cargo make testsys --help

```
[cargo-make] INFO - cargo make 0.35.12
[cargo-make] INFO - Build File: Makefile.toml
[cargo-make] INFO - Task: testsys
[cargo-make] INFO - Profile: development
[cargo-make] INFO - Running Task: setup
[cargo-make] INFO - Running Task: fetch-sources
[cargo-make] INFO - Running Task: test-tools
[cargo-make] INFO - Running Task: testsys
testsys 
A program for running and controlling Bottlerocket tests in a Kubernetes cluster using
https://github.com/bottlerocket-os/bottlerocket-test-system

USAGE:
    testsys [OPTIONS] <SUBCOMMAND>

OPTIONS:
    -h, --help                       Print help information
        --kubeconfig <KUBECONFIG>    Path to the kubeconfig file for the testsys cluster. Can also
                                     be passed with the KUBECONFIG environment variable
        --log-level <LOG_LEVEL>      How much detail to log; from least to most: ERROR, WARN, INFO,
                                     DEBUG, TRACE [default: INFO]

SUBCOMMANDS:
    add             Add a testsys object to the testsys cluster
    delete          Delete all tests and resources from a testsys cluster
    help            Print this message or the help of the given subcommand(s)
    install         The install subcommand is responsible for putting all of the necessary
                        components for testsys in a k8s cluster
    logs            Stream the logs of an object from a testsys cluster
    restart-test    Restart a test. This will delete the test object from the testsys cluster
                        and replace it with a new, identical test object with a clean state
    run             Run a set of tests on a given arch and variant
    status          Check the status of a TestSys object
[cargo-make] INFO - Build Done in 1.39 seconds.
```
<\details>

Previous description

``` testsys A program for running and controlling Bottlerocket tests in a Kubernetes cluster using https://github.com/bottlerocket-os/bottlerocket-test-system

USAGE:
testsys [OPTIONS]

OPTIONS:
-h, --help Print help information
--kubeconfig Path to the kubeconfig file for the testsys cluster. Can also
be passed with the KUBECONFIG environment variable

SUBCOMMANDS:
add Add a testsys object to the testsys cluster
delete Delete all tests and resources from a testsys cluster
help Print this message or the help of the given subcommand(s)
install The install subcommand is responsible for putting all of the necessary
components for testsys in a k8s cluster
logs Stream the logs of an object from a testsys cluster
restart-test Restart a test. This will delete the test object from the testsys cluster
and replace it with a new, identical test object with a clean state
run Run a set of tests on a given arch and variant
status Check the status of a TestSys object



**Testing done:**

Successfully followed instructions from `TESTING.md`.

Also tested using:
`cargo make install-testsys`
`cargo make testsys-setup`
`cargo make build`
`cargo make -e PUBLISH_REGIONS="us-west-2" ami`
`cargo make test`
 The sonobuoy quick test was completed successfully.

Additional testing:
`testsys status`
`testsys delete`
`testsys add secret map`
</details>
**Terms of contribution:**

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

ecpullen · 2022-06-02T18:00:55Z

^ Removes inconsistencies between TESTING.md and the testsys cli.

ecpullen · 2022-06-02T18:08:09Z

^ Cargo fmt

ecpullen · 2022-06-03T23:31:49Z

^ rebase to develop

TESTING.md

etungsten

🪐

Makefile.toml

TESTING.md

tools/testsys/src/main.rs

Makefile.toml

TESTING.md

tools/testsys/src/install.rs

tools/testsys/src/delete.rs

tools/testsys/src/logs.rs

tools/testsys/src/main.rs

tools/testsys/src/run.rs

ecpullen · 2022-06-15T15:51:36Z

^ Addresses all comments. Adds new cargo make targets for test logs, status, uninstall, and cargo make testsys.

etungsten

Can you please update the PR description to include testing of the new cargo make tasks. A demonstration of the invocation and what get printed out would be nice.

Makefile.toml

webern

We have been clear in documentation that the testing feature is in rapid iteration, so none of my comments are blocking. This is great! 🚀

webern · 2022-06-17T18:29:33Z

Makefile.toml

+    '''
+]
+
+[tasks.test-setup]


This name isn't working for me. Most of the other targets are verb-noun and it's not entirely clear what test-setup might be doing. It's not a blocker but here are a couple of ideas:

setup-test-cluster

install-test-system

Makefile.toml

webern · 2022-06-17T18:36:12Z

tools/testsys/src/status.rs

+use model::test_manager::{SelectionParams, TestManager};
+use terminal_size::{Height, Width};
+
+/// Check the status of a TestSys object.


Suggested change

/// Check the status of a TestSys object.

/// Check the status of TestSys objects.

tools/testsys/src/secret.rs

webern · 2022-06-17T18:40:19Z

tools/testsys/src/run.rs

+pub(crate) struct Image {
+    pub(crate) id: String,
+    #[serde(rename = "name")]
+    pub(crate) _name: String,


Why is this here? Is it required by k8s? Maybe document why there is an unused sctruct field.

tools/testsys/src/run.rs

webern · 2022-06-17T18:52:53Z

tools/testsys/src/run.rs

+#[derive(Debug, Deserialize)]
+#[serde(rename_all = "lowercase")]
+pub(crate) enum TestType {
+    /// Run conformance testing on a given arch and variant


I think these could be more generic and more specific at the same time:

Suggested change

/// Run conformance testing on a given arch and variant

/// Conformance testing is a full integration test that asserts that Bottlerocket is working for

/// customer workloads. For k8s variants, for example, this will run the full suite of sonobuoy

/// conformance tests.

webern · 2022-06-17T18:56:42Z

tools/testsys/src/run.rs

+pub(crate) enum TestType {
+    /// Run conformance testing on a given arch and variant
+    Conformance,
+    /// Run a quick test on a given arch and variant


I've mentioned ECS here even though it's not yet supported. But ECS is a fast-follow, and I think this is better than forgetting to augment the documentation with the ECS example.

Suggested change

/// Run a quick test on a given arch and variant

/// Run a quick test that ensures a basic workload can run on Bottlerocket. For example, on k8s

/// variance this will run sonobuoy in "quick" mode. For ECS variants, this will run a simple

/// ECS task.

webern · 2022-06-17T19:02:06Z

tools/testsys/src/logs.rs

+    #[clap(long = "state", conflicts_with = "test")]
+    resource_state: Option<ResourceState>,
+
+    /// Include logs from dependencies.


This is unused, right? Delete.

webern · 2022-06-17T19:03:35Z

tools/testsys/src/delete.rs

+                DeleteEvent::Starting(crd) => debug!("Starting delete for {}", crd.name()),
+                DeleteEvent::Deleted(crd) => info!("Delete finished for {}", crd.name()),
+                DeleteEvent::Failed(crd) => warn!("Delete failed for {}", crd.name()),


I actually think these should should be regular print statements instead of logging statements. i.e. I think these are part of the normal CLI user interface.

arnaldo2792

First round done! I have to come back and review a few more things, but great job!

arnaldo2792 · 2022-06-21T23:33:12Z

TESTING.md

+
+## Unit Tests
+
+Unit tests are easy, you can run them from the root of the repo with `cargo make unit-tests`.


Nit: The phrase Units tests are easy made me think that the doc suggests that unit tests are easy to write, not to run, should the line be more specific? Like "It is easy to execute unit tests, you can run them..."

arnaldo2792 · 2022-06-21T23:35:12Z

TESTING.md

+## Unit Tests
+
+Unit tests are easy, you can run them from the root of the repo with `cargo make unit-tests`.
+Note that some code in Bottlerocket is conditionally compiled based on variant.


Nit: maybe mention the side effect of the conditional compilation? "thus some tests won't be executed", or tell the users that some of their test might not run if the configuration isn't set properly?

tools/testsys/src/aws_resources.rs

arnaldo2792 · 2022-06-21T23:50:03Z

tools/testsys/src/install.rs

+    #[clap(
+        long = "controller-uri",
+        env = "TESTSYS_CONTROLLER_IMAGE",
+        default_value = "public.ecr.aws/bottlerocket-test-system/controller:v0.0.1"


I wonder if this will bite us in the future, we will have to remember to update this value whenever we release a new version of the images.

arnaldo2792 · 2022-06-21T23:57:55Z

tools/testsys/src/run.rs

+                let file = File::open(&self.ami_input).context("Unable to open amis.json")?;
+                let ami_input: HashMap<String, Image> = serde_json::from_reader(file)
+                    .context(format!("Unable to deserialize '{}'", self.ami_input))?;
+                ensure!(!ami_input.is_empty(), "amis.json is empty");
+                let bottlerocket_ami = &ami_input
+                    .get(&region)
+                    .context(format!("ami not found for region '{}'", region))?
+                    .id;
+                debug!("Using ami '{}'", bottlerocket_ami);


nit: this could be a standalone function, that takes ami_input, and returns the AMI id.

arnaldo2792 · 2022-06-22T15:30:39Z

tools/testsys/src/aws_resources.rs

+        let cluster_name = self.cluster_name(cluster_suffix);
+        let mut ec2_config = Ec2Config {
+            node_ami: self.ami.clone(),
+            // TODO - configurable


Nit: is there a task for this?

No. This just creates an ec2 instance.

Maybe I wasn't clear, I meant for the TODO, is there a task for the TODO?

tools/testsys/src/aws_resources.rs

tools/testsys/src/logs.rs

Makefile.toml

TESTING.md

tools/testsys/src/run.rs

bcressey · 2022-06-23T17:01:00Z

tools/testsys/src/run.rs

+    // This is used to deserialize amis.json
+    #[serde(rename = "name")]


Won't serde ignore unknown fields like this by default?

tools/testsys/src/uninstall.rs

.gitignore

tools/testsys/src/main.rs

TESTING.md

arnaldo2792

👢

ecpullen changed the title ~~Testsys~~ Add the testsys cli and implement cargo make test for aws-k8s Jun 1, 2022

ecpullen force-pushed the testsys branch from 92bf149 to fa8ff43 Compare June 2, 2022 17:59

ecpullen force-pushed the testsys branch from fa8ff43 to 2a785b0 Compare June 2, 2022 18:07

ecpullen force-pushed the testsys branch from 2a785b0 to 701c65e Compare June 3, 2022 23:31

ecpullen marked this pull request as ready for review June 6, 2022 16:54

mchaker reviewed Jun 13, 2022

View reviewed changes

TESTING.md Show resolved Hide resolved

etungsten reviewed Jun 13, 2022

View reviewed changes

bcressey reviewed Jun 14, 2022

View reviewed changes

ecpullen force-pushed the testsys branch 3 times, most recently from 420dde6 to 399abbe Compare June 15, 2022 15:49

ecpullen requested a review from webern June 15, 2022 15:52

ecpullen force-pushed the testsys branch from 399abbe to b9fcf11 Compare June 15, 2022 15:53

ecpullen requested a review from etungsten June 15, 2022 15:58

ecpullen force-pushed the testsys branch from b9fcf11 to 9ca429d Compare June 15, 2022 16:00

etungsten reviewed Jun 15, 2022

View reviewed changes

Makefile.toml Outdated Show resolved Hide resolved

Makefile.toml Outdated Show resolved Hide resolved

ecpullen force-pushed the testsys branch from 9ca429d to ec6c4f9 Compare June 16, 2022 13:36

ecpullen requested a review from etungsten June 16, 2022 13:40

ecpullen force-pushed the testsys branch 2 times, most recently from 11a746c to 027ceeb Compare June 16, 2022 13:54

webern approved these changes Jun 17, 2022

View reviewed changes

ecpullen force-pushed the testsys branch 2 times, most recently from 2426ddc to bf75a29 Compare June 21, 2022 11:53

etungsten approved these changes Jun 21, 2022

View reviewed changes

ecpullen requested a review from bcressey June 22, 2022 12:34

arnaldo2792 reviewed Jun 22, 2022

View reviewed changes

ecpullen force-pushed the testsys branch from bf75a29 to fdab82a Compare June 23, 2022 12:06

ecpullen requested a review from arnaldo2792 June 23, 2022 12:07

bcressey reviewed Jun 23, 2022

View reviewed changes

ecpullen force-pushed the testsys branch from fdab82a to 55e2e51 Compare June 24, 2022 12:34

ecpullen requested a review from bcressey June 24, 2022 13:58

bcressey approved these changes Jun 29, 2022

View reviewed changes

arnaldo2792 requested changes Jul 7, 2022

View reviewed changes

test: Add cargo make targets for testsys tests

dc09d23

ecpullen force-pushed the testsys branch from 55e2e51 to dc09d23 Compare July 11, 2022 12:29

ecpullen requested a review from arnaldo2792 July 11, 2022 13:16

arnaldo2792 approved these changes Jul 12, 2022

View reviewed changes

ecpullen merged commit f8f8a80 into bottlerocket-os:develop Jul 14, 2022

ecpullen deleted the testsys branch December 15, 2022 16:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add the `testsys` cli and implement `cargo make test` for `aws-k8s` #2165

Add the `testsys` cli and implement `cargo make test` for `aws-k8s` #2165

ecpullen commented Jun 1, 2022 •

edited

Loading

ecpullen commented Jun 2, 2022

ecpullen commented Jun 2, 2022

ecpullen commented Jun 3, 2022

etungsten left a comment

ecpullen commented Jun 15, 2022

etungsten left a comment

webern left a comment

webern Jun 17, 2022

webern Jun 17, 2022

webern Jun 17, 2022

webern Jun 17, 2022

webern Jun 17, 2022

webern Jun 17, 2022

webern Jun 17, 2022

arnaldo2792 left a comment

arnaldo2792 Jun 21, 2022

arnaldo2792 Jun 21, 2022

arnaldo2792 Jun 21, 2022

arnaldo2792 Jun 21, 2022

arnaldo2792 Jun 22, 2022

ecpullen Jun 23, 2022

arnaldo2792 Jul 7, 2022

bcressey Jun 23, 2022

arnaldo2792 left a comment

	/// Check the status of a TestSys object.
	/// Check the status of TestSys objects.

-    /// Run conformance testing on a given arch and variant
+    /// Conformance testing is a full integration test that asserts that Bottlerocket is working for
+    /// customer workloads. For k8s variants, for example, this will run the full suite of sonobuoy
+    /// conformance tests.

-    /// Run a quick test on a given arch and variant
+    /// Run a quick test that ensures a basic workload can run on Bottlerocket. For example, on k8s
+    /// variance this will run sonobuoy in "quick" mode. For ECS variants, this will run a simple
+    /// ECS task.


		## Unit Tests

		Unit tests are easy, you can run them from the root of the repo with `cargo make unit-tests`.

		// This is used to deserialize amis.json
		#[serde(rename = "name")]

Add the testsys cli and implement cargo make test for aws-k8s #2165

Add the testsys cli and implement cargo make test for aws-k8s #2165

Conversation

ecpullen commented Jun 1, 2022 • edited Loading

ecpullen commented Jun 2, 2022

ecpullen commented Jun 2, 2022

ecpullen commented Jun 3, 2022

etungsten left a comment

Choose a reason for hiding this comment

ecpullen commented Jun 15, 2022

etungsten left a comment

Choose a reason for hiding this comment

webern left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arnaldo2792 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arnaldo2792 left a comment

Choose a reason for hiding this comment

Add the `testsys` cli and implement `cargo make test` for `aws-k8s` #2165

Add the `testsys` cli and implement `cargo make test` for `aws-k8s` #2165

ecpullen commented Jun 1, 2022 •

edited

Loading