Skip to content

Commit 156c152

Browse files
heysteheyste
authored
Blog: Creating an e2e test for conformance (#22)
* Blog: Creating an e2e test for conformance * Updates after feedback * Add query for ServiceStatus endpoints * Touch ups after feedback
1 parent f14083e commit 156c152

File tree

2 files changed

+220
-0
lines changed

2 files changed

+220
-0
lines changed

‎blog/content/blog/creating-an-e2e-test-for-conformance.md

+220
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,220 @@
1+
+++
2+
title = "Creating an e2e test for Conformance"
3+
author = ["Stephen Heywood"]
4+
date = 2021-05-11
5+
lastmod = 2021-05-13T09:02:00+13:00
6+
categories = ["kubernetes"]
7+
draft = false
8+
summary = "Finding untested stable endpoints and creating an e2e test for conformance."
9+
+++
10+
11+
## Introduction
12+
13+
Since the 1.19 release of Kubernetes, the gap in e2e conformance tested endpoints has decreased due in part to the processes and tooling that the team at [ii.coop](https://ii.coop/) have developed.
14+
15+
![img](/images/2021/apisnoop-progress.png)
16+
17+
The process starts by using [APIsnoop](https://github.com/cncf/apisnoop) (which uses a postgres database containing audit logs from e2e test runs) to identify a set of untested endpoints that are part of the stable API endpoints. During this process various groups or patterns of endpoints are discovered. One such group of endpoints are “DaemonSetStatus”. Next we will explore these endpoints, create an e2e test that exercises each of them, then merge this test into the k8s repo.
18+
19+
APIsnoop results for untested “DaemonSetStatus” endpoints in [untested_stable_endpoints table](https://github.com/cncf/apisnoop/blob/main/apps/snoopdb/tables-views-functions.org#untested_stable_endpoints)
20+
21+
```sql
22+
select
23+
endpoint,
24+
path,
25+
kind
26+
from testing.untested_stable_endpoint
27+
where eligible is true
28+
and endpoint ilike '%DaemonSetStatus'
29+
order by kind, endpoint desc;
30+
```
31+
32+
33+
```
34+
endpoint | path | kind
35+
------------------------------------------+---------------------------------------------------------------+------------
36+
replaceAppsV1NamespacedDaemonSetStatus | /apis/apps/v1/namespaces/{namespace}/daemonsets/{name}/status | DaemonSet
37+
readAppsV1NamespacedDaemonSetStatus | /apis/apps/v1/namespaces/{namespace}/daemonsets/{name}/status | DaemonSet
38+
patchAppsV1NamespacedDaemonSetStatus | /apis/apps/v1/namespaces/{namespace}/daemonsets/{name}/status | DaemonSet
39+
(3 rows)
40+
```
41+
42+
43+
# Connecting an endpoint to a resource
44+
45+
Here are three possible ways use to connect an API endpoint to a resource in a cluster
46+
47+
1. Some initial details about the endpoint can be found via the [API Reference](https://kubernetes.io/docs/reference/kubernetes-api/). For this example about Daemonset we can locate [read](https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/daemon-set-v1/#get-read-status-of-the-specified-daemonset), [patch](https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/daemon-set-v1/#patch-partially-update-status-of-the-specified-daemonset) and [replace](https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/daemon-set-v1/#update-replace-status-of-the-specified-daemonset) for Daemonset Status.
48+
49+
2. `kubectl` has an option to describe the fields associated with each supported API resource. The following example shows how it can provide details around ’status conditions’.
50+
51+
```
52+
$ kubectl explain daemonset.status.conditions
53+
KIND: DaemonSet
54+
VERSION: apps/v1
55+
56+
RESOURCE: conditions <[]Object>
57+
58+
DESCRIPTION:
59+
Represents the latest available observations of a DaemonSet's current
60+
state.
61+
62+
DaemonSetCondition describes the state of a DaemonSet at a certain point.
63+
64+
FIELDS:
65+
lastTransitionTime <string>
66+
Last time the condition transitioned from one status to another.
67+
68+
message <string>
69+
A human readable message indicating details about the transition.
70+
71+
reason <string>
72+
The reason for the condition's last transition.
73+
74+
status <string> -required-
75+
Status of the condition, one of True, False, Unknown.
76+
77+
type <string> -required-
78+
Type of DaemonSet condition.
79+
```
80+
81+
3. Lastly, using both [APIsnoop in cluster](https://github.com/cncf/apisnoop/tree/main/kind) while reviewing the current [e2e test suite](https://github.com/kubernetes/kubernetes/tree/master/test/e2e) for existing conformance tests that test a similar set of endpoints. In this case we used [a Service Status test](https://github.com/kubernetes/kubernetes/blob/7b2776b89fb1be28d4e9203bdeec079be903c103/test/e2e/network/service.go#L2300-L2392) as a template for the new Daemonset test.
82+
83+
```sql
84+
with latest_release as (
85+
select release::semver as release
86+
from open_api
87+
order by release::semver desc
88+
limit 1
89+
)
90+
91+
select ec.endpoint, ec.path, ec.kind
92+
from endpoint_coverage ec
93+
join latest_release on(ec.release::semver = latest_release.release)
94+
where level = 'stable'
95+
and ec.endpoint ilike '%NamespacedServiceStatus'
96+
and tested is true
97+
ORDER BY endpoint desc;
98+
```
99+
100+
101+
```
102+
endpoint | path | kind
103+
--------------------------------------+-------------------------------------------------------+---------
104+
replaceCoreV1NamespacedServiceStatus | /api/v1/namespaces/{namespace}/services/{name}/status | Service
105+
readCoreV1NamespacedServiceStatus | /api/v1/namespaces/{namespace}/services/{name}/status | Service
106+
patchCoreV1NamespacedServiceStatus | /api/v1/namespaces/{namespace}/services/{name}/status | Service
107+
(3 rows)
108+
```
109+
110+
The Service status e2e test followed similar ideas and patterns from [/test/e2e/auth/certificates.go](https://github.com/kubernetes/kubernetes/blob/31030820be979ea0b2c39e08eb18fddd71f353ed/test/e2e/auth/certificates.go#L356-L383) and [/test/e2e/network/ingress.go](https://github.com/kubernetes/kubernetes/blob/31030820be979ea0b2c39e08eb18fddd71f353ed/test/e2e/network/ingress.go#L1091-L1127)
111+
112+
# Writing an e2e test
113+
114+
## Initial Exploration
115+
116+
Using [literate programming](https://wiki.c2.com/?LiterateProgramming) we created [Appsv1DaemonSetStatusLifecycleTest.org](https://github.com/apisnoop/ticket-writing/blob/create-daemonset-status-lifecycle-test/Appsv1DaemonSetStatusLifecycleTest.org)
117+
(via [pair](https://github.com/sharingio/pair)) to both test and document the explorations of the endpoints. This provides a clear outline that should be easily replicated and validated by others as needed.
118+
Once completed, the document is converted into markdown which becomes a GitHub [issue](https://github.com/kubernetes/kubernetes/issues/100437).
119+
120+
The issue provides the following before a PR is opened:
121+
- a starting point to discuss the endpoints
122+
- the approach taken to test them
123+
- whether they are [eligible for conformance](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md#conformance-test-requirements).
124+
125+
## Creating the e2e test
126+
127+
Utilizing the above document, the test is structured in to four parts;
128+
129+
1. Creating the resources for the test, in this case a DaemonSet and a &rsquo;watch&rsquo;.
130+
131+
2. Testing the first endpoint, `readAppsV1NamespacedReplicaSetStatus` via a [dynamic client](https://github.com/ii/kubernetes/blob/ca3aa6f5af1b545b116b52c717b866e43c79079b/test/e2e/apps/daemon_set.go#L841). This is due to the standard go client not being able to access the sub-resource. We also make sure there are no errors from either getting or decoding the response.
132+
133+
3. The next endpoint tested is `replaceAppsV1NamespacedDaemonSetStatus` which replaces all status conditions at the same time. As the resource version of the DaemonSet may change before the new status conditions are updated we may need to [retry the request if there is a conflict](https://github.com/ii/kubernetes/blob/ca3aa6f5af1b545b116b52c717b866e43c79079b/test/e2e/apps/daemon_set.go#L854). Monitoring the watch events for the Daemonset we can confirm that the status conditions have been [replaced](https://github.com/ii/kubernetes/blob/ca3aa6f5af1b545b116b52c717b866e43c79079b/test/e2e/apps/daemon_set.go#L884-L886).
134+
135+
4. The last endpoint tested is `patchAppsV1NamespacedReplicaSetStatus` which only patches a [single condition](https://github.com/ii/kubernetes/blob/ca3aa6f5af1b545b116b52c717b866e43c79079b/test/e2e/apps/daemon_set.go#L906) this time. Again, using the watch to monitor for events we can check that the single condition [has been updated](https://github.com/ii/kubernetes/blob/ca3aa6f5af1b545b116b52c717b866e43c79079b/test/e2e/apps/daemon_set.go#L931).
136+
137+
## Validating the e2e test
138+
139+
Using `go test` we can run a single test for quick feedback
140+
141+
```bash
142+
cd ~/go/src/k8s.io/kubernetes
143+
TEST_NAME="should verify changes to a daemon set status"
144+
go test ./test/e2e/ -v -timeout=0 --report-dir=/tmp/ARTIFACTS -ginkgo.focus="$TEST_NAME"
145+
```
146+
147+
Checking the e2e test logs we see that everything looks to be okay.
148+
149+
```
150+
[It] should verify changes to a daemon set status /home/ii/go/src/k8s.io/kubernetes/test/e2e/apps/daemon_set.go:812
151+
STEP: Creating simple DaemonSet "daemon-set"
152+
STEP: Check that daemon pods launch on every node of the cluster.
153+
May 10 17:36:36.106: INFO: Number of nodes with available pods: 0
154+
May 10 17:36:36.106: INFO: Node heyste-control-plane-fkjmr is running more than one daemon pod
155+
May 10 17:36:37.123: INFO: Number of nodes with available pods: 0
156+
May 10 17:36:37.123: INFO: Node heyste-control-plane-fkjmr is running more than one daemon pod
157+
May 10 17:36:38.129: INFO: Number of nodes with available pods: 0
158+
May 10 17:36:38.129: INFO: Node heyste-control-plane-fkjmr is running more than one daemon pod
159+
May 10 17:36:39.122: INFO: Number of nodes with available pods: 1
160+
May 10 17:36:39.122: INFO: Number of running nodes: 1, number of available pods: 1
161+
STEP: Getting /status
162+
May 10 17:36:39.142: INFO: Daemon Set daemon-set has Conditions: []
163+
STEP: updating the DaemonSet Status
164+
May 10 17:36:39.160: INFO: updatedStatus.Conditions: []v1.DaemonSetCondition{v1.DaemonSetCondition{Type:"StatusUpdate", Status:"True", LastTransitionTime:v1.Time{Time:time.Ti
165+
me{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Reason:"E2E", Message:"Set from e2e test"}}
166+
STEP: watching for the daemon set status to be updated
167+
May 10 17:36:39.163: INFO: Observed event: ADDED
168+
May 10 17:36:39.163: INFO: Observed event: MODIFIED
169+
May 10 17:36:39.163: INFO: Observed event: MODIFIED
170+
May 10 17:36:39.164: INFO: Observed event: MODIFIED
171+
May 10 17:36:39.164: INFO: Found daemon set daemon-set in namespace daemonsets-2986 with labels: map[daemonset-name:daemon-set] annotations: map[deprecated.daemonset.template
172+
.generation:1] & Conditions: [{StatusUpdate True 0001-01-01 00:00:00 +0000 UTC E2E Set from e2e test}]
173+
May 10 17:36:39.164: INFO: Daemon set daemon-set has an updated status
174+
STEP: patching the DaemonSet Status
175+
STEP: watching for the daemon set status to be patched
176+
May 10 17:36:39.180: INFO: Observed event: ADDED
177+
May 10 17:36:39.180: INFO: Observed event: MODIFIED
178+
May 10 17:36:39.181: INFO: Observed event: MODIFIED
179+
May 10 17:36:39.181: INFO: Observed event: MODIFIED
180+
May 10 17:36:39.181: INFO: Observed daemon set daemon-set in namespace daemonsets-2986 with annotations: map[deprecated.daemonset.template.generation:1] & Conditions: [{StatusUpdate True 0001-01-01 00:00:00 +0000 UTC E2E Set from e2e test}]
181+
May 10 17:36:39.181: INFO: Found daemon set daemon-set in namespace daemonsets-2986 with labels: map[daemonset-name:daemon-set] annotations: map[deprecated.daemonset.template.generation:1] & Conditions: [{StatusPatched True 0001-01-01 00:00:00 +0000 UTC }]
182+
May 10 17:36:39.181: INFO: Daemon set daemon-set has a patched status
183+
```
184+
185+
Verification that the test passed!
186+
187+
```
188+
Ran 1 of 5745 Specs in 18.473 seconds
189+
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 5744 Skipped
190+
--- PASS: TestE2E (18.62s)
191+
```
192+
193+
Using APISnoop with the audit logger we can also confirm that the endpoints where hit during the test.
194+
195+
```sql
196+
select distinct endpoint, right(useragent,75) AS useragent
197+
from testing.audit_event
198+
where endpoint ilike '%DaemonSetStatus%'
199+
and release_date::BIGINT > round(((EXTRACT(EPOCH FROM NOW()))::numeric)*1000,0) - 60000
200+
and useragent like 'e2e%'
201+
order by endpoint;
202+
```
203+
204+
```
205+
endpoint | useragent
206+
-----------------------------------------+-------------------------------------------------------------------
207+
patchAppsV1NamespacedReplicaSetStatus | [sig-apps] ReplicaSet should validate Replicaset Status endpoints
208+
readAppsV1NamespacedReplicaSetStatus | [sig-apps] ReplicaSet should validate Replicaset Status endpoints
209+
replaceAppsV1NamespacedReplicaSetStatus | [sig-apps] ReplicaSet should validate Replicaset Status endpoints
210+
(3 rows)
211+
```
212+
213+
Even though the test has passed here, once merged it will join other jobs on [TestGrid](https://testgrid.k8s.io/) to determine if the test is stable and after two weeks it can be [promoted to conformance](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md#promoting-tests-to-conformance).
214+
215+
216+
# Final Thoughts
217+
218+
The current workflow and tooling provides a high level of confidence when working through each e2e test. Following agreed coding patterns, styles and processes helps to minimise possible issues and test flakes. There&rsquo;s always opportunities to get support through GitHub tickets, [various Kubernetes slack channels](https://kubernetes.slack.com/messages/k8s-conformance) and conformance meetings.
219+
220+
Every e2e test that&rsquo;s merged and then promoted to conformance requires the input from a wide range of people. It is thanks to the support from community reviewers, SIGs and the direction provided by SIG-Architecture this work is not just possible but rewarding.

‎blog/static/images/2021/apisnoop-progress.png

66.8 KB
Loading

0 commit comments

Comments
 (0)