From 316c09350820be91238e10b6a28b54260663e106 Mon Sep 17 00:00:00 2001 From: Saylor Berman Date: Thu, 7 Dec 2023 13:44:17 -0700 Subject: [PATCH 1/2] Add zero-downtime-upgrades results for 1.1.0 Added results and fixed up the instructions to use helm. --- .../results/1.1.0/1.1.0.md | 101 +++ .../results/1.1.0/http.csv | 600 ++++++++++++++++++ .../results/1.1.0/http.png | Bin 0 -> 6318 bytes .../results/1.1.0/https.csv | 600 ++++++++++++++++++ .../results/1.1.0/https.png | Bin 0 -> 6441 bytes tests/zero-downtime-upgrades/values.yaml | 4 - .../zero-downtime-upgrades.md | 23 +- 7 files changed, 1317 insertions(+), 11 deletions(-) create mode 100644 tests/zero-downtime-upgrades/results/1.1.0/1.1.0.md create mode 100644 tests/zero-downtime-upgrades/results/1.1.0/http.csv create mode 100644 tests/zero-downtime-upgrades/results/1.1.0/http.png create mode 100644 tests/zero-downtime-upgrades/results/1.1.0/https.csv create mode 100644 tests/zero-downtime-upgrades/results/1.1.0/https.png diff --git a/tests/zero-downtime-upgrades/results/1.1.0/1.1.0.md b/tests/zero-downtime-upgrades/results/1.1.0/1.1.0.md new file mode 100644 index 0000000000..3d7a99bc10 --- /dev/null +++ b/tests/zero-downtime-upgrades/results/1.1.0/1.1.0.md @@ -0,0 +1,101 @@ +# Results + + +- [Results](#results) + - [Summary](#summary) + - [Versions](#versions) + - [Analyze](#analyze) + - [Tester VMs](#tester-vms) + + +## Summary + +For the most part everything looked good, though a couple requests failed during the https upgrade. +Fewer 499 responses than in past releases, and didn't see any socket errors when using wrk. + +## Versions + +Kubernetes: + +```json +"serverVersion": { + "major": "1", + "minor": "27", + "gitVersion": "v1.27.3-gke.100", + "gitCommit": "6466b51b762a5c49ae3fb6c2c7233ffe1c96e48c", + "gitTreeState": "clean", + "buildDate": "2023-06-23T09:27:28Z", + "goVersion": "go1.20.5 X:boringcrypto", + "compiler": "gc", + "platform": "linux/amd64" + } +``` + +Old NGF version: + +```text +"version":"1.0.0" +"commit":"55b17384cdd4042ffca8dcb8b5c7dec8f6729df9" +"date":"2023-10-24T18:39:05Z" +``` + +New NGF version: + +```text +version: "edge" +commit: "afba7606e2d9e4a5a08e0d0431df4b9e4f3a3437" +date: "2023-12-07T18:47:58Z" +``` + +### Analyze + +- NGF logs: no errors +- NGINX logs: no errors + +Lease was properly updated, and Gateway status updated with new listener. + +#### Tester VMs + +Tester 1 wrk output: + +```text +Running 1m test @ http://cafe.example.com/coffee + 2 threads and 100 connections + Thread Stats Avg Stdev Max +/- Stdev + Latency 35.73ms 39.12ms 753.38ms 89.85% + Req/Sec 1.75k 577.68 3.57k 68.25% + Latency Distribution + 50% 25.13ms + 75% 45.05ms + 90% 75.44ms + 99% 183.71ms + 209294 requests in 1.00m, 74.45MB read +Requests/sec: 3486.00 +Transfer/sec: 1.24MB +``` + +Tester 1 graph: + +![http.png](http.png) + +Tester 2 wrk output: + +```text +Running 1m test @ https://cafe.example.com/tea + 2 threads and 100 connections + Thread Stats Avg Stdev Max +/- Stdev + Latency 34.82ms 37.53ms 753.43ms 89.95% + Req/Sec 1.77k 565.66 4.29k 74.50% + Latency Distribution + 50% 24.07ms + 75% 42.99ms + 90% 72.51ms + 99% 176.96ms + 211751 requests in 1.00m, 73.91MB read +Requests/sec: 3527.10 +Transfer/sec: 1.23MB +``` + +Tester 2 graph: + +![https.png](https.png) diff --git a/tests/zero-downtime-upgrades/results/1.1.0/http.csv b/tests/zero-downtime-upgrades/results/1.1.0/http.csv new file mode 100644 index 0000000000..351bf08c25 --- /dev/null +++ b/tests/zero-downtime-upgrades/results/1.1.0/http.csv @@ -0,0 +1,600 @@ +2023-12-07 20:21:57.893832710+00:00,1 +2023-12-07 20:21:58.052545555+00:00,1 +2023-12-07 20:21:58.182304493+00:00,1 +2023-12-07 20:21:58.309085933+00:00,1 +2023-12-07 20:21:58.470498427+00:00,1 +2023-12-07 20:21:58.626623024+00:00,1 +2023-12-07 20:21:58.767032570+00:00,1 +2023-12-07 20:21:58.932539986+00:00,1 +2023-12-07 20:21:59.112041333+00:00,1 +2023-12-07 20:21:59.250733964+00:00,1 +2023-12-07 20:21:59.403672795+00:00,1 +2023-12-07 20:21:59.565224082+00:00,1 +2023-12-07 20:21:59.723041922+00:00,1 +2023-12-07 20:21:59.915642197+00:00,1 +2023-12-07 20:22:00.059353047+00:00,1 +2023-12-07 20:22:00.198848625+00:00,1 +2023-12-07 20:22:00.352904483+00:00,1 +2023-12-07 20:22:00.498799521+00:00,1 +2023-12-07 20:22:00.651655763+00:00,1 +2023-12-07 20:22:00.795615873+00:00,1 +2023-12-07 20:22:00.947942097+00:00,1 +2023-12-07 20:22:01.094704824+00:00,1 +2023-12-07 20:22:01.267090799+00:00,1 +2023-12-07 20:22:01.474175045+00:00,1 +2023-12-07 20:22:01.630889918+00:00,1 +2023-12-07 20:22:01.776209920+00:00,1 +2023-12-07 20:22:01.963277008+00:00,1 +2023-12-07 20:22:02.109281587+00:00,1 +2023-12-07 20:22:02.256920681+00:00,1 +2023-12-07 20:22:02.434249477+00:00,1 +2023-12-07 20:22:02.595920930+00:00,1 +2023-12-07 20:22:02.747673418+00:00,1 +2023-12-07 20:22:02.986789842+00:00,1 +2023-12-07 20:22:03.128919520+00:00,1 +2023-12-07 20:22:03.267694341+00:00,1 +2023-12-07 20:22:03.412457230+00:00,1 +2023-12-07 20:22:03.580886732+00:00,1 +2023-12-07 20:22:03.719837549+00:00,1 +2023-12-07 20:22:03.851402395+00:00,1 +2023-12-07 20:22:04.167063901+00:00,1 +2023-12-07 20:22:04.330671145+00:00,1 +2023-12-07 20:22:04.505506720+00:00,1 +2023-12-07 20:22:04.664757312+00:00,1 +2023-12-07 20:22:04.806819909+00:00,1 +2023-12-07 20:22:04.955014696+00:00,1 +2023-12-07 20:22:05.107147315+00:00,1 +2023-12-07 20:22:05.256011790+00:00,1 +2023-12-07 20:22:05.407287480+00:00,1 +2023-12-07 20:22:05.557054615+00:00,1 +2023-12-07 20:22:05.700077780+00:00,1 +2023-12-07 20:22:05.850609778+00:00,1 +2023-12-07 20:22:06.052107027+00:00,1 +2023-12-07 20:22:06.218472874+00:00,1 +2023-12-07 20:22:06.385605136+00:00,1 +2023-12-07 20:22:06.560842319+00:00,1 +2023-12-07 20:22:06.727059535+00:00,1 +2023-12-07 20:22:06.871659613+00:00,1 +2023-12-07 20:22:07.042970852+00:00,1 +2023-12-07 20:22:07.231499459+00:00,1 +2023-12-07 20:22:07.373393456+00:00,1 +2023-12-07 20:22:07.506736015+00:00,1 +2023-12-07 20:22:07.657093236+00:00,1 +2023-12-07 20:22:07.801531375+00:00,1 +2023-12-07 20:22:07.944606121+00:00,1 +2023-12-07 20:22:08.084082867+00:00,1 +2023-12-07 20:22:08.230758097+00:00,1 +2023-12-07 20:22:08.370066279+00:00,1 +2023-12-07 20:22:08.541582537+00:00,1 +2023-12-07 20:22:08.697435561+00:00,1 +2023-12-07 20:22:08.839889419+00:00,1 +2023-12-07 20:22:08.981228560+00:00,1 +2023-12-07 20:22:09.209799428+00:00,1 +2023-12-07 20:22:09.353793500+00:00,1 +2023-12-07 20:22:09.569485062+00:00,1 +2023-12-07 20:22:09.715824547+00:00,1 +2023-12-07 20:22:09.859674057+00:00,1 +2023-12-07 20:22:10.004146460+00:00,1 +2023-12-07 20:22:10.147557124+00:00,1 +2023-12-07 20:22:10.284063013+00:00,1 +2023-12-07 20:22:10.429818328+00:00,1 +2023-12-07 20:22:10.571090065+00:00,1 +2023-12-07 20:22:10.718947686+00:00,1 +2023-12-07 20:22:10.885530870+00:00,1 +2023-12-07 20:22:11.034706537+00:00,1 +2023-12-07 20:22:11.188526436+00:00,1 +2023-12-07 20:22:11.418873727+00:00,1 +2023-12-07 20:22:11.616110325+00:00,1 +2023-12-07 20:22:11.768329125+00:00,1 +2023-12-07 20:22:11.949190711+00:00,1 +2023-12-07 20:22:12.098259566+00:00,1 +2023-12-07 20:22:12.234914505+00:00,1 +2023-12-07 20:22:12.387574851+00:00,1 +2023-12-07 20:22:12.531148623+00:00,1 +2023-12-07 20:22:12.667253205+00:00,1 +2023-12-07 20:22:12.841114402+00:00,1 +2023-12-07 20:22:12.989977994+00:00,1 +2023-12-07 20:22:13.145704024+00:00,1 +2023-12-07 20:22:13.279013584+00:00,1 +2023-12-07 20:22:13.449887038+00:00,1 +2023-12-07 20:22:13.597098900+00:00,1 +2023-12-07 20:22:13.781020700+00:00,1 +2023-12-07 20:22:13.993716473+00:00,1 +2023-12-07 20:22:14.175378861+00:00,1 +2023-12-07 20:22:14.318988061+00:00,1 +2023-12-07 20:22:14.437863198+00:00,1 +2023-12-07 20:22:14.601566506+00:00,1 +2023-12-07 20:22:14.718861815+00:00,1 +2023-12-07 20:22:14.863839474+00:00,1 +2023-12-07 20:22:14.984323563+00:00,1 +2023-12-07 20:22:15.131189256+00:00,1 +2023-12-07 20:22:15.278154257+00:00,1 +2023-12-07 20:22:15.399602541+00:00,1 +2023-12-07 20:22:15.570259164+00:00,1 +2023-12-07 20:22:15.739043888+00:00,1 +2023-12-07 20:22:15.882412738+00:00,1 +2023-12-07 20:22:16.053014274+00:00,1 +2023-12-07 20:22:16.172101580+00:00,1 +2023-12-07 20:22:16.316330365+00:00,1 +2023-12-07 20:22:16.454558290+00:00,1 +2023-12-07 20:22:16.595364995+00:00,1 +2023-12-07 20:22:16.755081148+00:00,1 +2023-12-07 20:22:16.872037451+00:00,1 +2023-12-07 20:22:16.988010948+00:00,1 +2023-12-07 20:22:17.107700670+00:00,1 +2023-12-07 20:22:17.266522820+00:00,1 +2023-12-07 20:22:17.383828370+00:00,1 +2023-12-07 20:22:17.561221173+00:00,1 +2023-12-07 20:22:17.728191787+00:00,1 +2023-12-07 20:22:17.881629737+00:00,1 +2023-12-07 20:22:18.001245881+00:00,1 +2023-12-07 20:22:18.155265416+00:00,1 +2023-12-07 20:22:18.304808649+00:00,1 +2023-12-07 20:22:18.424190443+00:00,1 +2023-12-07 20:22:18.545506364+00:00,1 +2023-12-07 20:22:18.748292010+00:00,1 +2023-12-07 20:22:18.900824238+00:00,1 +2023-12-07 20:22:19.019796069+00:00,1 +2023-12-07 20:22:19.237008460+00:00,1 +2023-12-07 20:22:19.410995337+00:00,1 +2023-12-07 20:22:19.530274213+00:00,1 +2023-12-07 20:22:19.726097511+00:00,1 +2023-12-07 20:22:19.845303251+00:00,1 +2023-12-07 20:22:20.080165713+00:00,1 +2023-12-07 20:22:20.227969117+00:00,1 +2023-12-07 20:22:20.346959610+00:00,1 +2023-12-07 20:22:20.467075044+00:00,1 +2023-12-07 20:22:20.613955519+00:00,1 +2023-12-07 20:22:20.778384589+00:00,1 +2023-12-07 20:22:20.947871292+00:00,1 +2023-12-07 20:22:21.100521493+00:00,1 +2023-12-07 20:22:21.235260066+00:00,1 +2023-12-07 20:22:21.387029654+00:00,1 +2023-12-07 20:22:21.593515847+00:00,1 +2023-12-07 20:22:21.791159521+00:00,1 +2023-12-07 20:22:21.943768431+00:00,1 +2023-12-07 20:22:22.090742415+00:00,1 +2023-12-07 20:22:22.210514426+00:00,1 +2023-12-07 20:22:22.355192559+00:00,1 +2023-12-07 20:22:22.474200319+00:00,1 +2023-12-07 20:22:22.622710076+00:00,1 +2023-12-07 20:22:22.780226792+00:00,1 +2023-12-07 20:22:22.930855493+00:00,1 +2023-12-07 20:22:23.076806157+00:00,1 +2023-12-07 20:22:23.217008642+00:00,1 +2023-12-07 20:22:23.438645151+00:00,1 +2023-12-07 20:22:23.582520214+00:00,1 +2023-12-07 20:22:23.701498729+00:00,1 +2023-12-07 20:22:23.840090976+00:00,1 +2023-12-07 20:22:24.030119812+00:00,1 +2023-12-07 20:22:24.148050913+00:00,1 +2023-12-07 20:22:24.290334231+00:00,1 +2023-12-07 20:22:24.457189831+00:00,1 +2023-12-07 20:22:24.601073548+00:00,1 +2023-12-07 20:22:24.768541444+00:00,1 +2023-12-07 20:22:24.885965215+00:00,1 +2023-12-07 20:22:25.030309733+00:00,1 +2023-12-07 20:22:25.165065893+00:00,1 +2023-12-07 20:22:25.310089971+00:00,1 +2023-12-07 20:22:25.427364609+00:00,1 +2023-12-07 20:22:25.564205950+00:00,1 +2023-12-07 20:22:25.712545540+00:00,1 +2023-12-07 20:22:25.872487597+00:00,1 +2023-12-07 20:22:26.032532419+00:00,1 +2023-12-07 20:22:26.175757382+00:00,1 +2023-12-07 20:22:26.291994597+00:00,1 +2023-12-07 20:22:26.493135058+00:00,1 +2023-12-07 20:22:26.631801531+00:00,1 +2023-12-07 20:22:26.906249136+00:00,1 +2023-12-07 20:22:27.039703041+00:00,1 +2023-12-07 20:22:27.199285641+00:00,1 +2023-12-07 20:22:27.346109709+00:00,1 +2023-12-07 20:22:27.493389282+00:00,1 +2023-12-07 20:22:27.666093783+00:00,1 +2023-12-07 20:22:27.824102733+00:00,1 +2023-12-07 20:22:27.983329524+00:00,1 +2023-12-07 20:22:28.103264061+00:00,1 +2023-12-07 20:22:28.240662522+00:00,1 +2023-12-07 20:22:28.420599303+00:00,1 +2023-12-07 20:22:28.536875305+00:00,1 +2023-12-07 20:22:28.754619825+00:00,1 +2023-12-07 20:22:28.898649963+00:00,1 +2023-12-07 20:22:29.015938731+00:00,1 +2023-12-07 20:22:29.167447086+00:00,1 +2023-12-07 20:22:29.311769814+00:00,1 +2023-12-07 20:22:29.457817545+00:00,1 +2023-12-07 20:22:29.579254934+00:00,1 +2023-12-07 20:22:29.699011454+00:00,1 +2023-12-07 20:22:29.863657622+00:00,1 +2023-12-07 20:22:30.004908106+00:00,1 +2023-12-07 20:22:30.146968384+00:00,1 +2023-12-07 20:22:30.322836793+00:00,1 +2023-12-07 20:22:30.456772922+00:00,1 +2023-12-07 20:22:30.579392987+00:00,1 +2023-12-07 20:22:30.698808059+00:00,1 +2023-12-07 20:22:30.823236023+00:00,1 +2023-12-07 20:22:31.027995768+00:00,1 +2023-12-07 20:22:31.153384617+00:00,1 +2023-12-07 20:22:31.301600988+00:00,1 +2023-12-07 20:22:31.443130687+00:00,1 +2023-12-07 20:22:31.564427955+00:00,1 +2023-12-07 20:22:31.691450290+00:00,1 +2023-12-07 20:22:31.817185561+00:00,1 +2023-12-07 20:22:31.956433389+00:00,1 +2023-12-07 20:22:32.082230820+00:00,1 +2023-12-07 20:22:32.208642761+00:00,1 +2023-12-07 20:22:32.330590903+00:00,1 +2023-12-07 20:22:32.454662392+00:00,1 +2023-12-07 20:22:32.623549977+00:00,1 +2023-12-07 20:22:32.744407795+00:00,1 +2023-12-07 20:22:32.866589361+00:00,1 +2023-12-07 20:22:33.043411677+00:00,1 +2023-12-07 20:22:33.165724088+00:00,1 +2023-12-07 20:22:33.296752859+00:00,1 +2023-12-07 20:22:33.423310013+00:00,1 +2023-12-07 20:22:33.558451638+00:00,1 +2023-12-07 20:22:33.790728995+00:00,1 +2023-12-07 20:22:33.917225006+00:00,1 +2023-12-07 20:22:34.055920048+00:00,1 +2023-12-07 20:22:34.180356690+00:00,1 +2023-12-07 20:22:34.317300060+00:00,1 +2023-12-07 20:22:34.465733170+00:00,1 +2023-12-07 20:22:34.612770439+00:00,1 +2023-12-07 20:22:34.799033739+00:00,1 +2023-12-07 20:22:34.988412377+00:00,1 +2023-12-07 20:22:35.130220204+00:00,1 +2023-12-07 20:22:35.263569840+00:00,1 +2023-12-07 20:22:35.393458799+00:00,1 +2023-12-07 20:22:35.543309641+00:00,1 +2023-12-07 20:22:35.722439687+00:00,1 +2023-12-07 20:22:35.846644420+00:00,1 +2023-12-07 20:22:35.998988294+00:00,1 +2023-12-07 20:22:36.148869905+00:00,1 +2023-12-07 20:22:36.286988550+00:00,1 +2023-12-07 20:22:36.417926190+00:00,1 +2023-12-07 20:22:36.561950803+00:00,1 +2023-12-07 20:22:36.724312476+00:00,1 +2023-12-07 20:22:36.855165468+00:00,1 +2023-12-07 20:22:36.981712377+00:00,1 +2023-12-07 20:22:37.126245120+00:00,1 +2023-12-07 20:22:37.308820034+00:00,1 +2023-12-07 20:22:37.429797087+00:00,1 +2023-12-07 20:22:37.551291691+00:00,1 +2023-12-07 20:22:37.680916761+00:00,1 +2023-12-07 20:22:37.821121626+00:00,1 +2023-12-07 20:22:37.965414459+00:00,1 +2023-12-07 20:22:38.135216215+00:00,1 +2023-12-07 20:22:38.259082344+00:00,1 +2023-12-07 20:22:38.448253644+00:00,1 +2023-12-07 20:22:38.578000802+00:00,1 +2023-12-07 20:22:38.752391766+00:00,1 +2023-12-07 20:22:38.883612064+00:00,1 +2023-12-07 20:22:39.035794093+00:00,1 +2023-12-07 20:22:39.202297967+00:00,1 +2023-12-07 20:22:39.383739108+00:00,1 +2023-12-07 20:22:39.518606686+00:00,1 +2023-12-07 20:22:39.668678723+00:00,1 +2023-12-07 20:22:39.809150275+00:00,1 +2023-12-07 20:22:39.936710786+00:00,1 +2023-12-07 20:22:40.058526713+00:00,1 +2023-12-07 20:22:40.188108579+00:00,1 +2023-12-07 20:22:40.354171228+00:00,1 +2023-12-07 20:22:40.475680921+00:00,1 +2023-12-07 20:22:40.602934708+00:00,1 +2023-12-07 20:22:40.734961550+00:00,1 +2023-12-07 20:22:40.912762704+00:00,1 +2023-12-07 20:22:41.038558790+00:00,1 +2023-12-07 20:22:41.161266694+00:00,1 +2023-12-07 20:22:41.299132780+00:00,1 +2023-12-07 20:22:41.444049040+00:00,1 +2023-12-07 20:22:41.596009818+00:00,1 +2023-12-07 20:22:41.829947607+00:00,1 +2023-12-07 20:22:41.999045782+00:00,1 +2023-12-07 20:22:42.169058827+00:00,1 +2023-12-07 20:22:42.301186573+00:00,1 +2023-12-07 20:22:42.457125900+00:00,1 +2023-12-07 20:22:42.595520686+00:00,1 +2023-12-07 20:22:42.780565135+00:00,1 +2023-12-07 20:22:42.916850646+00:00,1 +2023-12-07 20:22:43.063403473+00:00,1 +2023-12-07 20:22:43.208517950+00:00,1 +2023-12-07 20:22:43.490764359+00:00,1 +2023-12-07 20:22:43.614222948+00:00,1 +2023-12-07 20:22:43.836642593+00:00,1 +2023-12-07 20:22:43.963189196+00:00,1 +2023-12-07 20:22:44.103954588+00:00,1 +2023-12-07 20:22:44.227635720+00:00,1 +2023-12-07 20:22:44.355337708+00:00,1 +2023-12-07 20:22:44.508672027+00:00,1 +2023-12-07 20:22:44.637512467+00:00,1 +2023-12-07 20:22:44.776880875+00:00,1 +2023-12-07 20:22:44.969700440+00:00,1 +2023-12-07 20:22:45.104391340+00:00,1 +2023-12-07 20:22:45.289470377+00:00,1 +2023-12-07 20:22:45.416648872+00:00,1 +2023-12-07 20:22:45.549929114+00:00,1 +2023-12-07 20:22:45.691541997+00:00,1 +2023-12-07 20:22:45.830843223+00:00,1 +2023-12-07 20:22:45.962434590+00:00,1 +2023-12-07 20:22:46.108216203+00:00,1 +2023-12-07 20:22:46.280948702+00:00,1 +2023-12-07 20:22:46.412620147+00:00,1 +2023-12-07 20:22:46.542606805+00:00,1 +2023-12-07 20:22:46.666597726+00:00,1 +2023-12-07 20:22:46.795918440+00:00,1 +2023-12-07 20:22:46.941100870+00:00,1 +2023-12-07 20:22:47.076953214+00:00,1 +2023-12-07 20:22:47.202051682+00:00,1 +2023-12-07 20:22:47.328803963+00:00,1 +2023-12-07 20:22:47.466318931+00:00,1 +2023-12-07 20:22:47.597150612+00:00,1 +2023-12-07 20:22:47.722044392+00:00,1 +2023-12-07 20:22:47.847769292+00:00,1 +2023-12-07 20:22:47.979713501+00:00,1 +2023-12-07 20:22:48.139672357+00:00,1 +2023-12-07 20:22:48.277907727+00:00,1 +2023-12-07 20:22:48.405676455+00:00,1 +2023-12-07 20:22:48.530678863+00:00,1 +2023-12-07 20:22:48.660040045+00:00,1 +2023-12-07 20:22:48.788804844+00:00,1 +2023-12-07 20:22:48.952514634+00:00,1 +2023-12-07 20:22:49.094242746+00:00,1 +2023-12-07 20:22:49.227175242+00:00,1 +2023-12-07 20:22:49.360326960+00:00,1 +2023-12-07 20:22:49.497018686+00:00,1 +2023-12-07 20:22:49.631183378+00:00,1 +2023-12-07 20:22:49.778727056+00:00,1 +2023-12-07 20:22:49.924597924+00:00,1 +2023-12-07 20:22:50.076163753+00:00,1 +2023-12-07 20:22:50.219000572+00:00,1 +2023-12-07 20:22:50.368734922+00:00,1 +2023-12-07 20:22:50.519226369+00:00,1 +2023-12-07 20:22:50.655669676+00:00,1 +2023-12-07 20:22:50.789684840+00:00,1 +2023-12-07 20:22:50.925569884+00:00,1 +2023-12-07 20:22:51.221958720+00:00,1 +2023-12-07 20:22:51.381882351+00:00,1 +2023-12-07 20:22:51.508750157+00:00,1 +2023-12-07 20:22:51.711229902+00:00,1 +2023-12-07 20:22:51.843982272+00:00,1 +2023-12-07 20:22:51.975739858+00:00,1 +2023-12-07 20:22:52.167976735+00:00,1 +2023-12-07 20:22:52.299623613+00:00,1 +2023-12-07 20:22:52.432149069+00:00,1 +2023-12-07 20:22:52.564903699+00:00,1 +2023-12-07 20:22:52.694477237+00:00,1 +2023-12-07 20:22:52.881022217+00:00,1 +2023-12-07 20:22:53.036691442+00:00,1 +2023-12-07 20:22:53.185148266+00:00,1 +2023-12-07 20:22:53.340655694+00:00,1 +2023-12-07 20:22:53.474363273+00:00,1 +2023-12-07 20:22:53.618049728+00:00,1 +2023-12-07 20:22:53.758918637+00:00,1 +2023-12-07 20:22:53.954951921+00:00,1 +2023-12-07 20:22:54.087287773+00:00,1 +2023-12-07 20:22:54.219902444+00:00,1 +2023-12-07 20:22:54.375433843+00:00,1 +2023-12-07 20:22:54.607705071+00:00,1 +2023-12-07 20:22:54.760365021+00:00,1 +2023-12-07 20:22:54.910393493+00:00,1 +2023-12-07 20:22:55.059528754+00:00,1 +2023-12-07 20:22:55.214461948+00:00,1 +2023-12-07 20:22:55.367805479+00:00,1 +2023-12-07 20:22:55.503988619+00:00,1 +2023-12-07 20:22:55.640504657+00:00,1 +2023-12-07 20:22:55.950958654+00:00,1 +2023-12-07 20:22:56.088993196+00:00,1 +2023-12-07 20:22:56.230085294+00:00,1 +2023-12-07 20:22:56.390491889+00:00,1 +2023-12-07 20:22:56.552187655+00:00,1 +2023-12-07 20:22:56.693811548+00:00,1 +2023-12-07 20:22:56.883621854+00:00,1 +2023-12-07 20:22:57.022487816+00:00,1 +2023-12-07 20:22:57.165679274+00:00,1 +2023-12-07 20:22:57.297857612+00:00,1 +2023-12-07 20:22:57.438098620+00:00,1 +2023-12-07 20:22:57.579057890+00:00,1 +2023-12-07 20:22:57.696627452+00:00,1 +2023-12-07 20:22:57.817872383+00:00,1 +2023-12-07 20:22:57.950289618+00:00,1 +2023-12-07 20:22:58.208905688+00:00,1 +2023-12-07 20:22:58.332188669+00:00,1 +2023-12-07 20:22:58.464628941+00:00,1 +2023-12-07 20:22:58.587206540+00:00,1 +2023-12-07 20:22:58.703863479+00:00,1 +2023-12-07 20:22:58.819068531+00:00,1 +2023-12-07 20:22:58.936372364+00:00,1 +2023-12-07 20:22:59.053245985+00:00,1 +2023-12-07 20:22:59.170604395+00:00,1 +2023-12-07 20:22:59.288186923+00:00,1 +2023-12-07 20:22:59.405205091+00:00,1 +2023-12-07 20:22:59.522260002+00:00,1 +2023-12-07 20:22:59.639584347+00:00,1 +2023-12-07 20:22:59.757113142+00:00,1 +2023-12-07 20:22:59.873174603+00:00,1 +2023-12-07 20:22:59.991787170+00:00,1 +2023-12-07 20:23:00.107146362+00:00,1 +2023-12-07 20:23:00.222884118+00:00,1 +2023-12-07 20:23:00.338036493+00:00,1 +2023-12-07 20:23:00.453315343+00:00,1 +2023-12-07 20:23:00.568870186+00:00,1 +2023-12-07 20:23:00.684212060+00:00,1 +2023-12-07 20:23:00.799735021+00:00,1 +2023-12-07 20:23:00.916403289+00:00,1 +2023-12-07 20:23:01.043372795+00:00,1 +2023-12-07 20:23:01.161268520+00:00,1 +2023-12-07 20:23:01.278872278+00:00,1 +2023-12-07 20:23:01.395574794+00:00,1 +2023-12-07 20:23:01.512487155+00:00,1 +2023-12-07 20:23:01.629349806+00:00,1 +2023-12-07 20:23:01.746382301+00:00,1 +2023-12-07 20:23:01.863374600+00:00,1 +2023-12-07 20:23:01.979232357+00:00,1 +2023-12-07 20:23:02.094357654+00:00,1 +2023-12-07 20:23:02.209163495+00:00,1 +2023-12-07 20:23:02.324126619+00:00,1 +2023-12-07 20:23:02.438960870+00:00,1 +2023-12-07 20:23:02.553599845+00:00,1 +2023-12-07 20:23:02.668330868+00:00,1 +2023-12-07 20:23:02.783046611+00:00,1 +2023-12-07 20:23:02.898410913+00:00,1 +2023-12-07 20:23:03.013279066+00:00,1 +2023-12-07 20:23:03.128807474+00:00,1 +2023-12-07 20:23:03.243953051+00:00,1 +2023-12-07 20:23:03.365033694+00:00,1 +2023-12-07 20:23:03.480142315+00:00,1 +2023-12-07 20:23:03.595664775+00:00,1 +2023-12-07 20:23:03.710969400+00:00,1 +2023-12-07 20:23:03.826292079+00:00,1 +2023-12-07 20:23:03.942016999+00:00,1 +2023-12-07 20:23:04.059050312+00:00,1 +2023-12-07 20:23:04.177045591+00:00,1 +2023-12-07 20:23:04.294173022+00:00,1 +2023-12-07 20:23:04.412078629+00:00,1 +2023-12-07 20:23:04.529687131+00:00,1 +2023-12-07 20:23:04.646686838+00:00,1 +2023-12-07 20:23:04.764094292+00:00,1 +2023-12-07 20:23:04.881578535+00:00,1 +2023-12-07 20:23:04.998905532+00:00,1 +2023-12-07 20:23:05.116409490+00:00,1 +2023-12-07 20:23:05.233787344+00:00,1 +2023-12-07 20:23:05.350857783+00:00,1 +2023-12-07 20:23:05.468448336+00:00,1 +2023-12-07 20:23:05.585489871+00:00,1 +2023-12-07 20:23:05.702973139+00:00,1 +2023-12-07 20:23:05.820756132+00:00,1 +2023-12-07 20:23:05.938497624+00:00,1 +2023-12-07 20:23:06.056413924+00:00,1 +2023-12-07 20:23:06.173928605+00:00,1 +2023-12-07 20:23:06.291241201+00:00,1 +2023-12-07 20:23:06.408268628+00:00,1 +2023-12-07 20:23:06.525204270+00:00,1 +2023-12-07 20:23:06.642484924+00:00,1 +2023-12-07 20:23:06.760431649+00:00,1 +2023-12-07 20:23:06.877642241+00:00,1 +2023-12-07 20:23:06.994414415+00:00,1 +2023-12-07 20:23:07.113200296+00:00,1 +2023-12-07 20:23:07.231116950+00:00,1 +2023-12-07 20:23:07.348826779+00:00,1 +2023-12-07 20:23:07.465735989+00:00,1 +2023-12-07 20:23:07.582672391+00:00,1 +2023-12-07 20:23:07.700657195+00:00,1 +2023-12-07 20:23:07.817800240+00:00,1 +2023-12-07 20:23:07.935003342+00:00,1 +2023-12-07 20:23:08.052575136+00:00,1 +2023-12-07 20:23:08.169997081+00:00,1 +2023-12-07 20:23:08.286998110+00:00,1 +2023-12-07 20:23:08.403841904+00:00,1 +2023-12-07 20:23:08.520458299+00:00,1 +2023-12-07 20:23:08.639707450+00:00,1 +2023-12-07 20:23:08.756605704+00:00,1 +2023-12-07 20:23:08.874623686+00:00,1 +2023-12-07 20:23:08.992475557+00:00,1 +2023-12-07 20:23:09.109490275+00:00,1 +2023-12-07 20:23:09.226499179+00:00,1 +2023-12-07 20:23:09.343694826+00:00,1 +2023-12-07 20:23:09.460725861+00:00,1 +2023-12-07 20:23:09.577844775+00:00,1 +2023-12-07 20:23:09.694757385+00:00,1 +2023-12-07 20:23:09.811853348+00:00,1 +2023-12-07 20:23:09.929653013+00:00,1 +2023-12-07 20:23:10.047022661+00:00,1 +2023-12-07 20:23:10.163308653+00:00,1 +2023-12-07 20:23:10.278282744+00:00,1 +2023-12-07 20:23:10.393476548+00:00,1 +2023-12-07 20:23:10.509976219+00:00,1 +2023-12-07 20:23:10.625140364+00:00,1 +2023-12-07 20:23:10.740367844+00:00,1 +2023-12-07 20:23:10.855885437+00:00,1 +2023-12-07 20:23:10.970966889+00:00,1 +2023-12-07 20:23:11.086653361+00:00,1 +2023-12-07 20:23:11.203974990+00:00,1 +2023-12-07 20:23:11.320599222+00:00,1 +2023-12-07 20:23:11.437476384+00:00,1 +2023-12-07 20:23:11.554158276+00:00,1 +2023-12-07 20:23:11.671266166+00:00,1 +2023-12-07 20:23:11.790041385+00:00,1 +2023-12-07 20:23:11.906944751+00:00,1 +2023-12-07 20:23:12.023740893+00:00,1 +2023-12-07 20:23:12.141211962+00:00,1 +2023-12-07 20:23:12.258251628+00:00,1 +2023-12-07 20:23:12.376009275+00:00,1 +2023-12-07 20:23:12.492627258+00:00,1 +2023-12-07 20:23:12.609828083+00:00,1 +2023-12-07 20:23:12.727619272+00:00,1 +2023-12-07 20:23:12.844861368+00:00,1 +2023-12-07 20:23:12.962508150+00:00,1 +2023-12-07 20:23:13.079444631+00:00,1 +2023-12-07 20:23:13.195707646+00:00,1 +2023-12-07 20:23:13.310788648+00:00,1 +2023-12-07 20:23:13.426061942+00:00,1 +2023-12-07 20:23:13.541422726+00:00,1 +2023-12-07 20:23:13.656734545+00:00,1 +2023-12-07 20:23:13.772053437+00:00,1 +2023-12-07 20:23:13.887519125+00:00,1 +2023-12-07 20:23:14.005919990+00:00,1 +2023-12-07 20:23:14.120679105+00:00,1 +2023-12-07 20:23:14.235541912+00:00,1 +2023-12-07 20:23:14.351444370+00:00,1 +2023-12-07 20:23:14.466662160+00:00,1 +2023-12-07 20:23:14.582324794+00:00,1 +2023-12-07 20:23:14.697702110+00:00,1 +2023-12-07 20:23:14.813533336+00:00,1 +2023-12-07 20:23:14.928933261+00:00,1 +2023-12-07 20:23:15.045303036+00:00,1 +2023-12-07 20:23:15.160818573+00:00,1 +2023-12-07 20:23:15.275906354+00:00,1 +2023-12-07 20:23:15.390841867+00:00,1 +2023-12-07 20:23:15.506089020+00:00,1 +2023-12-07 20:23:15.621052840+00:00,1 +2023-12-07 20:23:15.736232713+00:00,1 +2023-12-07 20:23:15.858317415+00:00,1 +2023-12-07 20:23:15.973760066+00:00,1 +2023-12-07 20:23:16.089026455+00:00,1 +2023-12-07 20:23:16.204705835+00:00,1 +2023-12-07 20:23:16.319618522+00:00,1 +2023-12-07 20:23:16.434259654+00:00,1 +2023-12-07 20:23:16.548860239+00:00,1 +2023-12-07 20:23:16.663787732+00:00,1 +2023-12-07 20:23:16.779029523+00:00,1 +2023-12-07 20:23:16.894582164+00:00,1 +2023-12-07 20:23:17.009571641+00:00,1 +2023-12-07 20:23:17.124345465+00:00,1 +2023-12-07 20:23:17.239290053+00:00,1 +2023-12-07 20:23:17.354769959+00:00,1 +2023-12-07 20:23:17.469822737+00:00,1 +2023-12-07 20:23:17.584386725+00:00,1 +2023-12-07 20:23:17.699278860+00:00,1 +2023-12-07 20:23:17.814048981+00:00,1 +2023-12-07 20:23:17.929134834+00:00,1 +2023-12-07 20:23:18.044567246+00:00,1 +2023-12-07 20:23:18.159978040+00:00,1 +2023-12-07 20:23:18.275693507+00:00,1 +2023-12-07 20:23:18.392683346+00:00,1 +2023-12-07 20:23:18.509417163+00:00,1 +2023-12-07 20:23:18.626265032+00:00,1 +2023-12-07 20:23:18.743001867+00:00,1 +2023-12-07 20:23:18.859760495+00:00,1 +2023-12-07 20:23:18.976399934+00:00,1 +2023-12-07 20:23:19.093556246+00:00,1 +2023-12-07 20:23:19.210522705+00:00,1 +2023-12-07 20:23:19.327231706+00:00,1 +2023-12-07 20:23:19.442477210+00:00,1 +2023-12-07 20:23:19.557665768+00:00,1 +2023-12-07 20:23:19.672657139+00:00,1 +2023-12-07 20:23:19.787814061+00:00,1 +2023-12-07 20:23:19.903575149+00:00,1 +2023-12-07 20:23:20.019355716+00:00,1 +2023-12-07 20:23:20.134602832+00:00,1 +2023-12-07 20:23:20.250045224+00:00,1 +2023-12-07 20:23:20.365564852+00:00,1 +2023-12-07 20:23:20.480575850+00:00,1 +2023-12-07 20:23:20.595532831+00:00,1 +2023-12-07 20:23:20.710716829+00:00,1 +2023-12-07 20:23:20.826357629+00:00,1 +2023-12-07 20:23:20.941827553+00:00,1 +2023-12-07 20:23:21.058468044+00:00,1 +2023-12-07 20:23:21.174384894+00:00,1 +2023-12-07 20:23:21.289618815+00:00,1 +2023-12-07 20:23:21.405023374+00:00,1 +2023-12-07 20:23:21.520278557+00:00,1 diff --git a/tests/zero-downtime-upgrades/results/1.1.0/http.png b/tests/zero-downtime-upgrades/results/1.1.0/http.png new file mode 100644 index 0000000000000000000000000000000000000000..b478a11bc108c1ccb7f5abdda3c80ffb482209fe GIT binary patch literal 6318 zcmeHLdpMNq+P@ncNe)F6$)uB#LsU~?Ob5}*q0&U5sZe1tlAOYfQYK+^K!mYWvOQ3LgjVkuuA8pzsRmFXME7jA|(lRqMv$wZDd-g1eL`q9b zd-m*ERaMoSH*b1-d!;VMOihzUk_tcdjp4_>BFISR6{NJ9F*1UD1gSUhl-@~tH}#Be zI{!)ET(KfR8VW#DZ1W(@hloJ)J4F2sLOL`?CL`7$CI|MXgpc*bS2BHHTvrr{zGRcp z$RilJGIAH%Io@Y4V`-MWet2@2kqZ-gjzwVq6b?RzUrNB1 z2{tdlx(xPKz@AFLvB9beENj6073`{m9SwkKgl(_E>kT=yd{`2dlb4=No8$2d!RM&IgTtST+D^L$GuhR7XKs07~PaI01`9uy6|G zra@K=GP8h^%wx?Gqx4*gV1)qN;}-#xU(H_&k`0u#0hS#-uy6O#(2U{E!N8mWE$h=` z>$9`tzRi(D>x`70s%7@y3fdo@ke7dXNixRFl>PO$sZ#y52X6nD;Z1Ye`S>%HA9M+S z%pM8L&h+3s%QvJd|ArJhwNpvM$5#)oFZJr6`i)?S#hjjE*L-mjag$-L?2-4Ee<|@f zle+8}-j(^u%VQ`dEZSB+UDMWVuJfbU@sPG_+@ixOmwaskq zLU|w4W-30!TH(@35-KLG?6V`E=5*X5Bg(w=_grE-gBam7EkeDVlj9h*y5>3;=jt_6 zr#c2I@a~35y`S@UiQ1lUQ2jKcB#r?H>s>xf$Mt`Imn^B3rdB>Z8@OzuZ)Y0~hXMw@YX<5u z3G8r6$($Trn!)#-wk2X~HumaKej#1+O8MDvxuS_eR`~6xcJGJbdUbi_b@0)`FMUzb7Wbks!8;=D%nSTy z$5Ddz*x(ZedzGQmJCv<$oMa&E-)oNlZ=x;Ut<)v)^@H<>12&Zs=AuBh~)`;Z_8{Q7}94)FooY zwE&rY6jF~IMOA)wIZ3HIPr_b_uTfH6?S_`;!IE#h7Yp=WS(V_T>i!{>^FZmvD1QH{ zKv6~N|FLhOW@ELQ>gu_KdHOx-R-&BGI{i58#kUQ$j)-%I)hu#D=`YTkhFV&^l*_+L@1#MupdU)-d~f>;kI?Jf7dD zw^+7Mo||Z74DY z@wVK?K^>Pfbhhe6Evea}t3u$`dOUXuo8I?Bu=*G&U86Cg+Uso-pG_3vH!qnlhvY{_ z#GJf=RN!ocwr9Ah86jXrkZ^jmBSLrIc`Kxw#kyluL?fVysFJ1X*1S5|r@H!a>e=h!jPj}VeLO`r53 zt~|j@C{CFo@J+(gDij$rD zWKDf?3&Us2iPvM@JAEE7I9%C0QN8@(C^7aU<0+k|Yn+rj)a;rn+nQ`o2x@xzP6Zk6 z5TsnPJo%QfebQ#S-`tJ;CM$|mfxCWjFE>McrV=->$vF$tha@^D^Y82ro5~*Q)VGQZ z8T{D(Zd4{~MTnnM;)YnS=)1#V;pKBg^UZhi(>2 z_&Gs~BeHW!(4k?wEo*I={JA9WZJ~1Sh+_$go+o6QxTxdJ%dztB7-N+x#o{&*Y6YQg zlekob-z-@wjaDpe3%~nBFHXgKaTA?qu#_m7`F8Xn%(sJ!e!l*!!~Z7;tf+hLkrxVx zcDR^;=#>@=q#H;n-HqBAiqd<<-Xf(jhN|=1%ee$?^q!St9<$qGXp>0bQ3cGO8WI2A zjk0~BZ7r-SH~;7OMdCQ&Cp~Ym@(;E@KACDz)?dl-VI6i#zC>aRn_{>H9i71rF4p0@ zPG5c{=47sRxOAM`xj{|NzCFU^bVIdE1FLjIFgZ|7&wGX$l96qfoo(m*UoH~>ma)z)k)?8nW*Jx}u z&vBa!y->4jvdN#(m5f<)603DzEqG=8eFbCP_}b?^1oyCXwtgy+xodr%_p@Kw=>gjY zeVCbQ$8KdgN-c^5Nt6rBzB>w$m(n#Gn-$EJ>o-YF8S|v3;+(1VeK9)nJ=nd1*6JO{ zXvAm5xOAWCnVSlOg-ox<5p5S)zWu0gz{r55~eJ#RUV?uQk_u`lc6s_T$H_Yc)_S-Jikuq?^v=ADMfFsc;kKI%?O0Af#~d|G zid_>Il34#C#!|lLYz;@gN55%2esZMTo?mS2c$O1$R`g~f^O?9ufo@Bn9m3dTu)pZ$ z{P-jBkGv*L-7`9A-GPJODz6rA$qDT_|Et5N3nu>S)frt&Y`jy~YAb%H=j|Q#xmFmJ zkIB;Xw{x);k8f=6qi|AZ_NwU+ERO;7FKCSh2Y{l$XXXqw0&3`(- z>)-0GUbq`9T2x)Xw}5d{v%;l<=EE)IRL=i1yN$MoiTiI`I!OR#n7dbZ+zog`NB{YnawH5X*{+9_LQw zmToBcFeGGN+ip`pVnwd5H_u@ZLWu+I)C*x7Nu>Gcmd;dO6?p!QacUy mIcfPfuz5@V*J&v4s}*YxkUwuKUm!hng#*?O`=0DMb>&}t5SDTP literal 0 HcmV?d00001 diff --git a/tests/zero-downtime-upgrades/results/1.1.0/https.csv b/tests/zero-downtime-upgrades/results/1.1.0/https.csv new file mode 100644 index 0000000000..ab1cf07e16 --- /dev/null +++ b/tests/zero-downtime-upgrades/results/1.1.0/https.csv @@ -0,0 +1,600 @@ +2023-12-07 20:21:59.354175038+00:00,1 +2023-12-07 20:21:59.551922574+00:00,1 +2023-12-07 20:21:59.724191763+00:00,1 +2023-12-07 20:21:59.892045575+00:00,1 +2023-12-07 20:22:00.083033659+00:00,1 +2023-12-07 20:22:00.264313068+00:00,1 +2023-12-07 20:22:00.447565060+00:00,1 +2023-12-07 20:22:00.670633965+00:00,1 +2023-12-07 20:22:00.826658593+00:00,1 +2023-12-07 20:22:00.980450402+00:00,1 +2023-12-07 20:22:01.144548136+00:00,1 +2023-12-07 20:22:01.379107056+00:00,1 +2023-12-07 20:22:01.554332912+00:00,1 +2023-12-07 20:22:01.781210353+00:00,1 +2023-12-07 20:22:01.948361526+00:00,1 +2023-12-07 20:22:02.106853038+00:00,1 +2023-12-07 20:22:02.274065682+00:00,1 +2023-12-07 20:22:02.456575953+00:00,1 +2023-12-07 20:22:02.604604420+00:00,1 +2023-12-07 20:22:02.772524983+00:00,1 +2023-12-07 20:22:02.941800579+00:00,1 +2023-12-07 20:22:03.112871753+00:00,1 +2023-12-07 20:22:03.298249509+00:00,1 +2023-12-07 20:22:03.445603738+00:00,1 +2023-12-07 20:22:03.632837669+00:00,1 +2023-12-07 20:22:03.809688187+00:00,1 +2023-12-07 20:22:04.125202596+00:00,1 +2023-12-07 20:22:04.343030245+00:00,1 +2023-12-07 20:22:04.531946664+00:00,1 +2023-12-07 20:22:04.701770240+00:00,1 +2023-12-07 20:22:04.866044425+00:00,1 +2023-12-07 20:22:05.090012910+00:00,1 +2023-12-07 20:22:05.262670816+00:00,1 +2023-12-07 20:22:05.450535341+00:00,1 +2023-12-07 20:22:05.617645368+00:00,1 +2023-12-07 20:22:05.806248917+00:00,1 +2023-12-07 20:22:05.979881878+00:00,1 +2023-12-07 20:22:06.158089402+00:00,1 +2023-12-07 20:22:06.324411483+00:00,1 +2023-12-07 20:22:06.608008125+00:00,1 +2023-12-07 20:22:06.780107290+00:00,1 +2023-12-07 20:22:06.945812006+00:00,1 +2023-12-07 20:22:07.122662483+00:00,1 +2023-12-07 20:22:07.277497626+00:00,1 +2023-12-07 20:22:07.433562791+00:00,1 +2023-12-07 20:22:07.616918273+00:00,1 +2023-12-07 20:22:07.991015012+00:00,1 +2023-12-07 20:22:08.168149345+00:00,1 +2023-12-07 20:22:08.315591720+00:00,1 +2023-12-07 20:22:08.579322108+00:00,1 +2023-12-07 20:22:08.755779269+00:00,1 +2023-12-07 20:22:08.920878875+00:00,1 +2023-12-07 20:22:09.097876094+00:00,1 +2023-12-07 20:22:09.278908677+00:00,1 +2023-12-07 20:22:09.503255225+00:00,1 +2023-12-07 20:22:09.698669394+00:00,1 +2023-12-07 20:22:09.898912685+00:00,1 +2023-12-07 20:22:10.120350206+00:00,1 +2023-12-07 20:22:10.286177036+00:00,1 +2023-12-07 20:22:10.444342802+00:00,1 +2023-12-07 20:22:10.607420907+00:00,1 +2023-12-07 20:22:10.783226397+00:00,1 +2023-12-07 20:22:10.964245380+00:00,1 +2023-12-07 20:22:11.128004514+00:00,1 +2023-12-07 20:22:11.336222042+00:00,1 +2023-12-07 20:22:11.509307829+00:00,1 +2023-12-07 20:22:11.703752528+00:00,1 +2023-12-07 20:22:11.898445533+00:00,1 +2023-12-07 20:22:12.059862397+00:00,1 +2023-12-07 20:22:12.223477443+00:00,1 +2023-12-07 20:22:12.392503048+00:00,1 +2023-12-07 20:22:12.561577048+00:00,1 +2023-12-07 20:22:12.726039479+00:00,1 +2023-12-07 20:22:12.888176497+00:00,1 +2023-12-07 20:22:13.044503939+00:00,1 +2023-12-07 20:22:13.209636159+00:00,1 +2023-12-07 20:22:13.368463345+00:00,1 +2023-12-07 20:22:13.553639689+00:00,1 +2023-12-07 20:22:13.765621022+00:00,1 +2023-12-07 20:22:13.987975673+00:00,1 +2023-12-07 20:22:14.158301395+00:00,1 +2023-12-07 20:22:14.317160456+00:00,1 +2023-12-07 20:22:14.500364120+00:00,1 +2023-12-07 20:22:14.671539045+00:00,1 +2023-12-07 20:22:14.876769055+00:00,1 +2023-12-07 20:22:15.099143782+00:00,1 +2023-12-07 20:22:15.234571948+00:00,1 +2023-12-07 20:22:15.370544664+00:00,1 +2023-12-07 20:22:15.531541570+00:00,1 +2023-12-07 20:22:15.665653321+00:00,1 +2023-12-07 20:22:15.843709632+00:00,1 +2023-12-07 20:22:15.979806006+00:00,1 +2023-12-07 20:22:16.145063383+00:00,1 +2023-12-07 20:22:16.278441109+00:00,1 +2023-12-07 20:22:16.411671206+00:00,1 +2023-12-07 20:22:16.569230821+00:00,1 +2023-12-07 20:22:16.700452885+00:00,1 +2023-12-07 20:22:16.833247261+00:00,1 +2023-12-07 20:22:16.965677670+00:00,1 +2023-12-07 20:22:17.140179115+00:00,1 +2023-12-07 20:22:17.274569899+00:00,1 +2023-12-07 20:22:17.583016858+00:00,1 +2023-12-07 20:22:17.722444203+00:00,1 +2023-12-07 20:22:17.890813886+00:00,1 +2023-12-07 20:22:18.084959293+00:00,1 +2023-12-07 20:22:18.258200588+00:00,1 +2023-12-07 20:22:18.392133104+00:00,1 +2023-12-07 20:22:18.554064862+00:00,1 +2023-12-07 20:22:18.754285164+00:00,1 +2023-12-07 20:22:18.920591958+00:00,1 +2023-12-07 20:22:19.151222965+00:00,1 +2023-12-07 20:22:19.335550371+00:00,1 +2023-12-07 20:22:19.502930515+00:00,1 +2023-12-07 20:22:19.679265901+00:00,1 +2023-12-07 20:22:19.840913346+00:00,1 +2023-12-07 20:22:19.974531297+00:00,1 +2023-12-07 20:22:20.131727871+00:00,1 +2023-12-07 20:22:20.265068216+00:00,1 +2023-12-07 20:22:20.454877200+00:00,1 +2023-12-07 20:22:20.587259566+00:00,1 +2023-12-07 20:22:20.764658694+00:00,1 +2023-12-07 20:22:20.924233064+00:00,1 +2023-12-07 20:22:21.088692679+00:00,1 +2023-12-07 20:22:21.238059926+00:00,1 +2023-12-07 20:22:21.398687227+00:00,1 +2023-12-07 20:22:21.570494570+00:00,1 +2023-12-07 20:22:21.702476131+00:00,1 +2023-12-07 20:22:21.909596898+00:00,1 +2023-12-07 20:22:22.079863364+00:00,1 +2023-12-07 20:22:22.259950071+00:00,1 +2023-12-07 20:22:22.430235308+00:00,1 +2023-12-07 20:22:22.561980396+00:00,1 +2023-12-07 20:22:22.722629806+00:00,1 +2023-12-07 20:22:22.878375596+00:00,1 +2023-12-07 20:22:23.066680242+00:00,1 +2023-12-07 20:22:23.216666577+00:00,1 +2023-12-07 20:22:23.347854952+00:00,1 +2023-12-07 20:22:23.543268667+00:00,1 +2023-12-07 20:22:23.677279550+00:00,1 +2023-12-07 20:22:23.854673912+00:00,1 +2023-12-07 20:22:23.989205028+00:00,1 +2023-12-07 20:22:24.166666958+00:00,1 +2023-12-07 20:22:24.324778322+00:00,1 +2023-12-07 20:22:24.526859285+00:00,1 +2023-12-07 20:22:24.702600424+00:00,1 +2023-12-07 20:22:24.906467482+00:00,1 +2023-12-07 20:22:25.090804440+00:00,1 +2023-12-07 20:22:25.263978943+00:00,1 +2023-12-07 20:22:25.430383454+00:00,1 +2023-12-07 20:22:25.562352808+00:00,1 +2023-12-07 20:22:25.696152159+00:00,1 +2023-12-07 20:22:25.875841912+00:00,1 +2023-12-07 20:22:26.063187463+00:00,1 +2023-12-07 20:22:26.268414285+00:00,1 +2023-12-07 20:22:26.507227547+00:00,1 +2023-12-07 20:22:26.662216269+00:00,1 +2023-12-07 20:22:26.829524403+00:00,1 +2023-12-07 20:22:27.050601376+00:00,1 +2023-12-07 20:22:27.184565206+00:00,1 +2023-12-07 20:22:27.337243075+00:00,1 +2023-12-07 20:22:27.495179100+00:00,1 +2023-12-07 20:22:27.626553424+00:00,1 +2023-12-07 20:22:27.801079107+00:00,1 +2023-12-07 20:22:27.984875788+00:00,1 +2023-12-07 20:22:28.152446181+00:00,1 +2023-12-07 20:22:28.542406624+00:00,1 +2023-12-07 20:22:28.778556221+00:00,1 +2023-12-07 20:22:28.956910175+00:00,1 +2023-12-07 20:22:29.091274246+00:00,1 +2023-12-07 20:22:29.296482633+00:00,1 +2023-12-07 20:22:29.455163002+00:00,1 +2023-12-07 20:22:29.638735905+00:00,1 +2023-12-07 20:22:29.832100007+00:00,1 +2023-12-07 20:22:30.019002933+00:00,1 +2023-12-07 20:22:30.155748376+00:00,1 +2023-12-07 20:22:30.330834744+00:00,1 +2023-12-07 20:22:30.469633561+00:00,1 +2023-12-07 20:22:30.630854978+00:00,1 +2023-12-07 20:22:30.766006300+00:00,1 +2023-12-07 20:22:30.944824262+00:00,1 +2023-12-07 20:22:31.113039471+00:00,1 +2023-12-07 20:22:31.317405967+00:00,1 +2023-12-07 20:22:31.487161110+00:00,1 +2023-12-07 20:22:31.650824218+00:00,1 +2023-12-07 20:22:31.798355605+00:00,1 +2023-12-07 20:22:31.945491644+00:00,1 +2023-12-07 20:22:32.093847774+00:00,1 +2023-12-07 20:22:32.240997590+00:00,1 +2023-12-07 20:22:32.427859727+00:00,1 +2023-12-07 20:22:32.591884900+00:00,1 +2023-12-07 20:22:32.733811215+00:00,1 +2023-12-07 20:22:32.893131836+00:00,1 +2023-12-07 20:22:33.039614904+00:00,1 +2023-12-07 20:22:33.202854217+00:00,1 +2023-12-07 20:22:33.367461674+00:00,1 +2023-12-07 20:22:33.517743656+00:00,1 +2023-12-07 20:22:33.655478182+00:00,1 +2023-12-07 20:22:33.800223921+00:00,1 +2023-12-07 20:22:33.956246056+00:00,1 +2023-12-07 20:22:34.097729479+00:00,1 +2023-12-07 20:22:34.260562539+00:00,1 +2023-12-07 20:22:34.429105064+00:00,1 +2023-12-07 20:22:34.617889857+00:00,1 +2023-12-07 20:22:34.830440471+00:00,1 +2023-12-07 20:22:35.006469433+00:00,1 +2023-12-07 20:22:35.185818989+00:00,1 +2023-12-07 20:22:35.336225926+00:00,1 +2023-12-07 20:22:35.508255214+00:00,1 +2023-12-07 20:22:35.693577705+00:00,1 +2023-12-07 20:22:35.860940938+00:00,1 +2023-12-07 20:22:36.022597280+00:00,1 +2023-12-07 20:22:36.169233156+00:00,1 +2023-12-07 20:22:36.341843986+00:00,1 +2023-12-07 20:22:36.490282535+00:00,1 +2023-12-07 20:22:36.662292751+00:00,1 +2023-12-07 20:22:36.895556619+00:00,1 +2023-12-07 20:22:37.185675228+00:00,1 +2023-12-07 20:22:37.339390993+00:00,1 +2023-12-07 20:22:37.499612496+00:00,1 +2023-12-07 20:22:37.653846143+00:00,1 +2023-12-07 20:22:37.823048304+00:00,1 +2023-12-07 20:22:37.984299742+00:00,1 +2023-12-07 20:22:38.130851012+00:00,1 +2023-12-07 20:22:38.445394978+00:00,1 +2023-12-07 20:22:38.622232848+00:00,1 +2023-12-07 20:22:38.851266255+00:00,1 +2023-12-07 20:22:39.015893667+00:00,1 +2023-12-07 20:22:39.171177460+00:00,1 +2023-12-07 20:22:39.354532512+00:00,1 +2023-12-07 20:22:39.551021863+00:00,1 +2023-12-07 20:22:39.708071296+00:00,1 +2023-12-07 20:22:39.854058321+00:00,1 +2023-12-07 20:22:39.994360927+00:00,1 +2023-12-07 20:22:40.151016855+00:00,1 +2023-12-07 20:22:40.367501877+00:00,1 +2023-12-07 20:22:40.517857299+00:00,1 +2023-12-07 20:22:40.676139784+00:00,1 +2023-12-07 20:22:40.880805276+00:00,1 +2023-12-07 20:22:41.047056177+00:00,1 +2023-12-07 20:22:41.212387489+00:00,1 +2023-12-07 20:22:41.351527399+00:00,1 +2023-12-07 20:22:41.491006678+00:00,1 +2023-12-07 20:22:41.745458415+00:00,1 +2023-12-07 20:22:41.888671101+00:00,1 +2023-12-07 20:22:42.091738261+00:00,1 +2023-12-07 20:22:42.237045673+00:00,1 +2023-12-07 20:22:42.394232191+00:00,1 +2023-12-07 20:22:42.534181732+00:00,1 +2023-12-07 20:22:42.715636226+00:00,1 +2023-12-07 20:22:42.857388050+00:00,1 +2023-12-07 20:22:43.007813527+00:00,1 +2023-12-07 20:22:43.178915303+00:00,1 +2023-12-07 20:22:43.317354957+00:00,1 +2023-12-07 20:22:43.471715103+00:00,1 +2023-12-07 20:22:43.646087727+00:00,1 +2023-12-07 20:22:43.826163584+00:00,1 +2023-12-07 20:22:43.976269286+00:00,1 +2023-12-07 20:22:44.131747213+00:00,1 +2023-12-07 20:22:44.269936337+00:00,1 +2023-12-07 20:22:44.445014012+00:00,1 +2023-12-07 20:22:44.591684719+00:00,1 +2023-12-07 20:22:44.728812248+00:00,1 +2023-12-07 20:22:44.882681565+00:00,1 +2023-12-07 20:22:45.037743052+00:00,1 +2023-12-07 20:22:45.207954144+00:00,1 +2023-12-07 20:22:45.500949380+00:00,1 +2023-12-07 20:22:45.661987081+00:00,1 +2023-12-07 20:22:45.810146063+00:00,1 +2023-12-07 20:22:45.953464968+00:00,1 +2023-12-07 20:22:46.109750557+00:00,1 +2023-12-07 20:22:46.268798896+00:00,1 +2023-12-07 20:22:46.420533443+00:00,1 +2023-12-07 20:22:46.566303735+00:00,1 +2023-12-07 20:22:46.722099803+00:00,1 +2023-12-07 20:22:46.867455199+00:00,1 +2023-12-07 20:22:47.036463670+00:00,1 +2023-12-07 20:22:47.180164213+00:00,1 +2023-12-07 20:22:47.335750947+00:00,1 +2023-12-07 20:22:47.484388926+00:00,1 +2023-12-07 20:22:47.644125301+00:00,1 +2023-12-07 20:22:47.798267973+00:00,1 +2023-12-07 20:22:47.964335015+00:00,1 +2023-12-07 20:22:48.138225569+00:00,1 +2023-12-07 20:22:48.295179460+00:00,1 +2023-12-07 20:22:48.443652649+00:00,1 +2023-12-07 20:22:48.597713573+00:00,1 +2023-12-07 20:22:48.748622445+00:00,1 +2023-12-07 20:22:48.938734254+00:00,1 +2023-12-07 20:22:49.093611473+00:00,1 +2023-12-07 20:22:49.273608529+00:00,1 +2023-12-07 20:22:49.437933235+00:00,1 +2023-12-07 20:22:49.591269123+00:00,1 +2023-12-07 20:22:49.747292396+00:00,1 +2023-12-07 20:22:49.911211870+00:00,1 +2023-12-07 20:22:50.088776242+00:00,1 +2023-12-07 20:22:50.287389367+00:00,1 +2023-12-07 20:22:50.436663166+00:00,1 +2023-12-07 20:22:50.776332484+00:00,1 +2023-12-07 20:22:50.927212033+00:00,1 +2023-12-07 20:22:51.079409493+00:00,1 +2023-12-07 20:22:51.240706230+00:00,1 +2023-12-07 20:22:51.411712852+00:00,1 +2023-12-07 20:22:51.626900042+00:00,1 +2023-12-07 20:22:51.777533097+00:00,1 +2023-12-07 20:22:51.973003806+00:00,1 +2023-12-07 20:22:52.175672588+00:00,1 +2023-12-07 20:22:52.320929257+00:00,1 +2023-12-07 20:22:52.470938247+00:00,1 +2023-12-07 20:22:52.633735932+00:00,1 +2023-12-07 20:22:52.798175061+00:00,1 +2023-12-07 20:22:52.966811991+00:00,1 +2023-12-07 20:22:53.144400544+00:00,1 +2023-12-07 20:22:53.351897917+00:00,1 +2023-12-07 20:22:53.519726777+00:00,1 +2023-12-07 20:22:53.696002408+00:00,1 +2023-12-07 20:22:53.857523998+00:00,1 +2023-12-07 20:22:54.007041438+00:00,1 +2023-12-07 20:22:54.221048253+00:00,1 +2023-12-07 20:22:54.390794503+00:00,1 +2023-12-07 20:22:54.548753363+00:00,1 +2023-12-07 20:22:54.702227805+00:00,1 +2023-12-07 20:22:54.884947711+00:00,1 +2023-12-07 20:22:55.042494952+00:00,1 +2023-12-07 20:22:55.212014258+00:00,1 +2023-12-07 20:22:55.422744564+00:00,1 +2023-12-07 20:22:55.574173889+00:00,1 +2023-12-07 20:22:55.721188900+00:00,1 +2023-12-07 20:22:55.872434289+00:00,1 +2023-12-07 20:22:56.139527454+00:00,1 +2023-12-07 20:22:56.290906064+00:00,1 +2023-12-07 20:22:56.444181633+00:00,1 +2023-12-07 20:22:56.637809862+00:00,1 +2023-12-07 20:22:56.787949203+00:00,1 +2023-12-07 20:22:56.970180024+00:00,1 +2023-12-07 20:22:57.141820094+00:00,1 +2023-12-07 20:22:57.290125403+00:00,1 +2023-12-07 20:22:57.452831120+00:00,1 +2023-12-07 20:22:57.642828015+00:00,1 +2023-12-07 20:22:57.836673407+00:00,1 +2023-12-07 20:22:57.998806591+00:00,1 +2023-12-07 20:22:58.155315787+00:00,1 +2023-12-07 20:22:58.312909883+00:00,1 +2023-12-07 20:22:58.586076111+00:00,1 +2023-12-07 20:22:58.773237159+00:00,1 +2023-12-07 20:22:58.902275032+00:00,1 +2023-12-07 20:22:59.029587065+00:00,1 +2023-12-07 20:22:59.159430512+00:00,1 +2023-12-07 20:22:59.287294054+00:00,1 +2023-12-07 20:22:59.415052622+00:00,1 +2023-12-07 20:22:59.542609103+00:00,1 +2023-12-07 20:22:59.671675360+00:00,1 +2023-12-07 20:22:59.801485238+00:00,1 +2023-12-07 20:22:59.930347381+00:00,1 +2023-12-07 20:23:00.066805247+00:00,1 +2023-12-07 20:23:00.195339778+00:00,1 +2023-12-07 20:23:00.323666257+00:00,1 +2023-12-07 20:23:00.450937422+00:00,1 +2023-12-07 20:23:00.578111381+00:00,1 +2023-12-07 20:23:00.707004951+00:00,0 +2023-12-07 20:23:02.721092840+00:00,1 +2023-12-07 20:23:02.849351257+00:00,1 +2023-12-07 20:23:02.977520370+00:00,1 +2023-12-07 20:23:03.105538488+00:00,1 +2023-12-07 20:23:03.232589497+00:00,0 +2023-12-07 20:23:05.245803173+00:00,1 +2023-12-07 20:23:05.376269337+00:00,1 +2023-12-07 20:23:05.504841777+00:00,1 +2023-12-07 20:23:05.633317261+00:00,1 +2023-12-07 20:23:05.761704789+00:00,1 +2023-12-07 20:23:05.891694930+00:00,1 +2023-12-07 20:23:06.020311716+00:00,1 +2023-12-07 20:23:06.150818057+00:00,1 +2023-12-07 20:23:06.278555778+00:00,1 +2023-12-07 20:23:06.407650159+00:00,1 +2023-12-07 20:23:06.534995236+00:00,1 +2023-12-07 20:23:06.662335919+00:00,1 +2023-12-07 20:23:06.789244591+00:00,1 +2023-12-07 20:23:06.919131325+00:00,1 +2023-12-07 20:23:07.047229900+00:00,1 +2023-12-07 20:23:07.175481731+00:00,1 +2023-12-07 20:23:07.302294510+00:00,1 +2023-12-07 20:23:07.429539187+00:00,1 +2023-12-07 20:23:07.559333613+00:00,1 +2023-12-07 20:23:07.687523396+00:00,1 +2023-12-07 20:23:07.814686317+00:00,1 +2023-12-07 20:23:07.942120864+00:00,1 +2023-12-07 20:23:08.070956186+00:00,1 +2023-12-07 20:23:08.198406655+00:00,1 +2023-12-07 20:23:08.324703617+00:00,1 +2023-12-07 20:23:08.452553149+00:00,1 +2023-12-07 20:23:08.579905991+00:00,1 +2023-12-07 20:23:08.707253737+00:00,1 +2023-12-07 20:23:08.834273375+00:00,1 +2023-12-07 20:23:08.963987759+00:00,1 +2023-12-07 20:23:09.092463983+00:00,1 +2023-12-07 20:23:09.219090950+00:00,1 +2023-12-07 20:23:09.348396961+00:00,1 +2023-12-07 20:23:09.476559232+00:00,1 +2023-12-07 20:23:09.603624582+00:00,1 +2023-12-07 20:23:09.731180308+00:00,1 +2023-12-07 20:23:09.858330583+00:00,1 +2023-12-07 20:23:09.987165535+00:00,1 +2023-12-07 20:23:10.116807206+00:00,1 +2023-12-07 20:23:10.244131259+00:00,1 +2023-12-07 20:23:10.371991695+00:00,1 +2023-12-07 20:23:10.499588787+00:00,1 +2023-12-07 20:23:10.630664837+00:00,1 +2023-12-07 20:23:10.758397092+00:00,1 +2023-12-07 20:23:10.887368089+00:00,1 +2023-12-07 20:23:11.019345169+00:00,1 +2023-12-07 20:23:11.150642334+00:00,1 +2023-12-07 20:23:11.278373714+00:00,1 +2023-12-07 20:23:11.406615105+00:00,1 +2023-12-07 20:23:11.535083192+00:00,1 +2023-12-07 20:23:11.662692899+00:00,1 +2023-12-07 20:23:11.791343425+00:00,1 +2023-12-07 20:23:11.920991798+00:00,1 +2023-12-07 20:23:12.050239984+00:00,1 +2023-12-07 20:23:12.180608839+00:00,1 +2023-12-07 20:23:12.308039475+00:00,1 +2023-12-07 20:23:12.435655722+00:00,1 +2023-12-07 20:23:12.562666770+00:00,1 +2023-12-07 20:23:12.689421172+00:00,1 +2023-12-07 20:23:12.818077878+00:00,1 +2023-12-07 20:23:12.947557649+00:00,1 +2023-12-07 20:23:13.076886716+00:00,1 +2023-12-07 20:23:13.206313189+00:00,1 +2023-12-07 20:23:13.334139633+00:00,1 +2023-12-07 20:23:13.461936638+00:00,1 +2023-12-07 20:23:13.595729030+00:00,1 +2023-12-07 20:23:13.723884053+00:00,1 +2023-12-07 20:23:13.851424309+00:00,1 +2023-12-07 20:23:13.979851273+00:00,1 +2023-12-07 20:23:14.107986384+00:00,1 +2023-12-07 20:23:14.235116889+00:00,1 +2023-12-07 20:23:14.362431716+00:00,1 +2023-12-07 20:23:14.490812682+00:00,1 +2023-12-07 20:23:14.617621151+00:00,1 +2023-12-07 20:23:14.745020812+00:00,1 +2023-12-07 20:23:14.871946394+00:00,1 +2023-12-07 20:23:14.999505983+00:00,1 +2023-12-07 20:23:15.131580441+00:00,1 +2023-12-07 20:23:15.259577723+00:00,1 +2023-12-07 20:23:15.387389795+00:00,1 +2023-12-07 20:23:15.514898282+00:00,1 +2023-12-07 20:23:15.642172610+00:00,1 +2023-12-07 20:23:15.770101259+00:00,1 +2023-12-07 20:23:15.898686476+00:00,1 +2023-12-07 20:23:16.026204185+00:00,1 +2023-12-07 20:23:16.154044291+00:00,1 +2023-12-07 20:23:16.282059801+00:00,1 +2023-12-07 20:23:16.409194505+00:00,1 +2023-12-07 20:23:16.537082871+00:00,1 +2023-12-07 20:23:16.664318576+00:00,1 +2023-12-07 20:23:16.792611730+00:00,1 +2023-12-07 20:23:16.921156122+00:00,1 +2023-12-07 20:23:17.049217141+00:00,1 +2023-12-07 20:23:17.177161886+00:00,1 +2023-12-07 20:23:17.304452871+00:00,1 +2023-12-07 20:23:17.432220552+00:00,1 +2023-12-07 20:23:17.559380255+00:00,1 +2023-12-07 20:23:17.686857297+00:00,1 +2023-12-07 20:23:17.814719115+00:00,1 +2023-12-07 20:23:17.942483451+00:00,1 +2023-12-07 20:23:18.069133274+00:00,1 +2023-12-07 20:23:18.196120359+00:00,1 +2023-12-07 20:23:18.326630798+00:00,1 +2023-12-07 20:23:18.454135426+00:00,1 +2023-12-07 20:23:18.581638545+00:00,1 +2023-12-07 20:23:18.708940729+00:00,1 +2023-12-07 20:23:18.836707023+00:00,1 +2023-12-07 20:23:18.964256397+00:00,1 +2023-12-07 20:23:19.091985442+00:00,1 +2023-12-07 20:23:19.219895593+00:00,1 +2023-12-07 20:23:19.348067914+00:00,1 +2023-12-07 20:23:19.476224623+00:00,1 +2023-12-07 20:23:19.603087369+00:00,1 +2023-12-07 20:23:19.730833735+00:00,1 +2023-12-07 20:23:19.858004828+00:00,1 +2023-12-07 20:23:19.986233750+00:00,1 +2023-12-07 20:23:20.113980681+00:00,1 +2023-12-07 20:23:20.240746429+00:00,1 +2023-12-07 20:23:20.368555392+00:00,1 +2023-12-07 20:23:20.496438131+00:00,1 +2023-12-07 20:23:20.624200851+00:00,1 +2023-12-07 20:23:20.751964334+00:00,1 +2023-12-07 20:23:20.879228450+00:00,1 +2023-12-07 20:23:21.006273927+00:00,1 +2023-12-07 20:23:21.133668032+00:00,1 +2023-12-07 20:23:21.261193211+00:00,1 +2023-12-07 20:23:21.389594515+00:00,1 +2023-12-07 20:23:21.516733530+00:00,1 +2023-12-07 20:23:21.644174238+00:00,1 +2023-12-07 20:23:21.772936407+00:00,1 +2023-12-07 20:23:21.900522373+00:00,1 +2023-12-07 20:23:22.028971175+00:00,1 +2023-12-07 20:23:22.157274475+00:00,1 +2023-12-07 20:23:22.284838198+00:00,1 +2023-12-07 20:23:22.412483107+00:00,1 +2023-12-07 20:23:22.539658693+00:00,1 +2023-12-07 20:23:22.667768380+00:00,1 +2023-12-07 20:23:22.796694934+00:00,1 +2023-12-07 20:23:22.924670185+00:00,1 +2023-12-07 20:23:23.052259855+00:00,1 +2023-12-07 20:23:23.180284402+00:00,1 +2023-12-07 20:23:23.309025712+00:00,1 +2023-12-07 20:23:23.436490382+00:00,1 +2023-12-07 20:23:23.564052188+00:00,1 +2023-12-07 20:23:23.692088405+00:00,1 +2023-12-07 20:23:23.820239457+00:00,1 +2023-12-07 20:23:23.947639353+00:00,1 +2023-12-07 20:23:24.075370357+00:00,1 +2023-12-07 20:23:24.203261429+00:00,1 +2023-12-07 20:23:24.333210356+00:00,1 +2023-12-07 20:23:24.461401764+00:00,1 +2023-12-07 20:23:24.589209463+00:00,1 +2023-12-07 20:23:24.716728517+00:00,1 +2023-12-07 20:23:24.845159342+00:00,1 +2023-12-07 20:23:24.975528822+00:00,1 +2023-12-07 20:23:25.102088447+00:00,1 +2023-12-07 20:23:25.228770253+00:00,1 +2023-12-07 20:23:25.355859502+00:00,1 +2023-12-07 20:23:25.484390582+00:00,1 +2023-12-07 20:23:25.611511432+00:00,1 +2023-12-07 20:23:25.738719496+00:00,1 +2023-12-07 20:23:25.867738865+00:00,1 +2023-12-07 20:23:25.994083972+00:00,1 +2023-12-07 20:23:26.121617617+00:00,1 +2023-12-07 20:23:26.248420910+00:00,1 +2023-12-07 20:23:26.375075807+00:00,1 +2023-12-07 20:23:26.501965715+00:00,1 +2023-12-07 20:23:26.629495416+00:00,1 +2023-12-07 20:23:26.757648934+00:00,1 +2023-12-07 20:23:26.884268966+00:00,1 +2023-12-07 20:23:27.012969660+00:00,1 +2023-12-07 20:23:27.140127338+00:00,1 +2023-12-07 20:23:27.267756708+00:00,1 +2023-12-07 20:23:27.396314416+00:00,1 +2023-12-07 20:23:27.525270596+00:00,1 +2023-12-07 20:23:27.654080958+00:00,1 +2023-12-07 20:23:27.782357554+00:00,1 +2023-12-07 20:23:27.909846838+00:00,1 +2023-12-07 20:23:28.038193831+00:00,1 +2023-12-07 20:23:28.165201845+00:00,1 +2023-12-07 20:23:28.293275758+00:00,1 +2023-12-07 20:23:28.422335401+00:00,1 +2023-12-07 20:23:28.550307912+00:00,1 +2023-12-07 20:23:28.678352834+00:00,1 +2023-12-07 20:23:28.806032601+00:00,1 +2023-12-07 20:23:28.935739926+00:00,1 +2023-12-07 20:23:29.064369587+00:00,1 +2023-12-07 20:23:29.191790679+00:00,1 +2023-12-07 20:23:29.318667122+00:00,1 +2023-12-07 20:23:29.446150343+00:00,1 +2023-12-07 20:23:29.573502339+00:00,1 +2023-12-07 20:23:29.700820314+00:00,1 +2023-12-07 20:23:29.828045809+00:00,1 +2023-12-07 20:23:29.956252451+00:00,1 +2023-12-07 20:23:30.084376536+00:00,1 +2023-12-07 20:23:30.211835948+00:00,1 +2023-12-07 20:23:30.339673881+00:00,1 +2023-12-07 20:23:30.467106601+00:00,1 +2023-12-07 20:23:30.593301022+00:00,1 +2023-12-07 20:23:30.721149000+00:00,1 +2023-12-07 20:23:30.849263212+00:00,1 +2023-12-07 20:23:30.977439835+00:00,1 +2023-12-07 20:23:31.103723843+00:00,1 +2023-12-07 20:23:31.231175540+00:00,1 +2023-12-07 20:23:31.358296559+00:00,1 +2023-12-07 20:23:31.485680384+00:00,1 +2023-12-07 20:23:31.613079372+00:00,1 +2023-12-07 20:23:31.741006222+00:00,1 +2023-12-07 20:23:31.869721688+00:00,1 +2023-12-07 20:23:31.997336519+00:00,1 +2023-12-07 20:23:32.125361365+00:00,1 +2023-12-07 20:23:32.255175898+00:00,1 +2023-12-07 20:23:32.382403464+00:00,1 +2023-12-07 20:23:32.509107929+00:00,1 +2023-12-07 20:23:32.636933262+00:00,1 +2023-12-07 20:23:32.764345569+00:00,1 +2023-12-07 20:23:32.893941155+00:00,1 +2023-12-07 20:23:33.021903010+00:00,1 +2023-12-07 20:23:33.149973453+00:00,1 +2023-12-07 20:23:33.278577543+00:00,1 +2023-12-07 20:23:33.406785112+00:00,1 +2023-12-07 20:23:33.534280386+00:00,1 +2023-12-07 20:23:33.662537251+00:00,1 +2023-12-07 20:23:33.792096905+00:00,1 +2023-12-07 20:23:33.923470080+00:00,1 +2023-12-07 20:23:34.054144319+00:00,1 +2023-12-07 20:23:34.182156650+00:00,1 +2023-12-07 20:23:34.311320307+00:00,1 +2023-12-07 20:23:34.439297962+00:00,1 +2023-12-07 20:23:34.566336124+00:00,1 +2023-12-07 20:23:34.694896111+00:00,1 +2023-12-07 20:23:34.822539517+00:00,1 +2023-12-07 20:23:34.951802998+00:00,1 +2023-12-07 20:23:35.080501769+00:00,1 +2023-12-07 20:23:35.207030949+00:00,1 +2023-12-07 20:23:35.334571972+00:00,1 diff --git a/tests/zero-downtime-upgrades/results/1.1.0/https.png b/tests/zero-downtime-upgrades/results/1.1.0/https.png new file mode 100644 index 0000000000000000000000000000000000000000..d85f8de785828e281be70545e191239cb61f37e8 GIT binary patch literal 6441 zcmeHLX;f3!_TR!%S`ld#K|lfRt6Gsk6c8aM0xDJ!CqzUDg#wDqC}B>J3Ti;oR+)d;WyU#wq^Sk?; zv-i1k`qVL%wOVTdK;^id?O6atJphG^pDD^)ik^m8%NyYI$@51L1d+F7XJ@Zu^#BB< z9uN=;>oL&M0*9{-BlUVfz|<2EV|M}(07C$TfZ&lgk9Y5ucf}07miN=r(%QFgpS``k zx3@PQkALvsL0MT@eSLjfTN{tZlMgX%VnPy5>P0&LggYc&CK6G=kIi%AV$7@CeH6W;|5is&ZAh_2P+!4YGehiW=Pwq%|5K>={ z$mfI9Gb=Nvk?HIT<_qKnv;J8$g2_ho>gwv`=%fm;tO!f1VQ~$JKZE(TFsBSNDj-@9 zQ(u5!1MoM(hfOe{4&$4Frv;i|~|INM?1E9m*>G#`M*gYeQ4enUY$8WI?U;!B74`3WSs( zNDhIdP`G;oZsQ;^9BxIz4^eP41_*e-6Cma$MEwAfw;((baJS*cT?kErkYor-fj|=Y z{{+|WgKsKaA%oAa;FSiR55Xe?+_M0i4XzY$$$?9c;bJa0Jq5>nI9CW~i{LaB9G-!F zDV!*WZ=Zu*B^;}QBQ&t70qa`8FaTW-sNcZyB^+#o1I=LG0;aEEUmKV(!MFpAUV}jw z=)Zy8EZD_{9dF_5KKNBo4L2h^K{&?b_Tze7=7QzTZe3TiGug9MMt7xtYqwzUi3(;lZA~RRXfc^;dQkt=gp?9u1K}M z?+XlNVCs4$oB&C8^8xqZ@m!7D)I)QWMM3v)Mkd8Fi#+9#JSE#tE)w+=2hb;+mad96 zL@eGJw7NoTe&<^F9iw3tTX~dN%N0~~9B@S|#YD!f@LY;XIM*`$rR@zJLpQXNW+iRH z9O3m02jbsEn4WRdrY&1;o6VlkVXJOa&x&6sPX7ZoSHQc%wnW!r)Y0j4nuiR2)Cr2S zQ?0`M4=X;-aQsHJJXiWF>fUcPo3tixS6X{kV`90FUdd~%t7jxSj$=iyFD5$cwUs2U zi+NenyiV$z`0(ae35VYGdM3yUZBD)T(ndF!zt;w@zU!`9_Jy02_bA<^$ORGlxwHYX z_dBhB_{>~pM>J~r>Oxq8$mN8vZh8q3`wRXQ&036Hn2@N`x_vZSxG^}T=Nr_HOf9yt z8=S~UotXD(_(A`|jUCk3zN(=- zN_*rXN6>E-|Jm?9tSY)FGn7HSY*C!+?WQ+-Z!BFmB<_z7Se7)j{9J_e=)&+f7jRk{ zE!P`w94nZ@bI0fxw=c-&j%&C!GhtLXsy&=O>`4+ew*4(Iu4gS-G!oDMd_7LOtnl{p zH{YICi_y?%nZ%sQffY<^SNQ%^{KUgwMsQc`^0vp(Im^;oZK8WHfS1SDXwLJ6GdU?y z^i)UG^4pA9!&h8}6fsm!pKbJrxwbb@11F9xk5@{J{9bcNScMggx9WO!Y(s8zPy@5@ zeaTRLeV=5Ks5GZiRFkIOlC4?!VYs7sA}j1UF}c`K!|CBK+JQ$GC#t0x7s~LaJhxoywUto%4vWy#X~DSw(U2tI^=)?)+GJi76B5CKcVRt;H;b+OQtd zJgPAtcALAIJ;AG*r;>j@8X?Z%r|v?diWqF)k12?|-+sO#HK{-J-1?n~5pnm&TL(K@ z@mUhDvyLMkm;7or@p66U#eZr|PKX&RdQ`kyLpg_DNNhdobkr>=iF(*jJM{j*_#3Qb z6pcpZF`R>Gwc!g&o*e2mAN25>EbBB;#QEnjYwmoUDSe!QWhCJuhpScu{~lkk+QbpL z9419K29++%nA*kc8w*%fyXC(}9`YZocz>#)bTc>K*Noe^n>JgG_h5yMOpMV)-uJUI zd$_))l4iVd#oCWD`o;15Jq4nZLHVy}>4%i6WBETO51&i%U6AY=!JgD~i@a<;m^9P9 zQDRx-jV!1(c{4D0MuS6_xmd7sf;*{wX;J0VKJaN1{B#0*Itf3y0sm)r=G8{5?tRp%3Tz0#=)bNxI*UTqQ*x9qbV5iVlzv^83$PZszlFc;T#vAmp9 zI%-cjGKLi_veEHOB4WyoUNi9_|Kc7}u^&xnRdeuaYbcv#NE%t39rr$7Xeb+x<`U=f zB%Ehj#me7bHXPYl5PbFeW988+vgK;$4FyC5#f?~F;nZMlVW4goky*^}b2KD93XZEW zQqIZi8~#VhMLk!|Zq}s6M77`zcNF~^FKbI^B@)qDf(GRB(t^Koj@GFS0u3Khuxa=j ziya+$Q>_Y`Y>Yg58rzpZJ-q(a>Y2JdB$?d(q`B%f&@Udr9g-OK1TPx)^p|32-V2*Y zz2oNCj$Su(0zQV49?gW3bQSJW5@l933tR=?umAG&?N)s%u7ok4c=$8&`~T#jZX++Z zNylXF%VwQ@b3R)`*52E^blWpUbE#~-#p+}K4pmbg83*2KI`~c=!y1yf;-@C^ntT=4 z9ZQYS->-f(&dv&5yXQY~v2+8cdUwGSZL|I-d+3uYf`zwUrkuRISF|YOuoCh-f#a#M z*Q_Z()y-`nX6T&jg&UH-Ubk~*M#&u0&Q?;vDtfap?LBMAbz|+J>C8KBDF!l&PTdPm z{+$;Z8oGOOLoIZvu6|B8*tylMCMg+}!ZrM@^rVu#VNugF?*$#7p|7jfdWPfI-EmVHH0>-Xq3?*4Dk4!7G~a5u^OxKSYYW`~QB_70bEXFM z(c8#eF)YRPWv><;6&@@~Ra8|>V>gAdu0{@orN}`N{{X2j*I^pYw&m(pg-24Rnq;9A zh7GyFZx8878ZxkjK5%JWFDmTIk>TE=7i1!JOI6Dc4C9Kt*^Ts&*InFe(KJ%x5^rQ)6Ed_tmoN3~mmRO4 zs}ktYqbRu~s{NdYUr7`tP-!atJ*!tv+7`*EE&c*7!aBMP)6SXRA5=y0RW}bvX7LqQX0DsZH`9(O=SG zKbOcNvoW}1={EjH^2XWzMDo23Q~>s#L1y$o#Tu+(UC2;t>kE!&Gd38B9WYCwU!!PH zopBCxf=N!s#u&?R?w`6Edw1?!Doa~56>>V8W_#m*err|XT;mZVY(wUIk0c%2*(t}h zLRGj(N!Y~1XD*^c2PUO=rSe(1@H7523N4qK1x2()XV#6(pXFo=$Ts7ReV3N49BRV` zHq&p5hP-;mh9(=G)?Z#1Xk2GpQM-&C>?5a0=4`Px>W#uXb(${xhkw~uP>Cmn?3tDR zzT=?L;I`iIyU(SDL-Vv*W|2dUq{<@fOHWC14fDl#qkfB2$?)tOc9U#`)12s;mHFOc z^n*LSSzYp%5p1mrkEJj+OYkx2;>xsJ8tXaOEmiN8cekjVDLf8WE#q$8`op&eX`KW=R)zwd?PM^4!mTf5)<7g!3ts{jB1 literal 0 HcmV?d00001 diff --git a/tests/zero-downtime-upgrades/values.yaml b/tests/zero-downtime-upgrades/values.yaml index 679e200cb2..04899c1e68 100644 --- a/tests/zero-downtime-upgrades/values.yaml +++ b/tests/zero-downtime-upgrades/values.yaml @@ -1,9 +1,5 @@ nginxGateway: replicaCount: 2 - image: - repository: gcr.io//nginx-gateway-fabric - tag: - pullPolicy: Always lifecycle: preStop: exec: diff --git a/tests/zero-downtime-upgrades/zero-downtime-upgrades.md b/tests/zero-downtime-upgrades/zero-downtime-upgrades.md index 4dd57e757f..6fbaf6e599 100644 --- a/tests/zero-downtime-upgrades/zero-downtime-upgrades.md +++ b/tests/zero-downtime-upgrades/zero-downtime-upgrades.md @@ -85,9 +85,14 @@ Notes: ### Start 1. Create a cluster. -2. Deploy a previous latest stable version with 2 replicas with added [anti-affinity](#pod-affinity). -3. Expose NGF via a Service Load Balancer, internal (only accessible within the Google Cloud region) by adding - `networking.gke.io/load-balancer-type: "Internal"` annotation to the Service. +2. Install the Gateway API resources that are supported by the latest release of NGF. +3. Install the latest release of NGF: + + ```console + helm install ngf-test oci://ghcr.io/nginxinc/charts/nginx-gateway-fabric --create-namespace --wait -n nginx-gateway --values values.yaml + ``` + + This deploys NGF with 2 replicas with added [anti-affinity](#pod-affinity). 4. Deploy backend apps: ```console @@ -118,7 +123,7 @@ Notes: ### Upgrade -1. Follow the [upgrade instructions](https://docs.nginx.com/nginx-gateway-fabric/installation/installing-ngf/manifests/) to: +1. Follow the [upgrade instructions](https://docs.nginx.com/nginx-gateway-fabric/installation/installing-ngf/helm/#upgrade-nginx-gateway-fabric) to: 1. Upgrade Gateway API version to the one that matches the supported version of new release. 2. Upgrade NGF CRDs. 2. Start sending traffic using wrk from tester VMs for 1 minute: @@ -148,9 +153,12 @@ Notes: for i in `seq 1 600`; do printf "\nRequest $i\n" && date --rfc-3339=ns && curl -k -sS --connect-timeout 2 https://cafe.example.com/tea 2>&1 && sleep 0.1s; done > results.txt ``` -3. **Immediately** upgrade NGF manifests by - following [upgrade instructions](https://docs.nginx.com/nginx-gateway-fabric/installation/installing-ngf/manifests/). - > Don't forget to modify the manifests to have 2 replicas and Pod affinity. +3. **Immediately** upgrade NGF: + + ```console + helm upgrade ngf-test oci://ghcr.io/nginxinc/charts/nginx-gateway-fabric -n nginx-gateway --version 0.0.0-edge --values values.yaml --wait + ``` + 4. Ensure the new Pods are running and the old ones terminate. ### After Upgrade @@ -203,6 +211,7 @@ Notes: - [1.0.0](results/1.0.0/1.0.0.md) - [1.0.0-special](results/1.0.0-special/1.0.0-special.md) +- [1.1.0](results/1.1.0/1.1.0.md) ## Appendix From e0b5f2c3026fcab6c5066eb395d9273cf55ca173 Mon Sep 17 00:00:00 2001 From: Saylor Berman Date: Thu, 7 Dec 2023 15:37:20 -0700 Subject: [PATCH 2/2] Update tests/zero-downtime-upgrades/zero-downtime-upgrades.md Co-authored-by: Michael Pleshakov --- tests/zero-downtime-upgrades/zero-downtime-upgrades.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/zero-downtime-upgrades/zero-downtime-upgrades.md b/tests/zero-downtime-upgrades/zero-downtime-upgrades.md index 6fbaf6e599..e1b11b166d 100644 --- a/tests/zero-downtime-upgrades/zero-downtime-upgrades.md +++ b/tests/zero-downtime-upgrades/zero-downtime-upgrades.md @@ -153,7 +153,7 @@ Notes: for i in `seq 1 600`; do printf "\nRequest $i\n" && date --rfc-3339=ns && curl -k -sS --connect-timeout 2 https://cafe.example.com/tea 2>&1 && sleep 0.1s; done > results.txt ``` -3. **Immediately** upgrade NGF: +3. **Immediately** upgrade NGF to the edge version: ```console helm upgrade ngf-test oci://ghcr.io/nginxinc/charts/nginx-gateway-fabric -n nginx-gateway --version 0.0.0-edge --values values.yaml --wait