diff --git a/tests/reconfig/results/v1.0.0.md b/tests/reconfig/results/v1.0.0.md new file mode 100644 index 000000000..803ede268 --- /dev/null +++ b/tests/reconfig/results/v1.0.0.md @@ -0,0 +1,61 @@ +# Reconfiguration testing Results + + +- [Reconfiguration testing Results](#reconfiguration-testing-results) + - [Test environment](#test-environment) + - [Results Table](#results-table) + - [NumResources -\> Total Resources](#numresources---total-resources) + - [Observations](#observations) + + +## Test environment + +GKE cluster: + +- Node count: 3 +- Instance Type: e2-medium +- k8s version: 1.27.4-gke.900 +- Zone: europe-west2-b +- Total vCPUs: 6 +- Total RAM: 12GB +- Max pods per node: 110 + +NGF deployment: + +- NGF version: edge - git commit 72b6c6ef8915c697626eeab88fdb6a3ce15b8da0 +- NGINX Version: 1.25.2 + +## Results Table + +| Test number | NumResources | TimeToReadyTotal (s) | TimeToReadyAvgSingle (s) | NGINX reloads | NGINX reload avg time (ms) | +| ----------- | ------------ | -------------------- | ------------------------ | ------------- | -------------------------- | +| 1 | 30 | 5 | 5 | 2 | 166 | +| 1 | 150 | 7 | 7 | 2 | 353 | +| 2 | 30 | 21 | <1 | 30 | 142 | +| 2 | 150 | 123 | <1 | 46 | 190 | +| 3 | 30 | <1 | <1 | 93 | 137 | +| 3 | 150 | 1 | 1 | 453 | 127 | + +## NumResources -> Total Resources +| NumResources | Gateways | Secrets | ReferenceGrants | Namespaces | application Pods | application Services | HTTPRoutes | Total Resources | +| ------------ | -------- | ------- | --------------- | ---------- | ---------------- | -------------------- | ---------- | --------------- | +| x | 1 | 1 | 1 | x+1 | 2x | 2x | 3x | | +| 30 | 1 | 1 | 1 | 31 | 60 | 60 | 90 | 244 | +| 150 | 1 | 1 | 1 | 151 | 300 | 300 | 450 | 1204 | + +## Observations + +1. We are reloading after reconciling a ReferenceGrant even when there is no Gateway. This is because we treat every + upsert/delete of a ReferenceGrant as a change. This means we will regenerate NGINX config every time a ReferenceGrant + is created, updated (generation must change), or deleted, even if it does not apply to the accepted Gateway. + + Issue filed: https://github.com/nginxinc/nginx-gateway-fabric/issues/1124 + +2. We are reloading after reconciling a HTTPRoute even when there is no accepted Gateway and no config being generated. + + Issue filed: https://github.com/nginxinc/nginx-gateway-fabric/issues/1123 + +3. All reloads were in the <500ms bucket. A slight increase in the reload time based on number of configured resources + resulting in NGINX configuration changes was observed. + +4. No errors (NGF or NGINX) were observed in any test run. diff --git a/tests/reconfig/scripts/cafe-routes.yaml b/tests/reconfig/scripts/cafe-routes.yaml new file mode 100644 index 000000000..3d182d0ad --- /dev/null +++ b/tests/reconfig/scripts/cafe-routes.yaml @@ -0,0 +1,57 @@ +apiVersion: gateway.networking.k8s.io/v1beta1 +kind: HTTPRoute +metadata: + name: cafe-tls-redirect +spec: + parentRefs: + - name: gateway.networking.k8s.io/v1beta1 + namespace: default + sectionName: http + hostnames: + - "cafe.example.com" + rules: + - filters: + - type: RequestRedirect + requestRedirect: + scheme: https + port: 443 +--- +apiVersion: gateway.networking.k8s.io/v1beta1 +kind: HTTPRoute +metadata: + name: coffee +spec: + parentRefs: + - name: gateway + namespace: default + sectionName: https + hostnames: + - "cafe.example.com" + rules: + - matches: + - path: + type: PathPrefix + value: /coffee + backendRefs: + - name: coffee + port: 80 +--- +apiVersion: gateway.networking.k8s.io/v1beta1 +kind: HTTPRoute +metadata: + name: tea +spec: + parentRefs: + - name: gateway + sectionName: https + namespace: default + hostnames: + - "cafe.example.com" + rules: + - matches: + - path: + type: PathPrefix + value: /tea + backendRefs: + - name: tea + port: 80 diff --git a/tests/reconfig/scripts/cafe.yaml b/tests/reconfig/scripts/cafe.yaml new file mode 100644 index 000000000..2d03ae59f --- /dev/null +++ b/tests/reconfig/scripts/cafe.yaml @@ -0,0 +1,65 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: coffee +spec: + replicas: 1 + selector: + matchLabels: + app: coffee + template: + metadata: + labels: + app: coffee + spec: + containers: + - name: coffee + image: nginxdemos/nginx-hello:plain-text + ports: + - containerPort: 8080 +--- +apiVersion: v1 +kind: Service +metadata: + name: coffee +spec: + ports: + - port: 80 + targetPort: 8080 + protocol: TCP + name: http + selector: + app: coffee +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: tea +spec: + replicas: 1 + selector: + matchLabels: + app: tea + template: + metadata: + labels: + app: tea + spec: + containers: + - name: tea + image: nginxdemos/nginx-hello:plain-text + ports: + - containerPort: 8080 +--- +apiVersion: v1 +kind: Service +metadata: + name: tea +spec: + ports: + - port: 80 + targetPort: 8080 + protocol: TCP + name: http + selector: + app: tea diff --git a/tests/reconfig/scripts/certificate-ns-and-cafe-secret.yaml b/tests/reconfig/scripts/certificate-ns-and-cafe-secret.yaml new file mode 100644 index 000000000..d4037e2d6 --- /dev/null +++ b/tests/reconfig/scripts/certificate-ns-and-cafe-secret.yaml @@ -0,0 +1,14 @@ +apiVersion: v1 +kind: Namespace +metadata: + name: certificate +--- +apiVersion: v1 +kind: Secret +metadata: + name: cafe-secret + namespace: certificate +type: kubernetes.io/tls +data: + tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUNzakNDQVpvQ0NRQzdCdVdXdWRtRkNEQU5CZ2txaGtpRzl3MEJBUXNGQURBYk1Sa3dGd1lEVlFRRERCQmoKWVdabExtVjRZVzF3YkdVdVkyOXRNQjRYRFRJeU1EY3hOREl4TlRJek9Wb1hEVEl6TURjeE5ESXhOVEl6T1ZvdwpHekVaTUJjR0ExVUVBd3dRWTJGbVpTNWxlR0Z0Y0d4bExtTnZiVENDQVNJd0RRWUpLb1pJaHZjTkFRRUJCUUFECmdnRVBBRENDQVFvQ2dnRUJBTHFZMnRHNFc5aStFYzJhdnV4Q2prb2tnUUx1ek10U1Rnc1RNaEhuK3ZRUmxIam8KVzFLRnMvQVdlS25UUStyTWVKVWNseis4M3QwRGtyRThwUisxR2NKSE50WlNMb0NEYUlRN0Nhck5nY1daS0o4Qgo1WDNnVS9YeVJHZjI2c1REd2xzU3NkSEQ1U2U3K2Vab3NPcTdHTVF3K25HR2NVZ0VtL1Q1UEMvY05PWE0zZWxGClRPL051MStoMzROVG9BbDNQdTF2QlpMcDNQVERtQ0thaEROV0NWbUJQUWpNNFI4VERsbFhhMHQ5Z1o1MTRSRzUKWHlZWTNtdzZpUzIrR1dYVXllMjFuWVV4UEhZbDV4RHY0c0FXaGRXbElweHlZQlNCRURjczN6QlI2bFF1OWkxZAp0R1k4dGJ3blVmcUVUR3NZdWxzc05qcU95V1VEcFdJelhibHhJZVVDQXdFQUFUQU5CZ2txaGtpRzl3MEJBUXNGCkFBT0NBUUVBcjkrZWJ0U1dzSnhLTGtLZlRkek1ISFhOd2Y5ZXFVbHNtTXZmMGdBdWVKTUpUR215dG1iWjlpbXQKL2RnWlpYVE9hTElHUG9oZ3BpS0l5eVVRZVdGQ2F0NHRxWkNPVWRhbUloOGk0Q1h6QVJYVHNvcUNOenNNLzZMRQphM25XbFZyS2lmZHYrWkxyRi8vblc0VVNvOEoxaCtQeDljY0tpRDZZU0RVUERDRGh1RUtFWXcvbHpoUDJVOXNmCnl6cEJKVGQ4enFyM3paTjNGWWlITmgzYlRhQS82di9jU2lyamNTK1EwQXg4RWpzQzYxRjRVMTc4QzdWNWRCKzQKcmtPTy9QNlA0UFlWNTRZZHMvRjE2WkZJTHFBNENCYnExRExuYWRxamxyN3NPbzl2ZzNnWFNMYXBVVkdtZ2todAp6VlZPWG1mU0Z4OS90MDBHUi95bUdPbERJbWlXMGc9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg== + tls.key: LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1JSUV2UUlCQURBTkJna3Foa2lHOXcwQkFRRUZBQVNDQktjd2dnU2pBZ0VBQW9JQkFRQzZtTnJSdUZ2WXZoSE4KbXI3c1FvNUtKSUVDN3N6TFVrNExFeklSNS9yMEVaUjQ2RnRTaGJQd0ZuaXAwMFBxekhpVkhKYy92TjdkQTVLeApQS1VmdFJuQ1J6YldVaTZBZzJpRU93bXF6WUhGbVNpZkFlVjk0RlAxOGtSbjl1ckV3OEpiRXJIUncrVW51L25tCmFMRHF1eGpFTVBweGhuRklCSnYwK1R3djNEVGx6TjNwUlV6dnpidGZvZCtEVTZBSmR6N3Rid1dTNmR6MHc1Z2kKbW9RelZnbFpnVDBJek9FZkV3NVpWMnRMZllHZWRlRVJ1VjhtR041c09va3R2aGxsMU1udHRaMkZNVHgySmVjUQo3K0xBRm9YVnBTS2NjbUFVZ1JBM0xOOHdVZXBVTHZZdFhiUm1QTFc4SjFINmhFeHJHTHBiTERZNmpzbGxBNlZpCk0xMjVjU0hsQWdNQkFBRUNnZ0VBQnpaRE50bmVTdWxGdk9HZlFYaHRFWGFKdWZoSzJBenRVVVpEcUNlRUxvekQKWlV6dHdxbkNRNlJLczUyandWNTN4cU9kUU94bTNMbjNvSHdNa2NZcEliWW82MjJ2dUczYnkwaVEzaFlsVHVMVgpqQmZCcS9UUXFlL2NMdngvSkczQWhFNmJxdFRjZFlXeGFmTmY2eUtpR1dzZk11WVVXTWs4MGVJVUxuRmZaZ1pOCklYNTlSOHlqdE9CVm9Sa3hjYTVoMW1ZTDFsSlJNM3ZqVHNHTHFybmpOTjNBdWZ3ZGRpK1VDbGZVL2l0K1EvZkUKV216aFFoTlRpNVFkRWJLVStOTnYvNnYvb2JvandNb25HVVBCdEFTUE05cmxFemIralQ1WHdWQjgvLzRGY3VoSwoyVzNpcjhtNHVlQ1JHSVlrbGxlLzhuQmZ0eVhiVkNocVRyZFBlaGlPM1FLQmdRRGlrR3JTOTc3cjg3Y1JPOCtQClpoeXltNXo4NVIzTHVVbFNTazJiOTI1QlhvakpZL2RRZDVTdFVsSWE4OUZKZnNWc1JRcEhHaTFCYzBMaTY1YjIKazR0cE5xcVFoUmZ1UVh0UG9GYXRuQzlPRnJVTXJXbDVJN0ZFejZnNkNQMVBXMEg5d2hPemFKZUdpZVpNYjlYTQoybDdSSFZOcC9jTDlYbmhNMnN0Q1lua2Iwd0tCZ1FEUzF4K0crakEyUVNtRVFWNXA1RnRONGcyamsyZEFjMEhNClRIQ2tTazFDRjhkR0Z2UWtsWm5ZbUt0dXFYeXNtekJGcnZKdmt2eUhqbUNYYTducXlpajBEdDZtODViN3BGcVAKQWxtajdtbXI3Z1pUeG1ZMXBhRWFLMXY4SDNINGtRNVl3MWdrTWRybVJHcVAvaTBGaDVpaGtSZS9DOUtGTFVkSQpDcnJjTzhkUVp3S0JnSHA1MzRXVWNCMVZibzFlYStIMUxXWlFRUmxsTWlwRFM2TzBqeWZWSmtFb1BZSEJESnp2ClIrdzZLREJ4eFoyWmJsZ05LblV0YlhHSVFZd3lGelhNcFB5SGxNVHpiZkJhYmJLcDFyR2JVT2RCMXpXM09PRkgKcmppb21TUm1YNmxhaDk0SjRHU0lFZ0drNGw1SHhxZ3JGRDZ2UDd4NGRjUktJWFpLZ0w2dVJSSUpBb0dCQU1CVApaL2p5WStRNTBLdEtEZHUrYU9ORW4zaGxUN3hrNXRKN3NBek5rbWdGMU10RXlQUk9Xd1pQVGFJbWpRbk9qbHdpCldCZ2JGcXg0M2ZlQ1Z4ZXJ6V3ZEM0txaWJVbWpCTkNMTGtYeGh3ZEVteFQwVit2NzZGYzgwaTNNYVdSNnZZR08KditwVVovL0F6UXdJcWZ6dlVmV2ZxdStrMHlhVXhQOGNlcFBIRyt0bEFvR0FmQUtVVWhqeFU0Ym5vVzVwVUhKegpwWWZXZXZ5TW54NWZyT2VsSmRmNzlvNGMvMHhVSjh1eFBFWDFkRmNrZW96dHNpaVFTNkN6MENRY09XVWxtSkRwCnVrdERvVzM3VmNSQU1BVjY3NlgxQVZlM0UwNm5aL2g2Tkd4Z28rT042Q3pwL0lkMkJPUm9IMFAxa2RjY1NLT3kKMUtFZlNnb1B0c1N1eEpBZXdUZmxDMXc9Ci0tLS0tRU5EIFBSSVZBVEUgS0VZLS0tLS0K diff --git a/tests/reconfig/scripts/create-resources-gw-last.sh b/tests/reconfig/scripts/create-resources-gw-last.sh new file mode 100644 index 000000000..cf01af317 --- /dev/null +++ b/tests/reconfig/scripts/create-resources-gw-last.sh @@ -0,0 +1,31 @@ +#!/bin/bash + +num_namespaces=$1 + +# Create namespaces +for ((i=1; i<=$num_namespaces; i++)); do + namespace_name="namespace$i" + kubectl create namespace "$namespace_name" +done + +# Create single instance resources +kubectl create -f certificate-ns-and-cafe-secret.yaml +kubectl create -f reference-grant.yaml + +# Create backend service and apps +for ((i=1; i<=$num_namespaces; i++)); do + namespace_name="namespace$i" + sed -e "s/coffee/coffee${namespace_name}/g" -e "s/tea/tea${namespace_name}/g" cafe.yaml | kubectl apply -n "$namespace_name" -f - +done + +# Create routes +for ((i=1; i<=$num_namespaces; i++)); do + namespace_name="namespace$i" + sed -e "s/coffee/coffee${namespace_name}/g" -e "s/tea/tea${namespace_name}/g" cafe-routes.yaml | kubectl apply -n "$namespace_name" -f - +done + +# Wait for apps to be ready +sleep 60 + +# Create Gateway +kubectl create -f gateway.yaml diff --git a/tests/reconfig/scripts/create-resources-routes-last.sh b/tests/reconfig/scripts/create-resources-routes-last.sh new file mode 100644 index 000000000..88d819ec7 --- /dev/null +++ b/tests/reconfig/scripts/create-resources-routes-last.sh @@ -0,0 +1,29 @@ +#!/bin/bash + +num_namespaces=$1 + +# Create namespaces +for ((i=1; i<=$num_namespaces; i++)); do + namespace_name="namespace$i" + kubectl create namespace "$namespace_name" +done + +# Create backend service and apps +for ((i=1; i<=$num_namespaces; i++)); do + namespace_name="namespace$i" + sed -e "s/coffee/coffee${namespace_name}/g" -e "s/tea/tea${namespace_name}/g" cafe.yaml | kubectl apply -n "$namespace_name" -f - +done + +# Wait for apps to be ready +sleep 60 + +# Create single instance resources +kubectl create -f certificate-ns-and-cafe-secret.yaml +kubectl create -f reference-grant.yaml +kubectl create -f gateway.yaml + +# Create routes +for ((i=1; i<=$num_namespaces; i++)); do + namespace_name="namespace$i" + sed -e "s/coffee/coffee${namespace_name}/g" -e "s/tea/tea${namespace_name}/g" cafe-routes.yaml | kubectl apply -n "$namespace_name" -f - +done diff --git a/tests/reconfig/scripts/delete-multiple.sh b/tests/reconfig/scripts/delete-multiple.sh new file mode 100644 index 000000000..0e46bc275 --- /dev/null +++ b/tests/reconfig/scripts/delete-multiple.sh @@ -0,0 +1,14 @@ +#!/bin/bash + +num_namespaces=$1 + +# Delete namespaces +for ((i=1; i<=$num_namespaces; i++)); do + namespace_name="namespace$i" + kubectl delete namespace "$namespace_name" +done + +# Delete single instance resources +kubectl delete -f gateway.yaml +kubectl delete -f reference-grant.yaml +kubectl delete -f certificate-ns-and-cafe-secret.yaml diff --git a/tests/reconfig/scripts/gateway.yaml b/tests/reconfig/scripts/gateway.yaml new file mode 100644 index 000000000..a0fd45cc2 --- /dev/null +++ b/tests/reconfig/scripts/gateway.yaml @@ -0,0 +1,25 @@ +apiVersion: gateway.networking.k8s.io/v1beta1 +kind: Gateway +metadata: + name: gateway +spec: + gatewayClassName: nginx + listeners: + - name: http + port: 80 + protocol: HTTP + allowedRoutes: + namespaces: + from: "All" + - name: https + port: 443 + protocol: HTTPS + allowedRoutes: + namespaces: + from: "All" + tls: + mode: Terminate + certificateRefs: + - kind: Secret + name: cafe-secret + namespace: certificate diff --git a/tests/reconfig/scripts/reference-grant.yaml b/tests/reconfig/scripts/reference-grant.yaml new file mode 100644 index 000000000..053bbbdcc --- /dev/null +++ b/tests/reconfig/scripts/reference-grant.yaml @@ -0,0 +1,14 @@ +apiVersion: gateway.networking.k8s.io/v1beta1 +kind: ReferenceGrant +metadata: + name: access-to-cafe-secret + namespace: certificate +spec: + to: + - group: "" + kind: Secret + name: cafe-secret # if you omit this name, then Gateways in default ns can access all Secrets in the certificate ns + from: + - group: gateway.networking.k8s.io + kind: Gateway + namespace: default diff --git a/tests/reconfig/setup.md b/tests/reconfig/setup.md new file mode 100644 index 000000000..4ad3e7048 --- /dev/null +++ b/tests/reconfig/setup.md @@ -0,0 +1,105 @@ +# Reconfig tests + + +- [Reconfig tests](#reconfig-tests) + - [Goals](#goals) + - [Test Environment](#test-environment) + - [Setup](#setup) + - [Tests](#tests) + - [Test 1: Resources exist before start-up](#test-1-resources-exist-before-start-up) + - [Test 2: Start NGF, deploy Gateway, create many resources attached to GW](#test-2-start-ngf-deploy-gateway-create-many-resources-attached-to-gw) + - [Test 3: Start NGF, create many resources attached to a Gateway, deploy the Gateway](#test-3-start-ngf-create-many-resources-attached-to-a-gateway-deploy-the-gateway) + + +## Goals + +- Measure how long it takes NGF to reconfigure NGINX when a number of Gateway API and referenced core Kubernetes + resources are created at once. +- Two runs of each test should be ran with differing numbers of resources. Each run will deploy: + - a single Gateway, Secret, and ReferenceGrant resources + - `x+1` number of namespaces + - `2x` number of backend apps and services + - `3x` number of HTTPRoutes. +- Where x=30 OR x=150. + +## Test Environment + + The following cluster will be sufficient: + +- A Kubernetes cluster with 3 nodes on GKE + - Node: e2-medium (2 vCPU, 4GB memory) + +## Setup + +1. Create cloud cluster +2. Deploy CRDs: + + ```bash + kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v0.8.1/standard-install.yaml + ``` + +3. Deploy NGF from edge using Helm install (NOTE: For Test 1, deploy AFTER resources): + + ```console + helm install my-release oci://ghcr.io/nginxinc/charts/nginx-gateway-fabric --version 0.0.0-edge \ + --create-namespace --wait -n nginx-gateway + ``` + +4. Run tests: + 1. There are 3 versions of the reconfiguration tests that need to be ran, with a low and high number of resources. + Therefore, a full test suite includes 6 test runs. + 2. There are scripts to generate the required resources and config changes. + 3. Run each test using the provided script (`scripts/create-resources-gw-last.sh` or + `scripts/create-resources-routes-last.sh` depending on the test). + 4. The scripts accept a number parameter to indicate how many resources should be created. Currently, we are running + with 30 or 150. The scripts will create a single Gateway, Secret and ReferenceGrant resources, `x+1` number of + namespaces, `2x` number of backend apps and services, and `3x` number of HTTPRoutes. + - Note: Clean up after each test run for isolated results. There's a script provided for removing all the test + fixtures `scripts/delete-multiple.sh` which takes a number (needs to be the same number as what was used in the + create script.) +5. After each individual test run, grab logs of both NGF containers and grab metrics. + Note: You can expose metrics by running the below snippet and then navigating to `127.0.0.1:9113/metrics`: + + ```console + GW_POD=$(k get pods -n nginx-gateway | sed -n '2s/^\([^[:space:]]*\).*$/\1/p') + kubectl port-forward $GW_POD -n nginx-gateway 9113:9113 & + ``` + +6. Measure Time To Ready as described in each test, get the reload count, and get the average NGINX reload duration. + The average reload duration can be computed by taking the `nginx_gateway_fabric_nginx_reloads_milliseconds_sum` + metric value and dividing it by the `nginx_gateway_fabric_nginx_reloads_milliseconds_count` metric value. +7. For accuracy, repeat the test suite once or twice, take the averages, and look for any anomolies or outliers. + +## Tests + +### Test 1: Resources exist before start-up + +1. Deploy Gateway resources before start-up: + 1. Use either of the provided scripts with the required number of resources, + e.g. `cd scripts && bash create-resources-gw-last.sh 30`. The script will deploy backend apps and services, wait + 60 seconds for them to be ready, and deploy 1 Gateway, 1 RefGrant, 1 Secret, and HTTPRoutes. + 2. Deploy NGF + 3. Check logs for time it takes from start-up -> config written and NGINX reloaded. Get reload count and average reload + duration from metrics and logs. + +### Test 2: Start NGF, deploy Gateway, create many resources attached to GW + +1. Deploy all Gateway resources, NGF running: + 1. Deploy NGF + 2. Run the provided script with the required number of resources, + e.g. `cd scripts && bash create-resources-routes-last.sh 30`. The script will deploy backend apps and services, + wait 60 seconds for them to be ready, and deploy 1 Gateway, 1 Secret, 1 RefGrant, and HTTPRoutes at the same time. + 3. Check logs for time it takes from NGF receiving first resource update -> final config written, and NGINX's final + reload. Check logs for average individual HTTPRoute TTR also. Get reload count and average reload duration from + metrics and logs. + +### Test 3: Start NGF, create many resources attached to a Gateway, deploy the Gateway + +1. Deploy HTTPRoute resources, NGF running, Gateway last: + 1. Deploy NGF + 2. Run the provided script with the required number of resources, + e.g. `cd scripts && bash create-resources-gw-last.sh 30`. + The script will deploy the namespaces, backend apps and services, 1 Secret, 1 ReferenceGrant, and the HTTPRoutes; + wait 60 seconds for the backend apps to be ready, and then deploy 1 Gateway for all HTTPRoutes. + 3. Check logs for time it takes from NGF receiving gateway resource -> config written and NGINX reloaded. Get reload + count and average reload duration from metrics and logs.