sotw: rlp workflow

[guicassolato]

State-of-the-world reconciler – RLP workflow

• spec.targetRef.sectionName
• spec.(defaults|overrides).strategy
• Defaults & Overrides merge strategies (RFC-0009)
• Effective RateLimitPolicy
• Limitador CR (limits)
• Istio cluster (EnvoyFilter)
• istio extension (WasmPlugin) - requires Config revamp wasm-shim#110
• envoy cluster (EnvoyPatchPolicy)
• envoy extension (EnvoyExtensionPolicy) - requires Config revamp wasm-shim#110
• Accepted status condition
• Enforced status condition
• Split the workflow tasks into separate files

Closes #821
Closes #477
Closes #475

Verification steps

Make sure you have the latest versions of the binaries that are referred by the Makefile. (This is especially important regarding istioctl which is required to be 1.22.5)

rm -rfv ./bin/*

Setup the environment:

make local-setup

(Until Kuadrant/wasm-shim#110 is merged) Patch the deployment to enable the new wasm-shim config structure:

kubectl set env -n kuadrant-system deployment/kuadrant-operator-controller-manager RELATED_IMAGE_WASMSHIM=quay.io/kuadrant/wasm-shim:config-revamp

Enable Envoy Gateway alongside with Istio:

make envoy-gateway-install

kubectl apply -f -<<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: envoygateway
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
EOF

kubectl apply -n gateway-system -f -<<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: kuadrant-envoygateway
spec:
  gatewayClassName: envoygateway
  listeners:
  - name: http
    hostname: "*.eg.apps.io"
    protocol: HTTP
    port: 80
    allowedRoutes:
      namespaces:
        from: Same
EOF

Deploy an application:

kubectl apply -f examples/toystore/toystore.yaml

kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: toystore
spec:
  parentRefs:
  - name: kuadrant-ingressgateway
    namespace: gateway-system
  - name: kuadrant-envoygateway
    namespace: gateway-system
  rules:
  - backendRefs:
    - name: toystore
      port: 80
    matches:
    - method: GET
  - backendRefs:
    - name: toystore
      port: 80
    matches:
    - method: POST
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: other
spec:
  hostnames:
  - other.example.com
  parentRefs:
  - name: kuadrant-ingressgateway
    namespace: gateway-system
  rules:
  - backendRefs:
    - name: toystore
      port: 80
EOF

(From now on and at anytime) Send requests to the application:

export INGRESS_IP=$(kubectl get gateway/kuadrant-ingressgateway -n gateway-system -o jsonpath='{.status.addresses[0].value}')

curl --resolve toystore.example.com:80:$INGRESS_IP http://toystore.example.com --write-out '%{http_code}\n' --silent --output /dev/null
curl --resolve other.example.com:80:$INGRESS_IP http://other.example.com --write-out '%{http_code}\n' --silent --output /dev/null

Deploy Kuadrant:

kubectl -n kuadrant-system apply -f - <<EOF
apiVersion: kuadrant.io/v1beta1
kind: Kuadrant
metadata:
  name: kuadrant
spec: {}
EOF

Create a gateway atomic default policy:

kubectl apply -n gateway-system -f - <<EOF
apiVersion: kuadrant.io/v1beta3
kind: RateLimitPolicy
metadata:
  name: gw-rlp
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: kuadrant-ingressgateway
  defaults:
    limits:
      "global":
        rates:
        - limit: 5
          duration: 10
          unit: second
        when:
        - selector: source.address
          operator: neq
          value: 127.0.0.1
EOF

Create a route policy:

kubectl apply -f - <<EOF
apiVersion: kuadrant.io/v1beta3
kind: RateLimitPolicy
metadata:
  name: route-rlp
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: toystore
  limits:
    "specific":
      rates:
      - limit: 3
        duration: 5
        unit: second
      - limit: 20
        duration: 1
        unit: minute
EOF

Modify the gateway policy to atomic override strategy:

kubectl apply -n gateway-system -f - <<EOF
apiVersion: kuadrant.io/v1beta3
kind: RateLimitPolicy
metadata:
  name: gw-rlp
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: kuadrant-ingressgateway
  overrides:
    limits:
      "global":
        rates:
        - limit: 5
          duration: 10
          unit: second
        when:
        - selector: source.address
          operator: neq
          value: 127.0.0.1
EOF

Modify the gateway policy to merge override strategy:

kubectl apply -n gateway-system -f - <<EOF
apiVersion: kuadrant.io/v1beta3
kind: RateLimitPolicy
metadata:
  name: gw-rlp
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: kuadrant-ingressgateway
  overrides:
    limits:
      "global":
        rates:
        - limit: 5
          duration: 10
          unit: second
        when:
        - selector: source.address
          operator: neq
          value: 127.0.0.1
    strategy: merge
EOF

Modify the route policy to define limits with the same name as limits defined by the gateway policy:

kubectl apply -f - <<EOF
apiVersion: kuadrant.io/v1beta3
kind: RateLimitPolicy
metadata:
  name: route-rlp
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: toystore
  limits:
    "global":
      rates:
      - limit: 3
        duration: 5
        unit: second
      - limit: 20
        duration: 1
        unit: minute
EOF

Try other use cases not covered in the verification steps.

In general, this PR should enable:

• RateLimitPolicies targeting a Gateway
• RateLimitPolicies targeting a specific Listener of a Gateway (with spec.targetRef.sectionName)
• RateLimitPolicies targeting a HTTPRoute
• RateLimitPolicies targeting a specific HTTPRouteRule of a HTTPRoute (with spec.targetRef.sectionName)
• Multiple RateLimitPolicies targeting a same resource - different sections of the resource
• Multiple RateLimitPolicies targeting a same resource - same section of the resource
• Multiple RateLimitPolicies targeting a same resource - entire resource and section of the resource
• Multiple RateLimitPolicies targeting a same resource - entire resource
• Defaults and Overrides in RateLimitPolicies targeting Gateways
• Defaults and Overrides in RateLimitPolicies targeting HTTPRoutes
• Defaults and Overrides' policy merge strategy in RateLimitPolicies (spec.defaults.strategy: merge, spec.overrides.strategy: merge)
• Multiple Gateways parenting a HTTPRoute
• Gateways controlled by Istio only
• Gateways controlled by Envoy Gateway only
• Mixed Gateways controlled by Istio and Envoy Gateway
• Status of Gateways and HTTPRoutes reflected in the status Enforced condition of the policies
• Status of Limitador CR reflected in the status Enforced condition of the policies
• Presence of Istio WasmPlugin CRs reflected in the status Enforced condition of the policies
• Presence of Istio EnvoyFilter CRs reflected in the status Enforced condition of the policies
• Presence of Envoy Gateway EnvoyExtensionPolicy CRs reflected in the status Enforced condition of the policies
• Presence of Envoy Gateway EnvoyPatchPolicy CRs reflected in the status Enforced condition of the policies

[codecov]

Codecov Report

Attention: Patch coverage is 79.90588% with 427 lines in your changes missing coverage. Please review.

Project coverage is 78.75%. Comparing base (63f1d28) to head (b2d57ea).
Report is 26 commits behind head on main.

Files with missing lines	Patch %	Lines
api/v1/merge_strategies.go	41.83%	54 Missing and 3 partials ⚠️
controllers/istio_rate_limit_cluster_reconciler.go	72.63%	42 Missing and 13 partials ⚠️
controllers/envoy_gateway_extension_reconciler.go	80.82%	30 Missing and 7 partials ⚠️
pkg/wasm/utils.go	72.86%	27 Missing and 8 partials ⚠️
api/v1beta3/ratelimitpolicy_types.go	75.19%	27 Missing and 5 partials ⚠️
pkg/library/gatewayapi/types.go	56.52%	23 Missing and 7 partials ⚠️
...ers/envoy_gateway_rate_limit_cluster_reconciler.go	83.42%	20 Missing and 9 partials ⚠️
controllers/istio_extension_reconciler.go	83.79%	19 Missing and 10 partials ⚠️
pkg/wasm/types.go	69.79%	20 Missing and 9 partials ⚠️
pkg/istio/utils.go	70.76%	15 Missing and 4 partials ⚠️
... and 9 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #893      +/-   ##
==========================================
- Coverage   81.49%   78.75%   -2.75%     
==========================================
  Files         102      113      +11     
  Lines        7177     9558    +2381     
==========================================
+ Hits         5849     7527    +1678     
- Misses        898     1618     +720     
+ Partials      430      413      -17     

Flag	Coverage Δ
bare-k8s-integration	8.95% <6.21%> (+0.05%)	⬆️
controllers-integration	67.54% <74.96%> (+2.22%)	⬆️
envoygateway-integration	47.04% <53.22%> (-3.26%)	⬇️
gatewayapi-integration	12.89% <7.85%> (-1.52%)	⬇️
istio-integration	47.82% <58.58%> (-5.69%)	⬇️
unit	27.81% <8.32%> (-0.53%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
api/v1beta1 (u)	90.00% <ø> (-0.91%)	⬇️
api/v1beta2 (u)	∅ <ø> (∅)
pkg/common (u)	87.67% <88.88%> (-0.47%)	⬇️
pkg/istio (u)	58.57% <72.05%> (-12.95%)	⬇️
pkg/log (u)	93.18% <ø> (-1.56%)	⬇️
pkg/reconcilers (u)	∅ <ø> (∅)
pkg/rlptools (u)	∅ <ø> (∅)
controllers (i)	81.93% <84.02%> (-1.13%)	⬇️

Files with missing lines	Coverage Δ
api/v1beta3/topology.go	60.00% <ø> (-1.54%)	⬇️
controllers/authorino_reconciler.go	82.50% <100.00%> (-12.10%)	⬇️
controllers/consoleplugin_reconciler.go	80.24% <100.00%> (ø)
controllers/limitador_reconciler.go	81.15% <100.00%> (-18.85%)	⬇️
controllers/ratelimitpolicies_validator.go	100.00% <100.00%> (ø)
controllers/state_of_the_world.go	91.87% <100.00%> (-5.37%)	⬇️
controllers/test_common.go	100.00% <100.00%> (ø)
pkg/envoygateway/mutators.go	31.91% <ø> (-4.20%)	⬇️
pkg/istio/mutators.go	25.45% <ø> (-24.55%)	⬇️
pkg/library/reconcilers/base_reconciler.go	51.87% <100.00%> (-13.91%)	⬇️
... and 21 more

... and 56 files with indirect coverage changes

[eguzki]

leave the policy status to me, it is WIP in another branch

[eguzki]

leave the policy status to me, it is WIP in another branch

Policy status belongs to this workflow. On you now.