Spike investigation in using EnvoyPatchPolicy and/or Extension Manager to register our extensions and configure things according to the state of the cluster's network objects (we would initially need to watch all resources ourselves as far as I can tell, just like with Istio/OSSM)

[eguzki]

• Envoy Gateway integration API for rate limiting WASM module
  • Candidate API: EnvoyExtensionPolicy, Supports WASM filter. ⚠️ Not available in v1.0.X, has been merged into main, so it should be available in v1.1
  • Alternatives: both requiring high level of Envoy and Envoy Gateway expertise
    • Envoy Extension Manager can be used to programmatically mutate Listener filters and HTTP Connection Manager filters. ⚠️ Discarded in favor of Envoy Patch Policy as being more "kubernetes" native (use of CRDs instead of implementing service endpoint for patching resources)
    • Envoy Patch Policy can be used to patch Listener filters and HTTP Connection Manager filters.

Envoy Patch Policy approach

• Disabled by default, extra step is needed on Envoy Gateway deployment to enable this feature
• No way to rely on OCI/docker images with the Wasm module. Binary needs to be exposed using HTTP endpoint.
• WASM binary served from github release-binaries branch https://github.com/Kuadrant/wasm-shim/tree/release-binaries where binary can be downloaded without 30X redirects (like it happens with release assets https://github.com/Kuadrant/wasm-shim/releases/download/v0.4.0-alpha.1/kuadrant-ratelimit-wasm-v0.4.0-alpha.1)
• It requires creation of envoy clusters for limitador and also for the WASM module origin (https://raw.githubusercontent.com/Kuadrant/wasm-shim/release-binaries/releases/kuadrant-ratelimit-wasm-v0.4.0-alpha.1)
• The rate limit wasm http filter added as the first of the list. What if we need to put behind external auth filter???

Howto

git clone [email protected]:eguzki/poc-kuadrant-envoy-gateway.git
git checkout ff975c9fa4d36e457efb2556b828c11e75ebed4e
make local-env-setup

kubectl apply -f examples/toystore/toystore.yaml

kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: toystore
spec:
  parentRefs:
  - name: eg
    namespace: envoy-gateway-system
  hostnames:
  - api.example.com
  rules:
  - matches:
    - method: GET
      path:
        type: PathPrefix
        value: "/toys"
    backendRefs:
    - name: toystore
      port: 80
EOF

• Enable EnvoyPatchPolicy

kubectl get configmap -n envoy-gateway-system envoy-gateway-config -o jsonpath='{.data.envoy-gateway\.yaml}' > envoy-gateway.yaml
# Enable EnvoyPatchPolicy
yq e '.extensionApis.enableEnvoyPatchPolicy = true' -i envoy-gateway.yaml
kubectl create configmap -n envoy-gateway-system envoy-gateway-config --from-file=envoy-gateway.yaml -o yaml --dry-run=client | kubectl replace -f -

kubectl rollout restart deployment envoy-gateway -n envoy-gateway-system

• deploy limitador

k apply -n kuadrant-system -f - <<EOF
apiVersion: limitador.kuadrant.io/v1alpha1
kind: Limitador
metadata:
  name: kuadrant
spec:
  limits:
  - conditions: []
    max_value: 5
    namespace: "default/toystore"
    seconds: 10
    variables: []
EOF

• create envoy patch policy

kubectl apply -n envoy-gateway-system -f - <<EOF
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyPatchPolicy
metadata:
  name: kuadrant-rate-limiting-wasm-filter
  namespace: envoy-gateway-system
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: eg
    namespace: envoy-gateway-system
  type: JSONPatch
  jsonPatches:
    - type: "type.googleapis.com/envoy.config.cluster.v3.Cluster"
      name: kuadrant-rate-limiting-service
      operation:
        op: add
        path: ""
        value:
          name: kuadrant-rate-limiting-service
          type: STRICT_DNS
          connect_timeout: 1s
          lb_policy: ROUND_ROBIN
          http2_protocol_options: {}
          load_assignment:
            cluster_name: kuadrant-rate-limiting-service
            endpoints:
            - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: limitador-kuadrant.kuadrant-system.svc.cluster.local
                      port_value: 8081
    - type: "type.googleapis.com/envoy.config.cluster.v3.Cluster"
      name: raw_githubusercontent_com_443
      operation:
        op: add
        path: ""
        value:
          name: raw_githubusercontent_com_443
          type: STRICT_DNS
          connect_timeout: 1s
          dns_refresh_rate: 5s
          dns_lookup_family: V4_ONLY
          load_assignment:
            cluster_name: raw_githubusercontent_com_443
            endpoints:
            - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: raw.githubusercontent.com
                      port_value: 443
          transport_socket:
            name: envoy.transport_sockets.tls
            typed_config:
              '@type': type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
              sni: raw.githubusercontent.com
    - type: type.googleapis.com/envoy.config.listener.v3.Listener
      # The listener name is of the form <GatewayNamespace>/<GatewayName>/<GatewayListenerName>
      name: envoy-gateway-system/eg/http
      operation:
        op: add
        path: "/default_filter_chain/filters/0/typed_config/http_filters/0"
        value:
          name: kuadrant.ratelimiting.wasm
          typed_config:
            '@type': type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm
            config:
              name: kuadrant.ratelimiting
              root_id: kuadrant_ratelimiting
              vm_config:
                vm_id: kuadrant_ratelimiting_vm_id
                runtime: envoy.wasm.runtime.v8
                code:
                  remote:
                    sha256: b101508ddd5fd40eb2116204e6c768332a359c21feb2dbb348956459349e7d71
                    http_uri:
                      uri: https://raw.githubusercontent.com/Kuadrant/wasm-shim/release-binaries/releases/kuadrant-ratelimit-wasm-v0.4.0-alpha.1
                      cluster: raw_githubusercontent_com_443
                      timeout: 10s
              configuration:
                '@type': type.googleapis.com/google.protobuf.StringValue
                value: |
                  {
                    "failureMode": "deny",
                    "rateLimitPolicies": [
                      {
                        "domain": "default/toystore",
                        "hostnames": [
                          "api.example.com"
                        ],
                        "name": "default/toystore",
                        "rules": [
                          {
                            "conditions": [
                              {
                                "allOf": [
                                  {
                                    "operator": "eq",
                                    "selector": "request.url_path",
                                    "value": "/toys"
                                  },
                                  {
                                    "operator": "eq",
                                    "selector": "request.method",
                                    "value": "GET"
                                  }
                                ]
                              }
                            ],
                            "data": [
                              {
                                "static": {
                                  "key": "limit.create_toy__88f990c6",
                                  "value": "1"
                                }
                              }
                            ]
                          }
                        ],
                        "service": "kuadrant-rate-limiting-service"
                      }
                    ]
                  }
EOF

export INGRESS_HOST=$(kubectl get gtw eg -n envoy-gateway-system -o jsonpath='{.status.addresses[0].value}')
export INGRESS_PORT=$(kubectl get gtw eg -n envoy-gateway-system -o jsonpath='{.spec.listeners[?(@.name=="http")].port}')

curl -v --resolve api.example.com:$INGRESS_PORT:$INGRESS_HOST http://api.example.com:$INGRESS_PORT/toys -i

while :; do curl --write-out '%{http_code}\n' --silent --output /dev/null --resolve api.example.com:$INGRESS_PORT:$INGRESS_HOST "http://api.example.com:$INGRESS_PORT/toys" -i | grep -E --color "\b(429)\b|$"; sleep 1; done