install question: in v0.2.0 use Backend

[googs1025]

Description:

Describe the issue.

I'm adapting v0.2.0 to other projects here, but I seem to have some problems.

I use yaml like this:
But it seems that the request cannot be 🤔

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: default-envoy-ai-gateway
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: default-envoy-ai-gateway
spec:
  gatewayClassName: default-envoy-ai-gateway
  listeners:
    - name: http
      protocol: HTTP
      port: 80
---
apiVersion: aigateway.envoyproxy.io/v1alpha1
kind: AIGatewayRoute
metadata:
  name: default-envoy-ai-gateway
spec:
  schema:
    name: OpenAI
  targetRefs:
    - name: default-envoy-ai-gateway
      kind: Gateway
      group: gateway.networking.k8s.io
  rules:
    - matches:
        - headers:
            - type: Exact
              name: x-ai-eg-model
              value: qwen3-0--6b
      backendRefs:
        - name: qwen3-0--6b
      modelsOwnedBy: "llmaz"
      timeouts:
        request: 3m
---
apiVersion: aigateway.envoyproxy.io/v1alpha1
kind: AIServiceBackend
metadata:
  name: qwen3-0--6b
spec:
  schema:
    name: OpenAI
  backendRef:
    name: qwen3-0--6b
    kind: Backend
    group: gateway.envoyproxy.io
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: Backend
metadata:
  name: qwen3-0--6b
  namespace: default
spec:
  endpoints:
    - fqdn:
        hostname: qwen3-0--6b-lb.default.svc.cluster.local
        port: 8080

envoy-default-default-envoy-ai-gateway pod log:
It seems to return directly without any request to the backend.

root@VM-0-13-ubuntu:/home/ubuntu# kubectl get pods -nllmaz-system
NAME                                                              READY   STATUS    RESTARTS   AGE
ai-gateway-controller-659ccb85f-ms6rx                             1/1     Running   0          106m
envoy-default-default-envoy-ai-gateway-dbec795a-7c749cbb8bxkz8n   2/2     Running   0          70m
envoy-gateway-66bb759c9-fz4pd                                     1/1     Running   0          106m
llmaz-controller-manager-76fc68d85d-5h2zc                         1/1     Running   0          106m
llmaz-lws-controller-manager-6969954975-cgsh2                     1/1     Running   0          106m
open-webui-66d8b668b8-ft7tx                                       1/1     Running   0          106m
root@VM-0-13-ubuntu:/home/ubuntu# kubectl logs -f envoy-default-default-envoy-ai-gateway-dbec795a-7c749cbb8bxkz8n -nllmaz-system
Defaulted container "envoy" out of: envoy, shutdown-manager
{":authority":"localhost:8080","bytes_received":0,"bytes_sent":0,"connection_termination_details":null,"downstream_local_address":"127.0.0.1:10080","downstream_remote_address":"127.0.0.1:43924","duration":0,"method":"GET","protocol":"HTTP/1.1","requested_server_name":null,"response_code":500,"response_code_details":"direct_response","response_flags":"-","route_name":"httproute/default/default-envoy-ai-gateway/rule/2/match/0/*","start_time":"2025-07-05T02:46:49.051Z","upstream_cluster":null,"upstream_host":null,"upstream_local_address":null,"upstream_transport_failure_reason":null,"user-agent":"curl/8.5.0","x-envoy-origin-path":"/v1/models","x-envoy-upstream-service-time":null,"x-forwarded-for":"10.6.2.37","x-request-id":"4c148a11-2dc6-4728-a75d-561d36fe7086"}
{":authority":"localhost:8080","bytes_received":0,"bytes_sent":0,"connection_termination_details":null,"downstream_local_address":"127.0.0.1:10080","downstream_remote_address":"127.0.0.1:32994","duration":0,"method":"GET","protocol":"HTTP/1.1","requested_server_name":null,"response_code":500,"response_code_details":"direct_response","response_flags":"-","route_name":"httproute/default/default-envoy-ai-gateway/rule/2/match/0/*","start_time":"2025-07-05T02:47:17.912Z","upstream_cluster":null,"upstream_host":null,"upstream_local_address":null,"upstream_transport_failure_reason":null,"user-agent":"curl/8.5.0","x-envoy-origin-path":"/v1/models","x-envoy-upstream-service-time":null,"x-forwarded-for":"10.6.2.37","x-request-id":"c146e642-531b-41aa-9958-6f5d61965783"}
{":authority":"localhost:8080","bytes_received":0,"bytes_sent":0,"connection_termination_details":null,"downstream_local_address":"127.0.0.1:10080","downstream_remote_address":"127.0.0.1:59726","duration":0,"method":"GET","protocol":"HTTP/1.1","requested_server_name":null,"response_code":500,"response_code_details":"direct_response","response_flags":"-","route_name":"httproute/default/default-envoy-ai-gateway/rule/2/match/0/*","start_time":"2025-07-05T02:47:41.914Z","upstream_cluster":null,"upstream_host":null,"upstream_local_address":null,"upstream_transport_failure_reason":null,"user-agent":"curl/8.5.0","x-envoy-origin-path":"/v1/models","x-envoy-upstream-service-time":null,"x-forwarded-for":"10.6.2.37","x-request-id":"5693f1e9-2117-4baf-83bb-4514a132d193"}
{":authority":"localhost:8080","bytes_received":0,"bytes_sent":0,"connection_termination_details":null,"downstream_local_address":"127.0.0.1:10080","downstream_remote_address":"127.0.0.1:59738","duration":0,"method":"GET","protocol":"HTTP/1.1","requested_server_name":null,"response_code":500,"response_code_details":"direct_response","response_flags":"-","route_name":"httproute/default/default-envoy-ai-gateway/rule/2/match/0/*","start_time":"2025-07-05T02:47:46.777Z","upstream_cluster":null,"upstream_host":null,"upstream_local_address":null,"upstream_transport_failure_reason":null,"user-agent":"curl/8.5.0","x-envoy-origin-path":"/v1/models","x-envoy-upstream-service-time":null,"x-forwarded-for":"10.6.2.37","x-request-id":"2e4c9963-0da0-4b76-9905-e0c75b853694"}
{":authority":"localhost:8080","bytes_received":0,"bytes_sent":0,"connection_termination_details":null,"downstream_local_address":"127.0.0.1:10080","downstream_remote_address":"127.0.0.1:47424","duration":0,"method":"GET","protocol":"HTTP/1.1","requested_server_name":null,"response_code":500,"response_code_details":"direct_response","response_flags":"-","route_name":"httproute/default/default-envoy-ai-gateway/rule/2/match/0/*","start_time":"2025-07-05T02:53:37.248Z","upstream_cluster":null,"upstream_host":null,"upstream_local_address":null,"upstream_transport_failure_reason":null,"user-agent":"curl/8.5.0","x-envoy-origin-path":"/v1/mo

ai-gateway crd

root@VM-0-13-ubuntu:/home/ubuntu# kubectl get aigatewayroutes
NAME                       STATUS
default-envoy-ai-gateway   Accepted
root@VM-0-13-ubuntu:/home/ubuntu# kubectl get aiservicebackends
NAME             STATUS
qwen2--5-coder   Accepted
qwen2-0--5b      Accepted
root@VM-0-13-ubuntu:/home/ubuntu# kubectl get back

backend crd

ai-gateway/examples/basic/basic.yaml

Lines 195 to 206 in b305dd5

	---
	apiVersion: gateway.envoyproxy.io/v1alpha1
	kind: Backend
	metadata:
	name: envoy-ai-gateway-basic-openai
	namespace: default
	spec:
	endpoints:
	- fqdn:
	hostname: api.openai.com
	port: 443
	---

It seems that there is some problem here, but I am trying to add backend according to this example. Is there any other operation I have not done yet? 🤔

root@VM-0-13-ubuntu:/home/ubuntu# kubectl get backend
NAME                               STATUS   AGE
envoy-ai-gateway-extproc-backend            102s
qwen2--5-coder                              102s
qwen2-0--5b                                 102s
root@VM-0-13-ubuntu:/home/ubuntu# kubectl describe backend
Name:         envoy-ai-gateway-extproc-backend
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  gateway.envoyproxy.io/v1alpha1
Kind:         Backend
Metadata:
  Creation Timestamp:  2025-07-05T03:58:15Z
  Generation:          1
  Resource Version:    15943710
  UID:                 94ac0cce-49a4-43a6-b987-dff999e6d7f3
Spec:
  Endpoints:
    Unix:
      Path:  /etc/ai-gateway-extproc-uds/run.sock
  Type:      Endpoints
Status:
  Conditions:
    Last Transition Time:  2025-07-05T03:58:15Z
    Message:               The Backend was not accepted since Backend is not enabled in Envoy Gateway Config
    Observed Generation:   1
    Reason:                Accepted
    Status:                False
    Type:                  Invalid
Events:                    <none>


Name:         qwen2--5-coder
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  gateway.envoyproxy.io/v1alpha1
Kind:         Backend
Metadata:
  Creation Timestamp:  2025-07-05T03:58:15Z
  Generation:          1
  Resource Version:    15943701
  UID:                 cfa7a1a8-7fc9-4c46-95ee-06edb98dc169
Spec:
  Endpoints:
    Fqdn:
      Hostname:  qwen2--5-coder-lb.default.svc.cluster.local
      Port:      8080
  Type:          Endpoints
Status:
  Conditions:
    Last Transition Time:  2025-07-05T03:58:15Z
    Message:               The Backend was not accepted since Backend is not enabled in Envoy Gateway Config
    Observed Generation:   1
    Reason:                Accepted
    Status:                False
    Type:                  Invalid
Events:                    <none>


Name:         qwen2-0--5b
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  gateway.envoyproxy.io/v1alpha1
Kind:         Backend
Metadata:
  Creation Timestamp:  2025-07-05T03:58:15Z
  Generation:          1
  Resource Version:    15943700
  UID:                 15b9028f-b2f5-45aa-9d8c-d91fccf20d16
Spec:
  Endpoints:
    Fqdn:
      Hostname:  qwen2-0--5b-lb.default.svc.cluster.local
      Port:      8080
  Type:          Endpoints
Status:
  Conditions:
    Last Transition Time:  2025-07-05T03:58:15Z
    Message:               The Backend was not accepted since Backend is not enabled in Envoy Gateway Config
    Observed Generation:   1
    Reason:                Accepted
    Status:                False
    Type:                  Invalid
Events:                    <none>

[optional Relevant Links:]

Any extra documentation required to understand the issue.

[googs1025]

Am I missing this config? 🤔

apiVersion: v1
kind: ConfigMap
metadata:
  name: envoy-gateway-config
  namespace: llmaz-system
  labels:
    helm.sh/chart: gateway-helm-v0.0.0-latest
    app.kubernetes.io/name: gateway-helm
    app.kubernetes.io/instance: eg
    app.kubernetes.io/version: "latest"
    app.kubernetes.io/managed-by: Helm
data:
  envoy-gateway.yaml: |
    apiVersion: gateway.envoyproxy.io/v1alpha1
    kind: EnvoyGateway
    gateway:
      controllerName: gateway.envoyproxy.io/gatewayclass-controller
    logging:
      level:
        default: info
    provider:
      type: Kubernetes
    extensionApis:
      enableEnvoyPatchPolicy: true
      enableBackend: true

[ion-elgreco]

@googs1025

You definetely need to update that configmap, here is the contents of envoy-gateway.yaml data for my configmap:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyGateway
extensionApis:
  enableBackend: true
  enableEnvoyPatchPolicy: true
extensionManager:
  hooks:
    xdsTranslator:
      post:
      - VirtualHost
      - Translation
  service:
    fqdn:
      hostname: ai-gateway-controller.envoy-ai-gateway-system.svc.cluster.local
      port: 1063
gateway:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
logging:
  level:
    default: info
provider:
  kubernetes:
    rateLimitDeployment:
      container:
        image: privatecr.io/envoyproxy/ratelimit/master:master
      patch:
        type: StrategicMerge
        value:
          spec:
            template:
              spec:
                containers:
                - imagePullPolicy: IfNotPresent
                  name: envoy-ratelimit
    shutdownManager:
      image: privatecr.io/envoyproxy/gateway:v1.4.1
  type: Kubernetes

[googs1025]

thanks for this:
After I created cm and restarted the pod

root@VM-0-13-ubuntu:/home/ubuntu# kubectl get cm -nllmaz-system
NAME                                              DATA   AGE
envoy-default-default-envoy-ai-gateway-dbec795a   2      24m
envoy-gateway-config                              1      155m
kube-root-ca.crt                                  1      6d23h
llmaz-global-config                               1      155m
root@VM-0-13-ubuntu:/home/ubuntu# kubectl describe cm envoy-gateway-config -nllmaz-system
Name:         envoy-gateway-config
Namespace:    llmaz-system
Labels:       app.kubernetes.io/instance=llmaz
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=gateway-helm
              app.kubernetes.io/version=v1.4.0
              helm.sh/chart=gateway-helm-v1.4.0
Annotations:  meta.helm.sh/release-name: llmaz
              meta.helm.sh/release-namespace: llmaz-system

Data
====
envoy-gateway.yaml:
----
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyGateway
extensionApis:
  enableEnvoyPatchPolicy: true
  enableBackend: true
gateway:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
logging:
  level:
    default: info
provider:
  kubernetes:
    rateLimitDeployment:
      container:
        image: docker.io/envoyproxy/ratelimit:3e085e5b
      patch:
        type: StrategicMerge
        value:
          spec:
            template:
              spec:
                containers:
                - imagePullPolicy: IfNotPresent
                  name: envoy-ratelimit
    shutdownManager:
      image: docker.io/envoyproxy/gateway:v1.4.0
  type: Kubernetes



BinaryData
====

Events:  <none>

But it still doesn't seem to work 🤔

root@VM-0-13-ubuntu:/home/ubuntu# kubectl get pods
NAME               READY   STATUS    RESTARTS   AGE
qwen2--5-coder-0   1/1     Running   0          27m
qwen2-0--5b-0      1/1     Running   0          27m
root@VM-0-13-ubuntu:/home/ubuntu#

root@VM-0-13-ubuntu:/home/ubuntu# kubectl get backend
NAME                               STATUS     AGE
envoy-ai-gateway-extproc-backend   Accepted   26m
qwen2--5-coder                     Accepted   26m
qwen2-0--5b                        Accepted   26m
root@VM-0-13-ubuntu:/home/ubuntu#
root@VM-0-13-ubuntu:/home/ubuntu#
root@VM-0-13-ubuntu:/home/ubuntu# curl http://localhost:80/v1/models | jq .
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (7) Failed to connect to localhost port 80 after 0 ms: Couldn't connect to server
root@VM-0-13-ubuntu:/home/ubuntu#
root@VM-0-13-ubuntu:/home/ubuntu#
root@VM-0-13-ubuntu:/home/ubuntu# curl -H "Content-Type: application/json"     -d '{
        "model": "qwen2-0.5b",
        "messages": [
            {
                "role": "system",
                "content": "Hi."
            }
        ]
    }'     http://localhost:8080/v1/chat/completions | jq .
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   165    0     0  100   165      0  52331 --:--:-- --:--:-- --:--:-- 55000
root@VM-0-13-ubuntu:/home/ubuntu#

root@VM-0-13-ubuntu:/home/ubuntu# kubectl logs -f envoy-default-default-envoy-ai-gateway-dbec795a-7c749cbb8b2kzfl -nllmaz-system
Defaulted container "envoy" out of: envoy, shutdown-manager
[2025-07-05 12:59:32.936][1][warning][misc] [source/common/protobuf/message_validator_impl.cc:23] Deprecated field: type envoy.config.cluster.v3.Cluster Using deprecated option 'envoy.config.cluster.v3.Cluster.dns_refresh_rate' from file cluster.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2025-07-05 12:59:32.936][1][warning][misc] [source/common/protobuf/message_validator_impl.cc:23] Deprecated field: type envoy.config.cluster.v3.Cluster Using deprecated option 'envoy.config.cluster.v3.Cluster.respect_dns_ttl' from file cluster.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2025-07-05 12:59:32.936][1][warning][misc] [source/common/protobuf/message_validator_impl.cc:23] Deprecated field: type envoy.config.cluster.v3.Cluster Using deprecated option 'envoy.config.cluster.v3.Cluster.dns_refresh_rate' from file cluster.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2025-07-05 12:59:32.936][1][warning][misc] [source/common/protobuf/message_validator_impl.cc:23] Deprecated field: type envoy.config.cluster.v3.Cluster Using deprecated option 'envoy.config.cluster.v3.Cluster.respect_dns_ttl' from file cluster.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
{":authority":"localhost:8080","bytes_received":0,"bytes_sent":0,"connection_termination_details":null,"downstream_local_address":"127.0.0.1:10080","downstream_remote_address":"127.0.0.1:55992","duration":0,"method":"GET","protocol":"HTTP/1.1","requested_server_name":null,"response_code":500,"response_code_details":"ext_proc_error_gRPC_error_14{upstream_connect_error_or_disconnect/reset_before_headers._reset_reason:_remote_connection_failure,_transport_failure_reason:_immediate_connect_error:_No_such_file_or_directory|remote_address:/etc/ai-gateway-extproc-uds/run.sock}","response_flags":"-","route_name":"httproute/default/default-envoy-ai-gateway/rule/2/match/0/*","start_time":"2025-07-05T13:02:07.286Z","upstream_cluster":null,"upstream_host":null,"upstream_local_address":null,"upstream_transport_failure_reason":null,"user-agent":"curl/8.5.0","x-envoy-origin-path":"/v1/models","x-envoy-upstream-service-time":null,"x-forwarded-for":"10.6.2.49","x-request-id":"8e598610-1857-401f-b7dd-56ef44709499"}
{":authority":"localhost:8080","bytes_received":165,"bytes_sent":0,"connection_termination_details":null,"downstream_local_address":"127.0.0.1:10080","downstream_remote_address":"127.0.0.1:45228","duration":0,"method":"POST","protocol":"HTTP/1.1","requested_server_name":null,"response_code":500,"response_code_details":"ext_proc_error_gRPC_error_14{upstream_connect_error_or_disconnect/reset_before_headers._reset_reason:_remote_connection_failure,_transport_failure_reason:_immediate_connect_error:_No_such_file_or_directory|remote_address:/etc/ai-gateway-extproc-uds/run.sock}","response_flags":"-","route_name":"httproute/default/default-envoy-ai-gateway/rule/2/match/0/*","start_time":"2025-07-05T13:03:58.057Z","upstream_cluster":null,"upstream_host":null,"upstream_local_address":null,"upstream_transport_failure_reason":null,"user-agent":"curl/8.5.0","x-envoy-origin-path":"/v1/chat/completions","x-envoy-upstream-service-time":null,"x-forwarded-for":"10.6.2.49","x-request-id":"1ea29ab7-f6c4-4480-b67d-e30e58a165a2"}
{":authority":"localhost:8080","bytes_received":0,"bytes_sent":0,"connection_termination_details":null,"downstream_local_address":"127.0.0.1:10080","downstream_remote_address":"127.0.0.1:35262","duration":0,"method":"GET","protocol":"HTTP/1.1","requested_server_name":null,"response_code":500,"response_code_details":"ext_proc_error_gRPC_error_14{upstream_connect_error_or_disconnect/reset_before_headers._reset_reason:_remote_connection_failure,_transport_failure_reason:_immediate_connect_error:_No_such_file_or_directory|remote_address:/etc/ai-gateway-extproc-uds/run.sock}","response_flags":"-","route_name":"httproute/default/default-envoy-ai-gateway/rule/2/match/0/*","start_time":"2025-07-05T13:23:45.627Z","upstream_cluster":null,"upstream_host":null,"upstream_local_address":null,"upstream_transport_failure_reason":null,"user-agent":"curl/8.5.0","x-envoy-origin-path":"/v1/models","x-envoy-upstream-service-time":null,"x-forwarded-for":"10.6.2.49","x-request-id":"c29af8d7-c16f-4720-8eb3-c26c1bff2593"}
{":authority":"localhost:8080","bytes_received":165,"bytes_sent":0,"connection_termination_details":null,"downstream_local_address":"127.0.0.1:10080","downstream_remote_address":"127.0.0.1:37084","duration":0,"method":"POST","protocol":"HTTP/1.1","requested_server_name":null,"response_code":500,"response_code_details":"ext_proc_error_gRPC_error_14{upstream_connect_error_or_disconnect/reset_before_headers._reset_reason:_remote_connection_failure,_transport_failure_reason:_immediate_connect_error:_No_such_file_or_directory|remote_address:/etc/ai-gateway-extproc-uds/run.sock}","response_flags":"-","route_name":"httproute/default/default-envoy-ai-gateway/rule/2/match/0/*","start_time":"2025-07-05T13:26:28.006Z","upstream_cluster":null,"upstream_host":null,"upstream_local_address":null,"upstream_transport_failure_reason":null,"user-agent":"curl/8.5.0","x-envoy-origin-path":"/v1/chat/completions","x-envoy-upstream-service-time":null,"x-forwarded-for":"10.6.2.49","x-request-id":"e3b247a1-7651-4e1d-9fd7-8cc8062362c1"}

[ion-elgreco]

@googs1025 you haven't configured this:

extensionManager:
  hooks:
    xdsTranslator:
      post:
      - VirtualHost
      - Translation
  service:
    fqdn:
      hostname: ai-gateway-controller.envoy-ai-gateway-system.svc.cluster.local
      port: 1063

[googs1025]

change to this config, and restart

root@VM-0-13-ubuntu:/home/ubuntu# kubectl describe cm envoy-gateway-config -nllmaz-system
Name:         envoy-gateway-config
Namespace:    llmaz-system
Labels:       app.kubernetes.io/instance=llmaz
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=gateway-helm
              app.kubernetes.io/version=v1.4.0
              helm.sh/chart=gateway-helm-v1.4.0
Annotations:  meta.helm.sh/release-name: llmaz
              meta.helm.sh/release-namespace: llmaz-system

Data
====
envoy-gateway.yaml:
----
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyGateway
extensionApis:
  enableEnvoyPatchPolicy: true
  enableBackend: true
extensionManager:
  hooks:
    xdsTranslator:
      post:
      - VirtualHost
      - Translation
  service:
    fqdn:
      hostname: ai-gateway-controller.envoy-ai-gateway-system.svc.cluster.local
      port: 1063
gateway:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
logging:
  level:
    default: info
provider:
  kubernetes:
    rateLimitDeployment:
      container:
        image: docker.io/envoyproxy/ratelimit:3e085e5b
      patch:
        type: StrategicMerge
        value:
          spec:
            template:
              spec:
                containers:
                - imagePullPolicy: IfNotPresent
                  name: envoy-ratelimit
    shutdownManager:
      image: docker.io/envoyproxy/gateway:v1.4.0
  type: Kubernetes

new terminal for port-forward command
(Here when I request a command, it will be interrupted and an error will be reported)

root@VM-0-13-ubuntu:/home/ubuntu# kubectl -n llmaz-system port-forward \
  $(kubectl -n llmaz-system get svc \
    -l gateway.envoyproxy.io/owning-gateway-name=default-envoy-ai-gateway \
    -o name) \
  8080:80
Forwarding from 127.0.0.1:8080 -> 10080
Forwarding from [::1]:8080 -> 10080
Handling connection for 8080
E0705 22:55:51.370524 2273490 portforward.go:424] "Unhandled Error" err="an error occurred forwarding 8080 -> 10080: error forwarding port 10080 to pod fcffd4346fa9cab0a98703c35a3e32e70337eff97caeaa26e9b2ce48e875c989, uid : failed to execute portforward in network namespace \"/var/run/netns/cni-22c22b2c-0315-10f6-a8a1-73e2ff3b19c6\": failed to connect to localhost:10080 inside namespace \"fcffd4346fa9cab0a98703c35a3e32e70337eff97caeaa26e9b2ce48e875c989\", IPv4: dial tcp4 127.0.0.1:10080: connect: connection refused IPv6 dial tcp6 [::1]:10080: connect: connection refused "
error: lost connection to pod

it still don't work 🤔

root@VM-0-13-ubuntu:/home/ubuntu# curl -H "Content-Type: application/json"     -d '{
        "model": "qwen2-0.5b",
        "messages": [
            {
                "role": "system",
                "content": "Hi."
            }
        ]
    }'     http://localhost:8080/v1/chat/completions | jq .
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   165    0     0  100   165      0  27354 --:--:-- --:--:-- --:--:-- 27500
curl: (52) Empty reply from server
root@VM-0-13-ubuntu:/home/ubuntu# kubectl get cm -nllmaz-system

root@VM-0-13-ubuntu:/home/ubuntu# kubectl logs -f envoy-default-default-envoy-ai-gateway-dbec795a-7c749cbb8bzngj9 -nllmaz-system
Defaulted container "envoy" out of: envoy, shutdown-manager
[2025-07-05 14:53:14.576][1][warning][config] [source/extensions/config_subscription/grpc/grpc_subscription_impl.cc:130] gRPC config: initial fetch timed out for type.googleapis.com/envoy.config.cluster.v3.Cluster
[2025-07-05 14:53:29.578][1][warning][config] [source/extensions/config_subscription/grpc/grpc_subscription_impl.cc:130] gRPC config: initial fetch timed out for type.googleapis.com/envoy.config.listener.v3.Listener

[mathetake]

E0705 22:55:51.370524 2273490 portforward.go:424] "Unhandled Error" err="an error occurred forwarding 8080 -> 10080: error forwarding port 10080 to pod fcffd4346fa9cab0a98703c35a3e32e70337eff97caeaa26e9b2ce48e875c989, uid : failed to execute portforward in network namespace "/var/run/netns/cni-22c22b2c-0315-10f6-a8a1-73e2ff3b19c6": failed to connect to localhost:10080 inside namespace "fcffd4346fa9cab0a98703c35a3e32e70337eff97caeaa26e9b2ce48e875c989", IPv4: dial tcp4 127.0.0.1:10080: connect: connection refused IPv6 dial tcp6 [::1]:10080: connect: connection refused "

I believe it's gateway pod is not up and running. could you start from scratch, following the exact same instruction in getting-started doc without changing anything, then modifying your need?

[mathetake]

my guess is mutation webhook is not configured correctly which resulted in the extpric not being present. Maybe making sure that gateway pods are running with three containers including envoy, shutdown-manager and our extproc.

[googs1025]

@mathetake thanks for ask

Using https://aigateway.envoyproxy.io/docs/getting-started/basic-usage does work as expected. I noticed that the pod envoy-default-envoy-ai-gateway-basic-21a9f8f8-5554868f75-4c278 has three containers, but the example I gave only has two.

root@VM-0-13-ubuntu:/home/ubuntu# kubectl get pods -nllmaz-system
NAME                                                              READY   STATUS    RESTARTS   AGE
ai-gateway-controller-659ccb85f-ms6rx                             1/1     Running   0          106m
envoy-default-default-envoy-ai-gateway-dbec795a-7c749cbb8bxkz8n   2/2     Running   0          70m
envoy-gateway-66bb759c9-fz4pd                                     1/1     Running   0          106m
llmaz-controller-manager-76fc68d85d-5h2zc                         1/1     Running   0          106m
llmaz-lws-controller-manager-6969954975-cgsh2                     1/1     Running   0          106m
open-webui-66d8b668b8-ft7tx                                       1/1     Running   0          106m

Is this controlled by the webhook configuration? 🤔

root@VM-0-17-ubuntu:/home/ubuntu# kubectl get mutatingwebhookconfigurations
NAME                                                           WEBHOOKS   AGE
envoy-ai-gateway-gateway-pod-mutator.envoy-ai-gateway-system   1          22m
envoy-gateway-topology-injector.envoy-gateway-system           1          20m
root@VM-0-17-ubuntu:/home/ubuntu# kubectl get mutatingwebhookconfigurations -oyaml envoy-ai-gateway-gateway-pod-mutator.envoy-ai-gateway-system
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  annotations:
    meta.helm.sh/release-name: aieg
    meta.helm.sh/release-namespace: envoy-ai-gateway-system
  creationTimestamp: "2025-07-08T00:50:12Z"
  generation: 2
  labels:
    app.kubernetes.io/managed-by: Helm
  name: envoy-ai-gateway-gateway-pod-mutator.envoy-ai-gateway-system
  resourceVersion: "390765"
  uid: edf0eb3f-70c4-4b09-8359-83e0610846e2
webhooks:
- admissionReviewVersions:
  - v1
  clientConfig:
    caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURPekNDQWlPZ0F3SUJBZ0lVVStnMVVwcDFRdGZwazg3elk1SDIvRVk1NVFVd0RRWUpLb1pJaHZjTkFRRUwKQlFBd0xURUxNQWtHQTFVRUJoTUNRVlV4SGpBY0JnTlZCQU1NRldGcExXZGhkR1YzWVhrdFkyOXVkSEp2Ykd4bApjakFlRncweU5UQTFNakF4TmpRek5USmFGdzB6TlRBMU1UZ3hOalF6TlRKYU1DMHhDekFKQmdOVkJBWVRBa0ZWCk1SNHdIQVlEVlFRRERCVmhhUzFuWVhSbGQyRjVMV052Ym5SeWIyeHNaWEl3Z2dFaU1BMEdDU3FHU0liM0RRRUIKQVFVQUE0SUJEd0F3Z2dFS0FvSUJBUURLTjVZbU1oN1RnR3FOcGVkQzBEV0JXZG4ycE1pSHRDZVJsVGtsdURqSwpsK1plbGVpUjdyb29OVVhjNmdFMDJSQWFSQ0VhTk1TWkwzbTZCa1oxWG9vOTJNdmFidStPUmt3QXBPK09USXZqCk5zWWIzL2Jsc1NUMXFIWEFwbTduODg2RWQ4MENHM0pjemk3QWlvWHNBaFR2K1NvSmVRSnNvS0xlVllWNW01bC8KajR4b0psOWZZK2x6cG1nZGNBTEJtN0ZEckFic0Vnakt3bUZFUUF4VE54V293WkRpQVJXMjFpbzQ1c2FDNDExUwptL1podGhTeERRcHFTelB3WWNYd1IwNHN5WnhHVWV3WXJwSUU1NGhSc004S3dwcU5FWlZuamxhS0Jzc2lFZ0c4Cjk3c3g5d0RiM0hMemVwN0ZTaEt6NExzbGVQQWM4RG12ZFlqbm9vWmF4enNmQWdNQkFBR2pVekJSTUIwR0ExVWQKRGdRV0JCUzlwdUowaSt6S1c0WTNGWTJOdlJLQWIwT05ZekFmQmdOVkhTTUVHREFXZ0JTOXB1SjBpK3pLVzRZMwpGWTJOdlJLQWIwT05ZekFQQmdOVkhSTUJBZjhFQlRBREFRSC9NQTBHQ1NxR1NJYjNEUUVCQ3dVQUE0SUJBUUNaClpnUENRbmFYdC94U0o3b1VGQk1iYTVUTGVxdnpJS1BlRHZTMGlpNjR0WmVRNzVSN25RdlNWdDJRbkdEanB5SisKMDBFUmphN2pXcGpMM0lXaWptRTE5OXZ2NDBaWTRwYWpoQkFMM3k4d1BmNnZSaDhkN1RPNFhtTjdoc2VyMnRjaQpkZW5OR1B4dTFiWDJ0TEU4RkFHTThTVWFyVnk2dmVIZGlVeW9NbEpXcHZqV1lOZ2FWRTVZeC84MzlXbXhSaG5TCjJJT2xqQXNUd2FJa0kwd21zNTFsWlhHUGhSZ0VTOUFvTFB1eXdzZ3E3R2NqSWhZSHBmc28vM0RnUzgvTVRSNUIKaVdxaVhwZ2pENlpPelRReXA4enBuR3pZR2R4U0theGQxSTBMaExUdWF3ZExRK0RTM3p2UjVTNVY1dkxRVVoxcgowVW44RTY4bjdzOEVwbFY4TjZ4bAotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
    service:
      name: ai-gateway-controller
      namespace: envoy-ai-gateway-system
      path: /mutate
      port: 9443
  failurePolicy: Ignore
  matchPolicy: Equivalent
  name: ai-gateway-controller.envoy-ai-gateway-system.svc.cluster.local
  namespaceSelector: {}
  objectSelector:
    matchLabels:
      app.kubernetes.io/managed-by: envoy-gateway
  reinvocationPolicy: Never
  rules:
  - apiGroups:
    - ""
    apiVersions:
    - v1
    operations:
    - CREATE
    resources:
    - pods
    scope: '*'
  sideEffects: None
  timeoutSeconds: 10

[mathetake]

Yes it does

[ludens11]

hi @googs1025, did you manage to get your examples working? I’m running into the same error when extproc isn’t included in the gateway deployment. Wondering if you found a workaround for this 😄