envoy per gateway by arkodg · Pull Request #404 · envoyproxy/gateway

arkodg · 2022-09-21T23:20:45Z

create a xdsIR per gateway
create a infraIR per gateway
use the gateway namespace-name as the key for above IRs
populate the envoy bootstrap config with a service cluster val that matches the IR key
populate the xds server snapshot for each xds request based on the service cluster

Fixes: #349

Signed-off-by: Arko Dasgupta arko@tetrate.io

internal/gatewayapi/translator.go

arkodg · 2022-09-23T02:12:04Z

attaching some test logs done with 2 gateways

2 Gateway resources

$ kubectl get gateway/eg -o yaml
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"gateway.networking.k8s.io/v1alpha2","kind":"Gateway","metadata":{"annotations":{},"name":"eg","namespace":"default"},"spec":{"gatewayClassName":"eg","listeners":[{"name":"http","port":8080,"protocol":"HTTP"}]}}
  creationTimestamp: "2022-09-23T02:00:18Z"
  generation: 1
  name: eg
  namespace: default
  resourceVersion: "1007"
  uid: c1b84856-c1f8-4126-892d-f8762ed9da44
spec:
  gatewayClassName: eg
  listeners:
  - allowedRoutes:
      namespaces:
        from: Same
    name: http
    port: 8080
    protocol: HTTP
status:
  conditions:
  - lastTransitionTime: "2022-09-23T02:00:18Z"
    message: The Gateway has been scheduled by Envoy Gateway
    observedGeneration: 1
    reason: Scheduled
    status: "True"
    type: Scheduled
  - lastTransitionTime: "2022-09-23T02:00:18Z"
    message: No addresses have been assigned to the Gateway
    observedGeneration: 1
    reason: AddressNotAssigned
    status: "False"
    type: Ready
  listeners:
  - attachedRoutes: 1
    conditions:
    - lastTransitionTime: "2022-09-23T02:00:22Z"
      message: Listener is ready
      observedGeneration: 1
      reason: Ready
      status: "True"
      type: Ready
    name: http
    supportedKinds:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute

$ kubectl get gateway/eg-2 -o yaml
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"gateway.networking.k8s.io/v1alpha2","kind":"Gateway","metadata":{"annotations":{},"name":"eg-2","namespace":"default"},"spec":{"gatewayClassName":"eg","listeners":[{"hostname":"www.example-2.com","name":"http-2","port":8080,"protocol":"HTTP"}]}}
  creationTimestamp: "2022-09-23T02:00:53Z"
  generation: 1
  name: eg-2
  namespace: default
  resourceVersion: "1008"
  uid: c6eb611a-09cc-4880-8b33-e8535344e344
spec:
  gatewayClassName: eg
  listeners:
  - allowedRoutes:
      namespaces:
        from: Same
    hostname: www.example-2.com
    name: http-2
    port: 8080
    protocol: HTTP
status:
  conditions:
  - lastTransitionTime: "2022-09-23T02:00:53Z"
    message: The Gateway has been scheduled by Envoy Gateway
    observedGeneration: 1
    reason: Scheduled
    status: "True"
    type: Scheduled
  - lastTransitionTime: "2022-09-23T02:00:53Z"
    message: No addresses have been assigned to the Gateway
    observedGeneration: 1
    reason: AddressNotAssigned
    status: "False"
    type: Ready
  listeners:
  - attachedRoutes: 1
    conditions:
    - lastTransitionTime: "2022-09-23T02:00:53Z"
      message: Listener is ready
      observedGeneration: 1
      reason: Ready
      status: "True"
      type: Ready
    name: http-2
    supportedKinds:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute

2 HTTPRoute resources

$ kubectl get httproute/httpbin -o yaml
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"gateway.networking.k8s.io/v1alpha2","kind":"HTTPRoute","metadata":{"annotations":{},"name":"httpbin","namespace":"default"},"spec":{"hostnames":["www.example.com"],"parentRefs":[{"name":"eg"}],"rules":[{"backendRefs":[{"group":"","kind":"Service","name":"httpbin","port":80,"weight":1}],"matches":[{"path":{"type":"PathPrefix","value":"/"}}]}]}}
  creationTimestamp: "2022-09-23T02:00:22Z"
  generation: 3
  name: httpbin
  namespace: default
  resourceVersion: "1006"
  uid: aafe12cc-9ff7-489c-a9e8-8129aa2c4d80
spec:
  hostnames:
  - www.example.com
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: eg
  rules:
  - backendRefs:
    - group: ""
      kind: Service
      name: httpbin
      port: 80
      weight: 1
    matches:
    - path:
        type: PathPrefix
        value: /
status:
  parents:
  - conditions:
    - lastTransitionTime: "2022-09-23T02:00:22Z"
      message: Route is accepted
      observedGeneration: 1
      reason: Accepted
      status: "True"
      type: Accepted
    controllerName: gateway.envoyproxy.io/gatewayclass-controller
    parentRef:
      group: gateway.networking.k8s.io
      kind: Gateway
      name: eg
  - conditions:
    - lastTransitionTime: "2022-09-23T02:01:00Z"
      message: Route is accepted
      observedGeneration: 2
      reason: Accepted
      status: "True"
      type: Accepted
    controllerName: gateway.envoyproxy.io/gatewayclass-controller
    parentRef:
      group: gateway.networking.k8s.io
      kind: Gateway
      name: eg-2      
$ kubectl get httproute/httpbin-2 -o yaml
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"gateway.networking.k8s.io/v1alpha2","kind":"HTTPRoute","metadata":{"annotations":{},"name":"httpbin-2","namespace":"default"},"spec":{"hostnames":["www.example-2.com"],"parentRefs":[{"name":"eg-2"}],"rules":[{"backendRefs":[{"group":"","kind":"Service","name":"httpbin","port":80,"weight":1}],"matches":[{"path":{"type":"PathPrefix","value":"/"}}]}]}}
  creationTimestamp: "2022-09-23T02:01:22Z"
  generation: 1
  name: httpbin-2
  namespace: default
  resourceVersion: "993"
  uid: 47ff8327-7f01-45b7-8f9b-6236fe2b8a9a
spec:
  hostnames:
  - www.example-2.com
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: eg-2
  rules:
  - backendRefs:
    - group: ""
      kind: Service
      name: httpbin
      port: 80
      weight: 1
    matches:
    - path:
        type: PathPrefix
        value: /
status:
  parents:
  - conditions:
    - lastTransitionTime: "2022-09-23T02:01:22Z"
      message: Route is accepted
      observedGeneration: 1
      reason: Accepted
      status: "True"
      type: Accepted
    controllerName: gateway.envoyproxy.io/gatewayclass-controller
    parentRef:
      group: gateway.networking.k8s.io
      kind: Gateway
      name: eg-2

Port forward to the 2 envoy services

$ kubectl -n envoy-gateway-system port-forward service/envoy-default-eg 8888:8080 &
$ kubectl -n envoy-gateway-system port-forward service/envoy-default-eg-2 8889:8080 &

Can reach www.example.com from service/envoy-default-eg but not www.example-2.com

$ curl --verbose --header "Host: www.example.com" http://localhost:8888/get
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8888 (#0)
> GET /get HTTP/1.1
> Host: www.example.com
> User-Agent: curl/7.64.1
> Accept: */*
> 
Handling connection for 8888
< HTTP/1.1 200 OK
< server: envoy
< date: Fri, 23 Sep 2022 02:06:09 GMT
< content-type: application/json
< content-length: 238
< access-control-allow-origin: *
< access-control-allow-credentials: true
< x-envoy-upstream-service-time: 1
< 
{
  "args": {}, 
  "headers": {
    "Accept": "*/*", 
    "Host": "www.example.com", 
    "User-Agent": "curl/7.64.1", 
    "X-Envoy-Expected-Rq-Timeout-Ms": "15000"
  }, 
  "origin": "10.1.0.12", 
  "url": "http://www.example.com/get"
}
* Connection #0 to host localhost left intact
* Closing connection 0

$ curl --verbose --header "Host: www.example-2.com" http://localhost:8888/get
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8888 (#0)
> GET /get HTTP/1.1
> Host: www.example-2.com
> User-Agent: curl/7.64.1
> Accept: */*
> 
Handling connection for 8888
< HTTP/1.1 404 Not Found
< date: Fri, 23 Sep 2022 02:06:28 GMT
< server: envoy
< content-length: 0
< 
* Connection #0 to host localhost left intact
* Closing connection 0

Can reach www.example-2.com from service/envoy-default-eg-2 but not www.example.com

$ curl --verbose --header "Host: www.example.com" http://localhost:8889/get
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8889 (#0)
> GET /get HTTP/1.1
> Host: www.example.com
> User-Agent: curl/7.64.1
> Accept: */*
> 
Handling connection for 8889
< HTTP/1.1 404 Not Found
< date: Fri, 23 Sep 2022 02:06:35 GMT
< server: envoy
< content-length: 0
< 
* Connection #0 to host localhost left intact
* Closing connection 0
$ curl --verbose --header "Host: www.example-2.com" http://localhost:8889/get
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8889 (#0)
> GET /get HTTP/1.1
> Host: www.example-2.com
> User-Agent: curl/7.64.1
> Accept: */*
> 
Handling connection for 8889
< HTTP/1.1 200 OK
< server: envoy
< date: Fri, 23 Sep 2022 02:06:40 GMT
< content-type: application/json
< content-length: 242
< access-control-allow-origin: *
< access-control-allow-credentials: true
< x-envoy-upstream-service-time: 1
< 
{
  "args": {}, 
  "headers": {
    "Accept": "*/*", 
    "Host": "www.example-2.com", 
    "User-Agent": "curl/7.64.1", 
    "X-Envoy-Expected-Rq-Timeout-Ms": "15000"
  }, 
  "origin": "10.1.0.13", 
  "url": "http://www.example-2.com/get"
}
* Connection #0 to host localhost left intact
* Closing connection 0

arkodg · 2022-09-23T23:21:39Z

reviewers please note that this PR doesnt handle delete of Gateway resources, will raise an issue to track it once this PR is in, and will fix be fixed in a follow up commit.

codecov-commenter · 2022-09-26T23:14:41Z

Codecov Report

❌ Patch coverage is 70.22901% with 39 lines in your changes missing coverage. Please review.
✅ Project coverage is 60.13%. Comparing base (3ef5627) to head (41dc3c1).
⚠️ Report is 3641 commits behind head on main.

Files with missing lines	Patch %	Lines
internal/gatewayapi/runner/runner.go	0.00%	13 Missing ⚠️
internal/xds/server/runner/runner.go	0.00%	8 Missing ⚠️
internal/xds/translator/runner/runner.go	45.45%	3 Missing and 3 partials ⚠️
internal/message/types.go	0.00%	4 Missing ⚠️
internal/cmd/xdstest.go	0.00%	3 Missing ⚠️
internal/infrastructure/kubernetes/infra.go	0.00%	0 Missing and 3 partials ⚠️
internal/infrastructure/kubernetes/service.go	75.00%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #404      +/-   ##
==========================================
+ Coverage   60.00%   60.13%   +0.12%     
==========================================
  Files          40       40              
  Lines        4333     4337       +4     
==========================================
+ Hits         2600     2608       +8     
+ Misses       1580     1579       -1     
+ Partials      153      150       -3

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

danehans · 2022-09-27T18:32:25Z

xref: #380

danehans · 2022-09-27T18:34:31Z

xref: #382

arkodg · 2022-09-27T19:58:14Z

@danehans I cherry-picked #435 on top of this PR and CI now passes

* create a xdsIR per gateway * create a infraIR per gateway * use the gateway namespace-name as the key for above IRs * populate the envoy bootstrap config with a node id that matches the IR key * populate the xds server snapshot for each xds request based on the node id Fixes: envoyproxy#349 Signed-off-by: Arko Dasgupta <arko@tetrate.io>

Signed-off-by: Arko Dasgupta <arko@tetrate.io>

Signed-off-by: danehans <daneyonhansen@gmail.com>

Signed-off-by: Arko Dasgupta <arko@tetrate.io>

danehans · 2022-09-28T00:45:08Z

I tested this PR with #438 and the additional conformance tests pass.

danehans · 2022-09-28T00:55:57Z

internal/crypto/certgen.go


 	// DefaultEnvoyDNSPrefix defines the default Envoy DNS prefix.
-	DefaultEnvoyDNSPrefix = config.EnvoyServiceName
+	DefaultEnvoyDNSPrefix = "*"


Created #440 to develop a more secure control plane auth solution.

danehans · 2022-09-28T01:05:43Z

internal/infrastructure/kubernetes/deployment.go

+func expectedDeploymentName(proxyName string) string {
+	return fmt.Sprintf("%s-%s", config.EnvoyDeploymentPrefix, proxyName)
+}
+


This reverts commit 231e4fb. Signed-off-by: danehans <daneyonhansen@gmail.com>

arkodg requested a review from a team as a code owner September 21, 2022 23:20

arkodg marked this pull request as draft September 21, 2022 23:21

arkodg added this to the 0.2.0-rc2 milestone Sep 21, 2022

skriss reviewed Sep 22, 2022

View reviewed changes

internal/gatewayapi/translator.go Outdated Show resolved Hide resolved

danehans mentioned this pull request Sep 22, 2022

Adds Support for Control Plane TLS #208

Merged

arkodg force-pushed the ir-per-gateway branch from 5e8ec0e to 3435da1 Compare September 22, 2022 22:34

arkodg requested a review from skriss September 23, 2022 02:12

arkodg marked this pull request as ready for review September 23, 2022 02:13

arkodg requested a review from a team September 23, 2022 19:33

arkodg force-pushed the ir-per-gateway branch from 1a0d520 to 15af025 Compare September 23, 2022 23:34

arkodg mentioned this pull request Sep 26, 2022

Handle IR deletes #423

Closed

arkodg force-pushed the ir-per-gateway branch from 15af025 to c171031 Compare September 26, 2022 22:38

arkodg force-pushed the ir-per-gateway branch 2 times, most recently from 5f5f281 to f79369d Compare September 27, 2022 00:46

This was referenced Sep 27, 2022

Updates Infra IR to Support Multiple Envoy Services #389

Closed

HTTPRouteListenerHostnameMatching Conformance Test Fails #434

Closed

arkodg force-pushed the ir-per-gateway branch from f79369d to 5bd432b Compare September 27, 2022 19:47

Arko Dasgupta and others added 10 commits September 27, 2022 14:48

append proxy name to infra resources

7374ff7

Signed-off-by: Arko Dasgupta <arko@tetrate.io>

fix provider infra service and deployment names

214b980

Signed-off-by: Arko Dasgupta <arko@tetrate.io>

1:1 b/w service-cluster and irKey (gwNs-gwName)

20c1697

Signed-off-by: Arko Dasgupta <arko@tetrate.io>

detect conflicts per gateway instead of across all

e9a52f4

Signed-off-by: Arko Dasgupta <arko@tetrate.io>

infra found ports per gateway

1e25b90

Signed-off-by: Arko Dasgupta <arko@tetrate.io>

rm ()

eeb00c8

Signed-off-by: Arko Dasgupta <arko@tetrate.io>

Rebased and Fixed Test Failures

e9a56e8

Signed-off-by: danehans <daneyonhansen@gmail.com>

Updates Envoy Cert as Wildcard Hostname

96b4693

Signed-off-by: danehans <daneyonhansen@gmail.com>

sort to add stablity in translated o/p

886fba3

Signed-off-by: Arko Dasgupta <arko@tetrate.io>

lint

41dc3c1

Signed-off-by: Arko Dasgupta <arko@tetrate.io>

arkodg force-pushed the ir-per-gateway branch from 5bd432b to 41dc3c1 Compare September 27, 2022 21:48

This was referenced Sep 28, 2022

Refine Control Plane Auth #440

Closed

Hash the Names of Managed Infra Resources #441

Closed

danehans approved these changes Sep 28, 2022

View reviewed changes

arkodg merged commit 231e4fb into envoyproxy:main Sep 28, 2022

This was referenced Sep 28, 2022

Adds Gateway Watch to HTTPRoute Controller #403

Merged

Adds GatewayClass Watch to Gateway Controller #412

Closed

This was referenced Sep 28, 2022

Add support for multiple gateways #380

Closed

Xds Translator Incorrectly Creates Listener #382

Closed

danehans mentioned this pull request Sep 28, 2022

Revert "envoy per gateway" #445

Closed

danehans added a commit that referenced this pull request Sep 28, 2022

Revert "envoy per gateway (#404)"

fffe65a

This reverts commit 231e4fb. Signed-off-by: danehans <daneyonhansen@gmail.com>

arkodg mentioned this pull request Mar 27, 2023

Merge Gateways on to a single EnvoyProxy fleet #1230

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

envoy per gateway#404

envoy per gateway#404
arkodg merged 11 commits intoenvoyproxy:mainfrom
arkodg:ir-per-gateway

arkodg commented Sep 21, 2022 •

edited

Loading

Uh oh!

Uh oh!

arkodg commented Sep 23, 2022

Uh oh!

arkodg commented Sep 23, 2022

Uh oh!

codecov-commenter commented Sep 26, 2022 •

edited by codecov bot

Loading

Uh oh!

danehans commented Sep 27, 2022

Uh oh!

danehans commented Sep 27, 2022

Uh oh!

arkodg commented Sep 27, 2022

Uh oh!

danehans commented Sep 28, 2022

Uh oh!

danehans Sep 28, 2022

Uh oh!

danehans Sep 28, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

arkodg commented Sep 21, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

arkodg commented Sep 23, 2022

Uh oh!

arkodg commented Sep 23, 2022

Uh oh!

codecov-commenter commented Sep 26, 2022 • edited by codecov bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

danehans commented Sep 27, 2022

Uh oh!

danehans commented Sep 27, 2022

Uh oh!

arkodg commented Sep 27, 2022

Uh oh!

danehans commented Sep 28, 2022

Uh oh!

danehans Sep 28, 2022

Choose a reason for hiding this comment

Uh oh!

danehans Sep 28, 2022

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

arkodg commented Sep 21, 2022 •

edited

Loading

codecov-commenter commented Sep 26, 2022 •

edited by codecov bot

Loading