Run Conformance Tests on implementations in CI

[arkodg]

What would you like to be added:
Run the entire conformance test suite when a PR is raised on Gateway API implementations, but run it as a non blocking step.

Why this is needed:
As a developer writing conformance tests, I do not how to test my changes.

[arkodg]

Ive been creating my k8s cluster and running this script
hack/run-conformance-envoy-gateway.sh

#!/usr/bin/env bash

set -euo pipefail

# Deploy latest version of Envoy Gateway and Gateway API CRDs
kubectl apply -f https://github.com/envoyproxy/gateway/releases/download/latest/install.yaml

# Wait for Envoy Gateway to be ready
kubectl wait --timeout=5m -n gateway-system deployment/gateway-api-admission-server --for=condition=Available
kubectl wait --timeout=5m -n envoy-gateway-system deployment/envoy-gateway --for=condition=Available
kubectl wait --timeout=5m -n gateway-system job/gateway-api-admission --for=condition=Complete

# Deploy GatewayClass
kubectl apply -f - <<EOF
kind: GatewayClass
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
  name: envoy-gateway
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
EOF

# Run conformance test
go test -v ./conformance/... --run TestConformance/HTTPRouteHostnameIntersection  --gateway-class=envoy-gateway --debug=true

would help the community if something like this is incorporated into the CI

Sucessful conformance tests take around ~5min to finish today so running multiple in parallel on multiple implementations shouldnt slow down the CI too much imho.

A suggestion for acceptable criteria could be - implementation must be passing all conformance tests for it to be added into the CI

[arkodg]

another suggestion is to specify implementation specific steps in https://gateway-api.sigs.k8s.io/concepts/conformance/?h=conf#3-conformance-tests and linking this page to the PR template #1733 making it a one click discovery (CI would be the ideal auto magic way)

[shaneutt]

So interestingly enough we were already planning on this and actively working on this (the https://github.com/kong/blixt project (which is being donated to Kubernetes SIGs) is related as we intend to plug that implementation into CI).

You can find the relevant announcement #1706, but after searching I don't think we had a tracking issue for it so this works as the tracking issue, thank you. 👍

[arkodg]

@shaneutt I completely missed that discussion, thanks for surfacing it. This is great that Blixt will be incorporated into the CI. My follow up questions are

• Will Blixt support L7 APIs (HTTPRoute) ?
• If yes, any rough estimate timelines on when this will happen

The velocity of contributions for conformance related PRs should increase once ^ happens.
Speaking from personal experience, after introducing a regression in the conformance test suite, im hesitant to add more tests or review PRs without knowing that the PR has been tested with a known implementation

[shaneutt]

* Will Blixt support L7 APIs (HTTPRoute) ?
* If yes, any rough estimate timelines on when this will happen

Currently we don't have plans to do this, though it's not the first time it's been asked.

For right this moment, we're going to focus on Layer4 as that will get a fair amount of the ground work covered, while also having the side benefit of driving L4 conformance so that we can work on getting UDPRoute and TCPRoute matured to BETA sooner. Once we have that groundwork done, we could definitely have a discussion about the possibilities for L7 support. Just note that we're using the Linux kernel for the dataplane in Blixt because we consider this to be vendor neutral and we absolutely want to stay away from playing favorites with well established base tech like envoy or nginx, and that makes adding L7 support potentially costly.

If you're interested in pursuing this I would definitely recommend contributing to the project and checking in on our friday [SIG Network] Gateway API Code Jam meetings, which are currently mostly focused on the topic of Blixt.

[danehans]

It's great to see that Blixt will help drive L4 adoption. I know Blixt is meant to be used for testing only, so it seems like a good fit for L4 conformance testing. However, I'm concerned that most implementations are L7 with a focus on supporting the Beta APIs. What existing L4 implementations is Blixt supporting? I can't tell from this list.

[shaneutt]

It's great to see that Blixt will help drive L4 adoption. I know Blixt is meant to be used for testing only, so it seems like a good fit for L4 conformance testing. However, I'm concerned that most implementations are L7 with a focus on supporting the Beta APIs. What existing L4 implementations is Blixt supporting? I can't tell from this list.

Not sure I understand the question perfectly, but my take of the question was that you're asking what other L4 implementations there are today. If that's the case, we don't know exactly and that's actually something we're trying to solve by driving forward L4 conformance tests (with Blixt as a catalyst) and also the conformance profiles work which will include reporting back to upstream.

[danehans]

Thanks for the feedback @shaneutt. More specifically, why not prioritize L7 over L4 since most, if not all, current implementations are L7?

[shaneutt]

Thanks for the feedback @shaneutt. More specifically, why not prioritize L7 over L4 since most, if not all, current implementations are L7?

The main reason is that HTTPRoute already has momentum: there are already several L7 implementations helping to push conformance forward, in fact 99% of our conformance tests are L7 today because of those efforts. HTTPRoute is already tracking for a GA release this year. Focusing on L4 allows us to help push L4 to BETA by helping to pave the road for conformance as no implementation has really stepped up to do this yet and as a side effect we'll still get CI for exercising some of the common functionality (e.g. GatewayClass, Gateway, ReferenceGrant, e.t.c.).

A prominent secondary reason is that we wanted to provide vendor neutrality with our tech choice: we didn't want the "reference" implementation to end up being a specific vendor implementation, nor did we want it to be a custom one bound to envoy, or nginx. Being bound to native Linux functionality seemed very neutral to us and was a precondition of acceptance of a testing/reference implementation when proposed. As stated above we would still consider adding L7 functionality in time if we can navigate it neutrally, as the benefit of exercising the L7 conformance tests on each release is definitely a desire.

[danehans]

@shaneutt thanks for the additional feedback. From my understanding, L7 conformance test coverage was driven by affiliates implementing L7 proxies, load balancers, etc. I would have expected that the approach for L4 would follow this approach. Instead of pivoting to L4 due to L7-specific implementation details, I ask that the community consider an approach for running conformance tests on L7 implementations. If this has already been thoroughly discussed, do you mind sharing a link where I can learn more?

[shaneutt]

From my understanding, L7 conformance test coverage was driven by affiliates implementing L7 proxies, load balancers, etc.

Correct.

I would have expected that the approach for L4 would follow this approach.

Same for us maintainers, and that's part of what drove this. In fact, now because of the momentum started by the Blixt project we're starting to see some implementations out there coming forward wanting to join in on conformance, such as STUNner (https://github.com/l7mp/stunner) which was presented to us some months ago at a community sync and is an implementer of UDPRoute with a very special use case.

If this has already been thoroughly discussed, do you mind sharing a link where I can learn more?

Multiple conversations to build consensus almost a year ago prior to and during Kubecon NA. Check out the sync entries and community recordings from around that time and up until now.

Instead of pivoting to L4 due to L7-specific implementation details, I ask that the community consider an approach for running conformance tests on L7 implementations.

Given the wording "instead" and "pivot" it seems you're suggesting we stop current efforts which I don't see a compelling reason to do, particularly since we've even covered the desire and possibility to move into L7 space with our current direction. Perhaps now is a good a time as any to explore that direction further?

[danehans]

@shaneutt thanks for the additional clarity. To clarify, I'm not against running L4 conformance tests. My point is that the project should consider how to support L7 implementations, even though none are based on Linux from my knowledge.

The use case I'm trying to support is:

1. As a consumer interested in using a Gateway API implementation, I go to the implementations page to see the choices I have.
2. I have a ton of implementations to choose from. Some say ".. preview", "alpha", etc.
3. I would like to select one of the several implementations that are listed as "beta". Does this mean that the implementation supports the beta APIs?
4. How do I know if the implementation supports Gateway API core or extended features? Since this information is not surfaced, I need to dive into the code of the implementations that I'm interested in to dig further.

[robscott]

@danehans Agree that this is unnecessarily confusing today. On 2) so far that's represented the implementation's stability/support level and is populated/controlled by the owner of the implementation. I think most of these concerns are going to be resolved with #1709. Despite the name of "Conformance Profiles", the GEP has the larger goal of centralized conformance reporting which will likely be surfaced on the implementations page. How that will look is TBD. I think we could definitely use some help driving that forward so let us know if you're interested.

[danehans]

@robscott thanks for the feedback. I may be conflating this issue with the conformance profiles GEP, so I appreciate the clarification. I'm happy to help. Let me dive into that GEP to better understand the details.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten