Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add API gateway conformance tests #2403

Open
7 of 8 tasks
AmaliMatharaarachchi opened this issue Jul 23, 2024 · 14 comments · Fixed by #2406
Open
7 of 8 tasks

Add API gateway conformance tests #2403

AmaliMatharaarachchi opened this issue Jul 23, 2024 · 14 comments · Fixed by #2406
Assignees

Comments

@AmaliMatharaarachchi
Copy link
Contributor

AmaliMatharaarachchi commented Jul 23, 2024

Problem

Conformance test setup is failing for apk gateway. If we are adhering to gateway spec, their Conformance test should pass. Currently eventhough we support these CRDs, they are not supporting some gateway spec scenarios listed in https://gateway-api.sigs.k8s.io/guides/simple-gateway/.

Solution

https://docs.google.com/document/d/1fwDWKK_yZ3TXfdcA3vSX4Kg_plcg-JBYoobAiwyudtY/edit#heading=h.35yjoemxqx92

Affected Component

Adapter, Enforcer, router

Version

No response

Implementation

  • support gateway class test cases
  • support gateway test cases
  • Add status, condition support
  • Add gateway address support
  • Add dynamically deploying and deleting gateway for gateway CRs
  • Add reference grant
  • support httproute (without API CR)
  • support httproute test cases

Related Issues

No response

Suggested Labels

No response

@AmaliMatharaarachchi
Copy link
Contributor Author

Progress update:
since gateways will be managed by the adapter, adapter needs to know the deployment namespaces. Therefore configurations needs to be added to setup them. Working on this.

@AmaliMatharaarachchi
Copy link
Contributor Author

Progress update:
new configs should be added in adapter and helm levels to support gateway volumes, env support. working on this.

@AmaliMatharaarachchi
Copy link
Contributor Author

AmaliMatharaarachchi commented Aug 8, 2024

Above are done. Now there is a issue with not envoyproxy xds being updated. I'm checking this now.

Edited:
This is fixed now. #2414

@AmaliMatharaarachchi
Copy link
Contributor Author

AmaliMatharaarachchi commented Aug 8, 2024

Working on enabling ext_authz filter engagement for apis.

subtasks:

  • Enable ext_authz for all httproute calls
  • Make them 200, not engaging any filters
  • Enable them only if an api is attached to the route.

Update: Handed over to Tharsanan as this conflicts with his work

@AmaliMatharaarachchi
Copy link
Contributor Author

AmaliMatharaarachchi commented Aug 13, 2024

Update:
Working on supporting httproute tests.

  • HTTPExactPathMatching
  • HTTPRouteDisallowedKind
  • HTTPRouteCrossNamespace
  • HTTPRouteBackendProtocolH2C
  • HTTPRouteHeaderMatching
  • HTTPRouteHostnameIntersection
  • HTTPRouteInvalidNonExistentBackendRef
  • HTTPRouteInvalidBackendRefUnknownKind

@AmaliMatharaarachchi
Copy link
Contributor Author

AmaliMatharaarachchi commented Aug 14, 2024

Issue : Currently operator created gateway deployments dont get deleted when helm uninstalled.

  • Therefore, checking on deleting them automatically when helm uninstall is called

update: we can use "helm.sh/hook": pre-delete job to execute the manual deletion.
TODO: template it in apk helm

@AmaliMatharaarachchi
Copy link
Contributor Author

AmaliMatharaarachchi commented Aug 16, 2024

Currently working on these tasks

  • Templating feature flag to enable new controller model only when it s enabled
  • Support Backend CRD in new controller

@AmaliMatharaarachchi
Copy link
Contributor Author

AmaliMatharaarachchi commented Aug 26, 2024

Update: started working on

  • Exposing ports in gateway instances
  • establish xds connection from adapter to enforcer
  • Add context extentions to envoy route configs
  • Remove duplicate ext_authz filter creation
  • map invocation request to api configs
  • enabling enforcer filters

@AmaliMatharaarachchi
Copy link
Contributor Author

AmaliMatharaarachchi commented Aug 29, 2024

Update:

  • Encountering no healthy upstream error - due to issue in envoy cluster configs

@AmaliMatharaarachchi
Copy link
Contributor Author

AmaliMatharaarachchi commented Sep 10, 2024

Started working on following

  • merge the code to main branch
  • fix unit test failures in feature branch

@AmaliMatharaarachchi
Copy link
Contributor Author

AmaliMatharaarachchi commented Sep 19, 2024

@AmaliMatharaarachchi
Copy link
Contributor Author

AmaliMatharaarachchi commented Sep 26, 2024

The following tasks are remaining

  • Support all httproute test cases. currently support following. Add API gateway conformance tests  #2403 (comment)
  • engage cucumber and go integration tests to the new flow.
  • add pr test running the conformance tests.
  • add conformance test report in the k8 gateway repo and update in their docs after the upcoming release
  • Slowness in terminating gateway instances
  • intermittently once a gateway cr is deleted, gateway deployment is not getting deleted

@AmaliMatharaarachchi
Copy link
Contributor Author

AmaliMatharaarachchi commented Sep 26, 2024

The following issues were found in PR tests

Issue 1

  • helm deploy - gateway runtime os not getting ready.

Investigations -

Sometimes none of the apis get reconciled in adapter. This is not happening when webhooks are disabled, so it can be narrowed down to api webhooks.

Observations -

router is not ready
No logs of api reconcilations can be found in adapter logs (check for logs like "Reconciling for API " )

Issue 2

  • some newly added integration tests added while working on this separate branch are failing. GRPC api, and backendairatelimit

Issue 3

  • Gateways are not reconciled correctly.

Observations -

Some system apis give 404.
gateway logs have

WARN [api_controller.go:338] - [dp.(*APIReconciler).applyStartupAPIs] [-] Error retrieving ref CRs for API : apk-test-setup-wso2-apk-oauth-api in namespace : apk-integration-test with API UUID : 8b661ab7-f063-4037-8e37-6b2f6714d91a, no gateway available for httpRouteref [apk-test-setup-wso2-apk-oauth-ds-httproute] in namespace :apk-integration-test has not found []

@AmaliMatharaarachchi
Copy link
Contributor Author

AmaliMatharaarachchi commented Sep 30, 2024

Code review suggestions:

  • minimize permissions given in helm-charts/templates/serviceAccount/apk-cluster-role.yaml
  • remove shutdown manager from gateway deployments
  • instead of envoy image in the gateway instance, use router image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: In-Progress
Development

Successfully merging a pull request may close this issue.

1 participant