GEP: Conformance Profiles

[shaneutt]

In the community we've referred to the concept of adding "conformance levels" or "conformance profiles" to Gateway API for quite some time.

NOTE: We've chose conformance profiles over conformance levels in order to not be confused with support levels.

The concept is to have named groupings of conformance tests that can be easily selected for implementations, and correspond with the "category" of implementation. Initially we would like conformance profiles covering these "categories":

• Layer 7
• Layer 4
• Mesh

Profiles should also include options to create conformance testing reports which can be submitted back to the Gateway API project.

Sub Tasks

Beta Give feedback

1. Features for every API in conformance tests #1891
   area/conformance kind/cleanup priority/critical-urgent release-blocker +4
   
2. Initial conformance profiles and reporting prototype #1784
   area/conformance release-blocker +2
   
3. CLI for the experimental version of the conformance test suite #1990
   area/conformance help wanted kind/feature release-blocker triage/accepted +5
   
4. Conformance Profiles: reporting history #2287
   area/conformance kind/feature release-blocker +3
   
5. Conformance Profiles: initial display for reports #2286
   area/conformance kind/feature release-blocker +3
   
6. Conformance Profiles: Remove experimental build tag #2242
   area/conformance priority/important-longterm release-blocker triage/accepted +4
7. conformance profiles: how to handle multiple "modes" in a project #2233
   priority/important-longterm triage/accepted +2
   
8. Add Mode field to the ConformanceReport API #2739
   area/conformance kind/feature +2
   
9. Improve conformance reports structure #2740
   area/conformance +1
   
10. Automatically discover and set the Gateway API version and channel during conformance suite runs #2738
    area/conformance kind/feature +2
    
11. Promote the experimental conformance test suite to standard #2851
    area/conformance +1
    
12. Documentation for the conformance test suite #2322
    area/conformance documentation priority/important-soon +3
    

[riaankleinhans]

@shaneutt this would be a topic of SIG Arch meeting where we discuss Conformance topics. Next meeting is tomorrow.

[shaneutt]

@Riaankl sounds good, I will bring this up in that forum as we progress. However I will not be available for the meeting tomorrow, so I'll intend on a later date meeting which will provide some time for this to grow anyhow.

[shaneutt]

@Riaankl we've had a couple iterations on the relevant GEP 1709 now. This GEP is currently provisional and includes some of our early ideas about implementation. Now that we have some ideas down it's a good time for us to share this in a SIG Arch meeting. I intend to pick an upcoming meeting and attend it: if you think there's a particular meeting that would be ideal let me know.

[shaneutt]

I'm going to try and make it to the March 9th meeting, but I don't know for sure whether I can make it to that one yet. If not that one, then the following one on the 23rd.

[riaankleinhans]

@shaneutt IMO any SIG Arch meeting would be good for this.
To get more focus on your topic I would also suggest to share in SIG Arch mailing list before you attend the meeting

[shaneutt]

Sounds good 👍

[shaneutt]

At this point we've achieved everything that we considered a requirement for our GA release (all TODO issues labeled as release-blocker), so we no longer need to track this for GA.

We will continue working on the relevant sub-issues, there are just no remaining GA blockers.

[mlavacca]

This issue needs to be included as well in the epic, as the last step: #2851