Skip to content
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
74 changes: 44 additions & 30 deletions apis/v1beta1/gateway_types.go
Original file line number Diff line number Diff line change
Expand Up @@ -85,38 +85,52 @@ type GatewaySpec struct {
//
// Port and protocol combinations not listed above are considered Extended.
//
// An implementation MAY group Listeners by Port and then collapse each
// group of Listeners into a single Listener if the implementation
// determines that the Listeners in the group are "compatible". An
// implementation MAY also group together and collapse compatible
// Listeners belonging to different Gateways.
//
// For example, an implementation might consider Listeners to be
// compatible with each other if all of the following conditions are
// met:
//
// 1. Either each Listener within the group specifies the "HTTP"
// Protocol or each Listener within the group specifies either
// the "HTTPS" or "TLS" Protocol.
//
// 2. Each Listener within the group specifies a Hostname that is unique
// within the group.
//
// 3. As a special case, one Listener within a group may omit Hostname,
// in which case this Listener matches when no other Listener
// matches.
//
// If the implementation does collapse compatible Listeners, the
// hostname provided in the incoming client request MUST be
// matched to a Listener to find the correct set of Routes.
// The incoming hostname MUST be matched using the Hostname
// field for each Listener in order of most to least specific.
// That is, exact matches must be processed before wildcard
// A Gateway's Listeners are considered "compatible" if:
//
// 1. The implementation can serve them in compliance with the Addresses
// requirement that all Listeners are available on all assigned
// addresses.
// 2. No Listeners sharing the same Port share the same Hostname value,
// including the empty value, if this would prevent the implementation
// from matching an inbound request to a specific Listener.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One advantage of the previous wording here was that it was an example of what an implementation might consider compatible. In this case, the wording might prevent support for multiple protocols on the same port, which at least some vendors can support. (I don't think this explicitly prevents it, but it's not obvious if it would be compatible).

Copy link
Contributor Author

@rainest rainest Aug 15, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we not think the examples below cover that? They do intentionally include that case and indicate these are compatible if you can support it.

From the second point in my earlier self-review, would we want to instead use broader, but more direct language? "The implementation can unambiguously match a request to a single Listener." instead, for example. That's ultimately what we're going for, but quite abstract, so maybe harder to understand.

There is Hostname-specific language later, so we can omit that here if we want:

Implementations using the Hostname value to select between same-Port Listeners MUST match inbound request hostnames from the most specific to least specific...

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reading over this again, it may be better to introduce the concept of distinct first, and then use that as the basis for compatible - I think that it makes what we're doing easier to understand.

Suggested change
// A Gateway's Listeners are considered "compatible" if:
//
// 1. The implementation can serve them in compliance with the Addresses
// requirement that all Listeners are available on all assigned
// addresses.
// 2. No Listeners sharing the same Port share the same Hostname value,
// including the empty value, if this would prevent the implementation
// from matching an inbound request to a specific Listener.
// A Gateway's Listeners must be _distinct_.
//
// The key information that is used to determine if a set of Listeners is distinct
// differs based on if the protocol considers the `hostname` field relevant.
//
// For protocols `HTTP`, `HTTPS`, and `TLS`, Listeners must be distinct in terms of
// having a unique combination of `port`, `protocol`, and `hostname`. The empty
// hostname is also a unique value, so there can only be one Listener with an
// empty hostname in a set of Listeners when using these protocols.
// (This Listener then functions as a fallback, where any undefined hostname will
// match).
//
// For protocols `TCP` and `UDP`, Listeners must be distinct in terms of having
// a unique combination of `port` and `protocol` only. Hostname fields in these
// Listeners must be ignored.
//
// Implementation-specific protocols MAY choose if `hostname` is relevant to their
// protocol or not, and use either of these rules on that basis.
//
// A Gateway's Listeners are considered "compatible" if all the Listeners are distinct
// _and_ the implementation can serve them in compliance with the Addresses
// requirement that all Listeners are available on all assigned addresses.

I think that this sort of language will make the later changes easier as well, in that it makes it more clear why combinations may be incompatible.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we indeed want to codify specific compatibility cases or do we want to leave most of the extended space open to implementation capabilities? If we do want to codify the hostname/port/protocol relationships, we'd use something like @youngnick's suggestion, dispense with the separate examples, and just add "unless the implementation cannot serve otherwise compatible Listeners on the same addresses" as a final caveat.

I like the broad "what the implementation can route" version for its brevity, and think the select non-normative examples after ground that rule in more practical terms, without having to exhaustively list all cases.

Exhaustively listing compatible cases feels like it'd be more difficult to expand. We have to cover an N-dimensional matrix with complex rules (sometimes a dimension isn't relevant), so adding a new Listener field would create lots of cases.

We're hindered somewhat by needing to place the spec in struct comments--the more detailed rules would probably fit better in a breakout page, but AFAIK we don't use those in the reference.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tbh I think this is a case where it's better to be more specific for the protocols defined in the spec - I tried to leave some space for the use of implementation-specific protocols, but I don't anticipate us adding many more protocol options, or other options like hostname.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Trying to write an exhaustive set of rules from the angle of Listener properties is proving quite tricky. For example,

For protocols HTTP, HTTPS, and TLS, Listeners must be distinct in terms of having a unique combination of port, protocol, and hostname.

probably breaks when you have HTTP:443:example.com and HTTPS:443:example.com. Those are unique (they differ by protocol) but I expect most implementations couldn't actually handle it, and it'd violate a rule (insofar as the example is halfway a rule) from the original:

Either each Listener [using the same port] specifies the “HTTP” Protocol or each Listener within the group specifies either the “HTTPS” or “TLS” Protocol.

Trying to hew closer to the original and focus around sharing Port values gets confusing because transport (or the use of TLS) is implied but not part of the spec.

Ultimately, do we have goals for compatibility other than ensuring that an implementation can serve Listeners on the same address and match a request to exactly one Listener? If we do, and expect that compatibility will vary across implementations anyway, do we have a reason not to make that the actual rule?

Again, I understand the value of having practical examples--that's why this revision still has them, after this section. Do those not provide the clarity around common cases we expect to see?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, the example you gave makes sense, but I think that in the suggestion I made, that comes under the "implementations couldn't actually serve this" rule.

You're right that, at its heart, the rule is "implementations can serve all Listeners on the same address, and match a request to exactly one Listener". We can make that the rule, but I think that without something like what I wrote, we will end up with big incompatibilities between implementations. Just having examples is, in my experience, not enough.

To put this another way, for the included Protocols, I think that we should be as specific as we can manage, with a rule like "if you can't serve a distinct set of listeners, then it's not compatible" to cover the other edge cases. The idea behind adding "distinct" as a thing is to try to make it clear what we mean by "match a request to only one Listener" - because it really needs to be "match a request to only one Listener, with no re-entry to processing if that Listener's Routes don't work out." (Since that's what started this conversation to begin with).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Status quo is that we already only have examples (and effectively only one at that).

What incompatibilities do we expect in practice, and how bad will those be?

I grant that lack of upper bounds may allow some implementation to say "we do deep packet inspection, you can have HTTP, HTTPS, and TLS Listeners with the same Hostname, and a no-Hostname TCP Listener all on the same Port", and that this won't be portable. I think that's fine so long as there's a defined way for an implementation to report that they can't support that configuration, and expect that demand for it isn't so great that mixed support isn't going to break the ecosystem.

If there are areas where we're concerned about compatibility, we should be adding more SHOULD lower bounds after the existing MUSTs.

I could write something to the effect of:

+       // HTTPS and TLS Listeners MAY be compatible with each other if they share
+       // the same Port, They MUST NOT be compatible with each other if they share
+       // the same Port and same Hostname.
+       //
+       // HTTP Listeners MUST NOT be compatible with TLS, HTTPS, or TCP Listeners
+       // with the same Port. They MUST be compatible with HTTP Listeners that
+       // share the same port if they do not share Hostname values.
+       //
+       // HTTP, HTTPS, TLS, and TCP Listeners MAY be compatible with UDP Listeners
+       // with the same Port.
+       //
+       // Implementations MUST NOT consider the Hostname field for TCP and UDP
+       // Listeners when evaluating compatibility.
+
+      ...

but it's still likely to be incomplete and long despite, and I do think trying to limit the spec complexity is valuable.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree, I thought that's what I was trying to do by saying "Listeners must be distinct", and defining rules for distinct.

I think that this is now at the point where we need to talk about it in the meeting though - since I don't understand how the suggestion I supplied doesn't make the spec clearer while managing the spec complexity.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dd27869 rewords the second rule.

5af8b65 adds cascade matching language as a SHOULD elsewhere. IMO this would be a breaking change if it's a MUST. I'd need to review further to see if there was existing (if unclear) language that implied that, but I know there are at least no existing compatibility tests for it (we'd have failed it if so).

AFAIK we don't have much information from other implementations re their ability to cascade or not--I know it isn't Kong's routing behavior and isn't easily avoided.

The latter also can't be a compatibility rule since it doesn't block accepting both Listeners.

//
// Compatible combinations in Extended support are expected to vary across
// implementations. A combination that is compatible for one implementation
// may not be compatible for another.
//
// If this field specifies multiple Listeners that are not compatible, the
// implementation MUST raise a true "Conflicted" condition in the Listener
// Status.
//
// Implementations MAY choose to still accept a Gateway with conflicted
// Listeners if they accept a partial Listener set that contains no
// incompatible Listeners. They MUST set a "ListenersNotValid" condition
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: ListenersNotValid is a reason that can be used with "Accepted", generally to set the condition to "False". I'm not sure what we'd recommend in the case where Listeners were not valid and the Gateway was accepted.

Copy link
Contributor Author

@rainest rainest Aug 15, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We say it can be used when Accepted is True. From the first point in my self-review, this is indeed a confusing case, but probably one that's intentionally ambiguous because we expect it to vary so much between implementations.

If we want to leave that as-is, the change here is just to say (here) that implementations must set this, whereas previously it wasn't a strict requirement, and wasn't obvious unless you looked through the reason comments. Even if it changes, I think we should mention something here.

Changing the current vague guidelines to something more formal would be a significant breaking change. Between that and the expected variance across vendors, my vote would be to defer it to post GA, when we have more practical experience regarding which approaches are in use and their pros and cons.

// the Gateway Status when the Gateway contains incompatible Listeners
// whether or not they accept the Gateway.
Comment on lines +107 to +111
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the only reason we'd want to allow this is if a Gateway was already programmed with compatible listeners and then invalid/incompatible one(s) were added. It gets really tricky to represent this state. I may be remembering wrong here, but I think on GKE we will leave "Programmed" set to "True" with the last generation it was Programmed, but I don't think we have any clear guidance for these kinds of partially valid states in the spec.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

//
// For example, the following Listener scenarios may be compatible
// depending on implementation capabilities:
//
// 1. Multiple Listeners with the same Port that all use the "HTTP"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this example is a core feature

I wonder if "may be compatible" is applicable to it

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Compatibility is weird because it ends up being about implementation capabilities rather than rules in the spec. If your implementation isn't compatible with something the core requires, it just can't implement the spec.

We could bring this further out of core by making it something outside the language above and mentioning specific ports, e.g. "Multiple Listeners with Port "9999" that all use the "HTTP" protocol (and similar for the other examples, just to make them consistent).

We could also separate examples that are within core or outside it, but it wouldn't be great for brevity.

// Protocol that all have unique Hostname values.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know this is just an example, so it may not be critical here, but it might be helpful to define whether *.example.com and foo.example.com would be considered unique here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

8a9acfb adds this to the following Hostname precedence section. Do you think that works? It's not explicitly saying they are considered distinct, but it is using them as distinct values in the context where it matters.

// 2. Multiple Listeners with the same Port that use either the "HTTPS" or
// "TLS" Protocol that all have unique Hostname values.
// 3. A mixture of "TCP" and "UDP" Protocol Listeners, where no Listener
// with the same Protocol has the same Port value.
//
// An implementation that cannot serve both TCP and UDP listens on the same
// address, or cannot mix HTTPS and generic TLS listens on the same port
// would not consider those cases compatible.
//
// Implementations using the Hostname value to select between same-Port
// Listeners MUST match inbound request hostnames from the most specific
// to least specific Hostname values to find the correct set of Routes.
// Exact matches must be processed before wildcard matches, and wildcard
// matches must be processed before fallback (empty Hostname value)
// matches.
//
// If this field specifies multiple Listeners that have the same
// Port value but are not compatible, the implementation must raise
// a "Conflicted" condition in the Listener status.
// Implementations MAY collapse separate Gateways onto a single set of
// Addresses if the Listeners across all Gateways are compatible.
//
// Support: Core
//
Expand Down
112 changes: 68 additions & 44 deletions config/crd/experimental/gateway.networking.k8s.io_gateways.yaml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading