Skip to content

xds: allow "*" to indicate a wildcard subscription #16861

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
htuch merged 4 commits into from
Jun 11, 2021
Merged

xds: allow "*" to indicate a wildcard subscription #16861

Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
b956ea7
xds: allow "*" to indicate a wildcard subscription
markdroth Jun 7, 2021
4cac288
clarify wording about LDS and CDS
markdroth Jun 8, 2021
6c0d5e4
address reviewer comments
markdroth Jun 10, 2021
5933bb6
attempt to fix docs generation
markdroth Jun 11, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
71 changes: 35 additions & 36 deletions docs/root/api-docs/xds_protocol.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,18 +91,18 @@ combinations of two dimensions.

The first dimension is State of the World (SotW) vs. incremental. The SotW approach was the
original mechanism used by xDS, in which the client must specify all resource names it is
interested in with each request (except when making a wildcard request in LDS/CDS), and the server
must return all resources the client has subscribed to in each request (in LDS/CDS). This means
that if the client is already subscribing to 99 resources and wants to add an additional one, it
must send a request with all 100 resource names, rather than just the one new one. And the server
must then respond by sending all 100 resources, even if the 99 that were already subscribed to have
not changed (in LDS/CDS). This mechanism can be a scalability limitation, which is why the
incremental protocol variant was introduced. The incremental approach allows both the client and
server to indicate only deltas relative to their previous state -- i.e., the client can say that
it wants to add or remove its subscription to a particular resource name without resending those
that have not changed, and the server can send updates only for those resources that have changed.
The incremental protocol also provides a mechanism for lazy loading of resources. For details on
the incremental protocol, see :ref:`Incremental xDS <xds_protocol_delta>` below.
interested in with each request, and the server must return all resources the client has
subscribed to in each request (in LDS/CDS). This means that if the client is already subscribing

@dmitri-d dmitri-d Jun 7, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think (in LDS/CDS) part makes the sentence confusing. Perhaps it's worth clarifying that in LDS/CDS there could be multiple resources in a reply, while for RDS/EDS it's always one?

@markdroth markdroth Jun 8, 2021

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be clear, even in RDS and EDS, it's totally valid to have multiple resources per reply; for example, if the client is subscribed to 5 EDS resources and 2 of those resources change, the server can send a response including those two resources. The thing that's special about LDS and CDS is that the server is required to send all resources that the client is subscribed to in every response, even the ones that haven't changed. So if a client is subscribed to 5 LDS resources and 2 of them change, the server must send a response that includes all 5 resources. (This is because the SotW protocol doesn't provide a way for the server to explicitly indicate to the client that a resource has been removed, so the only way to remove an LDS or CDS resource is by not including it in a response. And the removal criteria for RDS and EDS resources is even messier; see the "Deleting Resources" section below.)

Anyway, I've attempted to reword this a bit to clarify it, but I haven't changed the phraseology about the condition this is discussing, because that was already correct.

@dmitri-d dmitri-d Jun 8, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the rewording, I think it's much clearer now.

To be clear, even in RDS and EDS, it's totally valid to have multiple resources per reply

Hmm, not according to the implementation (rds: https://github.com/envoyproxy/envoy/blob/main/source/common/router/rds_impl.cc#L227, eds: https://github.com/envoyproxy/envoy/blob/main/source/common/upstream/eds.cc#L166): updates with more than resource will be dropped.

@markdroth markdroth Jun 9, 2021

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe that Envoy currently creates a separate stream for each resource when using non-aggregated EDS, as per #2943, and the same may be true of RDS. However, this is a limitation of Envoy's implementation, not of the xDS protocol. There is no protocol-level reason why there cannot be more than one resource send per response for any resource type.

Note that when using ADS, Envoy does support multiple resources per response for EDS and RDS. But nothing about that functionality is actually tied to ADS; the exact same thing could be done in non-aggregated streams.

to 99 resources and wants to add an additional one, it must send a request with all 100 resource
names, rather than just the one new one. And the server must then respond by sending all 100
resources, even if the 99 that were already subscribed to have not changed (in LDS/CDS). This

@dmitri-d dmitri-d Jun 7, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto here re: (in LDS/CDS).

@markdroth markdroth Jun 8, 2021

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated wording here too.

mechanism can be a scalability limitation, which is why the incremental protocol variant was
introduced. The incremental approach allows both the client and server to indicate only deltas
relative to their previous state -- i.e., the client can say that it wants to add or remove its
subscription to a particular resource name without resending those that have not changed, and the
server can send updates only for those resources that have changed. The incremental protocol also
provides a mechanism for lazy loading of resources. For details on the incremental protocol, see
:ref:`Incremental xDS <xds_protocol_delta>` below.

The second dimension is using a separate gRPC stream for each resource type vs. aggregating all
resource types onto a single gRPC stream. The former approach was the original mechanism used by
Expand Down Expand Up @@ -287,9 +287,9 @@ stream, the client's initial request on the new stream should indicate the most
seen by the client on the previous stream. Servers may decide to optimize by not resending
resources that the client had already seen on the previous stream, but only if they know that the
client is not subscribing to a new resource that it was not previously subscribed to. For example,
it is generally safe for servers to do this optimization for wildcard LDS and CDS requests, and it
is safe to do in environments where the clients will always subscribe to exactly the same set of
resources.
it is generally safe for servers to do this optimization for LDS and CDS when the only subscription
is a wildcard subscription, and it is safe to do in environments where the clients will always
subscribe to exactly the same set of resources.

An example EDS request might be:

Expand Down Expand Up @@ -427,20 +427,21 @@ names becomes empty, that means that the client is no longer interested in any r
specified type.

For :ref:`Listener <envoy_v3_api_msg_config.listener.v3.Listener>` and :ref:`Cluster <envoy_v3_api_msg_config.cluster.v3.Cluster>` resource
types, there is also a "wildcard" mode, which is triggered when the initial request on the stream
for that resource type contains no resource names. In this case, the server should use
site-specific business logic to determine the full set of resources that the client is interested
in, typically based on the client's :ref:`node <envoy_v3_api_msg_config.core.v3.Node>` identification. Note
that once a stream has entered wildcard mode for a given resource type, there is no way to change
the stream out of wildcard mode; resource names specified in any subsequent request on the stream
will be ignored.
types, there is also a "wildcard" subscription, which is triggered when subscribing to the special

@htuch htuch Jun 9, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we cover how * and on-demand work together in a SotW request in this spec?

@htuch htuch Jun 9, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also how non-explicit * and explicit * subscriptions can be converted or used interchangeably? E.g. can I subscribe to {} but unsubscribe from *?

@markdroth markdroth Jun 10, 2021

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change isn't really about on-demand; it's just about how to represent a wildcard subscription as a discrete name that can coexist with other subscriptions for the same resource type on the same stream. This mechanism is a prerequisite for on-demand CDS, but it's not actually part of the same feature.

I've made some changes in the next paragraph to attempt to clarify the semantics, and I've added some examples to help illustrate.

name "*". In this case, the server should use site-specific business logic to determine the full
set of resources that the client is interested in, typically based on the client's
:ref:`node <envoy_v3_api_msg_config.core.v3.Node>` identification.

For historical reasons, if the initial request on the stream for a given resource type contains no
resource names, the server should treat that as a subscription to "*". A client may later
unsubscribe to "*", or it may later add a subscription to an additional resource name.

@htuch htuch Jun 9, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that we are going to break any existing setup that names a singleton resource *. Seems a bit far fetched, but technically a breaking change. If we instead used something like xds_internal://* here as a name (or something unlikely to collide), we might stand a lower chance of collision but don't feel super strong on this one.

@markdroth markdroth Jun 10, 2021

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that no matter what name we pick here, collisions are possible. I'd rather keep the name short and intuitive.


Client Behavior
"""""""""""""""

Envoy will always use wildcard mode for :ref:`Listener <envoy_v3_api_msg_config.listener.v3.Listener>` and
Envoy will always use wildcard subscriptions for :ref:`Listener <envoy_v3_api_msg_config.listener.v3.Listener>` and
:ref:`Cluster <envoy_v3_api_msg_config.cluster.v3.Cluster>` resources. However, other xDS clients (such as gRPC clients
that use xDS) may specify explicit resource names for these resource types, for example if they
that use xDS) may specify specific resource names for these resource types, for example if they
Comment thread
htuch marked this conversation as resolved.
Outdated
only have a singleton listener and already know its name from some out-of-band configuration.

Grouping Resources into Responses
Expand Down Expand Up @@ -522,13 +523,6 @@ not exist if they have not received the resource. In Envoy, this is done for
<envoy_v3_api_msg_config.endpoint.v3.ClusterLoadAssignment>` resources during :ref:`resource warming
<xds_protocol_resource_warming>`.

Note that this timeout is not strictly necessary when using wildcard mode for :ref:`Listener
<envoy_v3_api_msg_config.listener.v3.Listener>` and :ref:`Cluster <envoy_v3_api_msg_config.cluster.v3.Cluster>` resource types, because
in that case every response will contain all existing resources that are relevant to the
client, so the client can know that a resource does not exist by its absence in the next
response it sees. However, using a timeout is still recommended in this case, since it protects
against the case where the management server fails to send a response in a timely manner.

Note that even if a requested resource does not exist at the moment when the client requests it,
that resource could be created at any time. Management servers must remember the set of resources
being requested by the client, and if one of those resources springs into existence later, the
Expand All @@ -550,10 +544,10 @@ to. For example, if the client had previously been subscribed to resources A and
unsubscribe from B, it must send a new request containing only resource A.

Note that for :ref:`Listener <envoy_v3_api_msg_config.listener.v3.Listener>` and :ref:`Cluster <envoy_v3_api_msg_config.cluster.v3.Cluster>`
resource types where the stream is in "wildcard" mode (see :ref:`How the client specifies what
resource types where the client is using a "wildcard" subscription (see :ref:`How the client specifies what
resources to return <xds_protocol_resource_hints>` for details), the set of resources being
subscribed to is determined by the server instead of the client, so there is no mechanism
for the client to unsubscribe from resources.
subscribed to is determined by the server instead of the client, so the client cannot unsubscribe
from those resources individually; it can only unsubscribe from the wildcard as a whole.

@stevenzzzz stevenzzzz Sep 22, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you elaborate how to effectively unsubscribe resources that are "included " in the "wildcard subscription" when the client unsubscribes "*"?

@krnowak krnowak Sep 22, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd say that the client should be more or less able to track which resources are from wildcard subscription, so it should be able to drop them from its caches when unsubscribing from *.

@markdroth markdroth Sep 22, 2021

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, @krnowak is correct. The client already knows which specific non-wildcard resources it is subscribed to, so when it unsubscribes from *, it can immediately drop all other resources from its cache.


Requesting Multiple Resources on a Single Stream
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Expand Down Expand Up @@ -778,8 +772,13 @@ On reconnect the Incremental xDS client may tell the server of its known
resources to avoid resending them over the network by sending them in
:ref:`initial_resource_versions <envoy_v3_api_field_service.discovery.v3.deltadiscoveryrequest.initial_resource_versions>`.
Because no state is assumed to be preserved from the previous stream, the reconnecting
client must provide the server with all resource names it is interested in. Note that for wildcard
requests (CDS/LDS/SRDS), the request must have no resources in both
client must provide the server with all resource names it is interested in.

Note that for "wildcard" subscriptions (see :ref:`How the client specifies what
resources to return <xds_protocol_resource_hints>` for details), the
request must either specify "*" in the :ref:`resource_names_subscribe
<envoy_v3_api_field_service.discovery.v3.deltadiscoveryrequest.resource_names_subscribe>`
field or (legacy behavior) the request must have no resources in both
:ref:`resource_names_subscribe <envoy_v3_api_field_service.discovery.v3.deltadiscoveryrequest.resource_names_subscribe>` and
:ref:`resource_names_unsubscribe <envoy_v3_api_field_service.discovery.v3.deltadiscoveryrequest.resource_names_unsubscribe>`.

Expand Down