config: full delta xDS (including ADS) support

api/xds_protocol.rst

@@ -367,6 +367,8 @@ An example minimal ``bootstrap.yaml`` fragment for ADS configuration is:
     admin:
       ...
 
+.. _xds_protocol_delta:
+
 Incremental xDS
 ~~~~~~~~~~~~~~~
 

docs/root/configuration/overview/v2_overview.rst

@@ -570,6 +570,32 @@ to
 with the effect that the LDS stream will be directed to *some_ads_cluster* over
 the shared ADS channel.
 
+.. _config_overview_v2_delta:
+
+Delta endpoints
+---------------
+
+The REST, filesystem, and original gRPC xDS implementations all deliver "state of the world" updates:
+every CDS update must contain every cluster, with the absence of a cluster from an update implying
+that the cluster is gone. For Envoy deployments with huge amounts of resources and even a trickle of
+churn, these state-of-the-world updates can be cumbersome.
+
+As of 1.12.0, Envoy supports a "delta" variant of xDS (including ADS), where updates only contain
+resources added/changed/removed. Delta xDS is a gRPC (only) protocol. Delta uses different
+request/response protos than SotW (DeltaDiscovery{Request,Response}); see
+:repo:`discovery.proto <api/envoy/api/v2/discovery.proto>`. Conceptually, delta should be viewed as
+a new xDS transport type: there is static, filesystem, REST, gRPC-SotW, and now gRPC-delta.
+(Envoy's implementation of the gRPC-SotW/delta client happens to share most of its code between the
+two, and something similar is likely possible on the server side. However, they are in fact
+incompatible protocols.
+:ref:`The specification of the delta xDS protocol's behavior is here <xds_protocol_delta>`.)
+
+To use delta, simply set the api_type field of your
+:ref:`ApiConfigSource <envoy_api_msg_core.ApiConfigSource>` proto(s) to DELTA_GRPC.
+That works for both xDS and ADS; for ADS, it's the api_type field of
+:ref:`DynamicResources.ads_config <envoy_api_field_config.bootstrap.v2.Bootstrap.dynamic_resources>`,
+as described in the previous section.
+
 .. _config_overview_v2_mgmt_con_issues:
 
 Management Server Unreachability

docs/root/intro/arch_overview/operations/dynamic_configuration.rst

@@ -88,3 +88,15 @@ The :ref:`secret discovery service (SDS) <config_secret_discovery_service>` laye
 by which Envoy can discover cryptographic secrets (certificate plus private key, TLS session
 ticket keys) for its listeners, as well as configuration of peer certificate validation logic
 (trusted root certs, revocations, etc).
+
+Aggregated xDS ("ADS")
+-----------------------------
+
+EDS, CDS, etc. are each separate services, with different REST/gRPC service names, e.g. StreamListeners, StreamSecrets. For users looking to enforce the order in which resources of different types reach Envoy, there is aggregated xDS, a single gRPC service that carries all resource types in a single gRPC stream. (ADS is only supported by gRPC). :ref:`More details about ADS <config_overview_v2_ads>`.
+
+.. _arch_overview_dynamic_config_delta:
+
+Delta gRPC xDS
+-----------------------------
+
+Standard xDS is "state-of-the-world": every CDS update must contain every cluster, with the absence of a cluster from an update implying that the cluster is gone. As of 1.12.0, Envoy supports a "delta" variant of xDS (including ADS), where updates only contain resources added/changed/removed. Delta xDS is a new protocol, with request/response protos different from SotW; existing control plane servers will need support added. :ref:`More details about delta <config_overview_v2_delta>`.

docs/root/intro/version_history.rst

@@ -11,6 +11,7 @@ Version history
 * admin: added config dump support for Secret Discovery Service :ref:`SecretConfigDump <envoy_api_msg_admin.v2alpha.SecretsConfigDump>`.
 * api: added ::ref:`set_node_on_first_message_only <envoy_api_field_core.ApiConfigSource.set_node_on_first_message_only>` option to omit the node identifier from the subsequent discovery requests on the same stream.
 * buffer filter: the buffer filter populates content-length header if not present, behavior can be disabled using the runtime feature `envoy.reloadable_features.buffer_filter_populate_content_length`.
+* config: added support for :ref:`delta xDS <arch_overview_dynamic_config_delta>` (including ADS) delivery
 * config: enforcing that terminal filters (e.g. HttpConnectionManager for L4, router for L7) be the last in their respective filter chains.
 * config: added access log :ref:`extension filter<envoy_api_field_config.filter.accesslog.v2.AccessLogFilter.extension_filter>`.
 * config: added support for :option:`--reject-unknown-dynamic-fields`, providing independent control

include/envoy/config/BUILD

@@ -65,20 +65,3 @@ envoy_cc_library(
         "//source/common/protobuf",
     ],
 )
-
-envoy_cc_library(
-    name = "watch_map_interface",
-    hdrs = ["watch_map.h"],
-    deps = [
-        ":subscription_interface",
-    ],
-)
-
-envoy_cc_library(
-    name = "xds_grpc_context_interface",
-    hdrs = ["xds_grpc_context.h"],
-    deps = [
-        ":subscription_interface",
-        "//source/common/protobuf",
-    ],
-)

include/envoy/config/grpc_mux.h

@@ -67,6 +67,8 @@ class GrpcMuxWatch {
 
 using GrpcMuxWatchPtr = std::unique_ptr<GrpcMuxWatch>;
 
+struct Watch;
+
 /**
  * Manage one or more gRPC subscriptions on a single stream to management server. This can be used
  * for a single xDS API, e.g. EDS, or to combined multiple xDS APIs for ADS.
@@ -111,6 +113,13 @@ class GrpcMux {
    */
   virtual void resume(const std::string& type_url) PURE;
 
+  // For delta
+  virtual Watch* addOrUpdateWatch(const std::string& type_url, Watch* watch,
+                                  const std::set<std::string>& resources,
+                                  SubscriptionCallbacks& callbacks,
+                                  std::chrono::milliseconds init_fetch_timeout) PURE;
+  virtual void removeWatch(const std::string& type_url, Watch* watch) PURE;
+
   /**
    * Retrieves the current pause state as set by pause()/resume().
    * @param type_url type URL corresponding to xDS API, e.g.
@@ -121,6 +130,37 @@ class GrpcMux {
 };
 
 using GrpcMuxPtr = std::unique_ptr<GrpcMux>;
+using GrpcMuxSharedPtr = std::shared_ptr<GrpcMux>;
+
+/**
+ * A grouping of callbacks that a GrpcMux should provide to its GrpcStream.
+ */
+template <class ResponseProto> class GrpcStreamCallbacks {
+public:
+  virtual ~GrpcStreamCallbacks() = default;
+
+  /**
+   * For the GrpcStream to prompt the context to take appropriate action in response to the
+   * gRPC stream having been successfully established.
+   */
+  virtual void onStreamEstablished() PURE;
+
+  /**
+   * For the GrpcStream to prompt the context to take appropriate action in response to
+   * failure to establish the gRPC stream.
+   */
+  virtual void onEstablishmentFailure() PURE;
+
+  /**
+   * For the GrpcStream to pass received protos to the context.
+   */
+  virtual void onDiscoveryResponse(std::unique_ptr<ResponseProto>&& message) PURE;
+
+  /**
+   * For the GrpcStream to call when its rate limiting logic allows more requests to be sent.
+   */
+  virtual void onWriteable() PURE;
+};
 
 } // namespace Config
 } // namespace Envoy

include/envoy/config/subscription.h

@@ -89,7 +89,7 @@ class Subscription {
    * @param resources vector of resource names to fetch. It's a (not unordered_)set so that it can
    * be passed to std::set_difference, which must be given sorted collections.
    */
-  virtual void updateResources(const std::set<std::string>& update_to_these_names) PURE;
+  virtual void updateResourceInterest(const std::set<std::string>& update_to_these_names) PURE;
 };
 
 using SubscriptionPtr = std::unique_ptr<Subscription>;

include/envoy/config/subscription_factory.h

@@ -24,7 +24,7 @@ class SubscriptionFactory {
   virtual SubscriptionPtr
   subscriptionFromConfigSource(const envoy::api::v2::core::ConfigSource& config,
                                absl::string_view type_url, Stats::Scope& scope,
-                               SubscriptionCallbacks& callbacks) PURE;
+                               SubscriptionCallbacks& callbacks, bool is_delta) PURE;
 };
 
 } // namespace Config

include/envoy/router/route_config_provider_manager.h

@@ -39,7 +39,7 @@ class RouteConfigProviderManager {
   virtual RouteConfigProviderPtr createRdsRouteConfigProvider(
       const envoy::config::filter::network::http_connection_manager::v2::Rds& rds,
       Server::Configuration::FactoryContext& factory_context, const std::string& stat_prefix,
-      Init::Manager& init_manager) PURE;
+      Init::Manager& init_manager, bool is_delta) PURE;
 
   /**
    * Get a RouteConfigSharedPtr for a statically defined route. Ownership is as described for

include/envoy/server/listener_manager.h

@@ -39,7 +39,8 @@ class ListenerComponentFactory {
    * @return an LDS API provider.
    * @param lds_config supplies the management server configuration.
    */
-  virtual LdsApiPtr createLdsApi(const envoy::api::v2::core::ConfigSource& lds_config) PURE;
+  virtual LdsApiPtr createLdsApi(const envoy::api::v2::core::ConfigSource& lds_config,
+                                 bool is_delta) PURE;
 
   /**
    * Creates a socket.
@@ -128,7 +129,8 @@ class ListenerManager {
    * pieces of the server existing.
    * @param lds_config supplies the management server configuration.
    */
-  virtual void createLdsApi(const envoy::api::v2::core::ConfigSource& lds_config) PURE;
+  virtual void createLdsApi(const envoy::api::v2::core::ConfigSource& lds_config,
+                            bool is_delta) PURE;
 
   /**
    * @return std::vector<std::reference_wrapper<Network::ListenerConfig>> a list of the currently

include/envoy/upstream/cluster_manager.h

@@ -180,14 +180,14 @@ class ClusterManager {
   virtual const envoy::api::v2::core::BindConfig& bindConfig() const PURE;
 
   /**
-   * Return a reference to the singleton ADS provider for upstream control plane muxing of xDS. This
-   * is treated somewhat as a special case in ClusterManager, since it does not relate logically to
-   * the management of clusters but instead is required early in ClusterManager/server
+   * Returns a shared_ptr to the singleton xDS-over-gRPC provider for upstream control plane muxing
+   * of xDS. This is treated somewhat as a special case in ClusterManager, since it does not relate
+   * logically to the management of clusters but instead is required early in ClusterManager/server
    * initialization and in various sites that need ClusterManager for xDS API interfacing.
    *
    * @return GrpcMux& ADS API provider referencee.
    */
-  virtual Config::GrpcMux& adsMux() PURE;
+  virtual Config::GrpcMuxSharedPtr adsMux() PURE;
 
   /**
    * @return Grpc::AsyncClientManager& the cluster manager's gRPC client manager.
@@ -225,6 +225,8 @@ class ClusterManager {
   virtual Config::SubscriptionFactory& subscriptionFactory() PURE;
 
   virtual std::size_t warmingClusterCount() const PURE;
+
+  virtual bool xdsIsDelta() const PURE;
 };
 
 using ClusterManagerPtr = std::unique_ptr<ClusterManager>;
@@ -297,7 +299,7 @@ class ClusterManagerFactory {
   /**
    * Create a CDS API provider from configuration proto.
    */
-  virtual CdsApiPtr createCds(const envoy::api::v2::core::ConfigSource& cds_config,
+  virtual CdsApiPtr createCds(const envoy::api::v2::core::ConfigSource& cds_config, bool is_delta,
                               ClusterManager& cm) PURE;
 
   /**

source/common/config/BUILD

@@ -71,8 +71,8 @@ envoy_cc_library(
     srcs = ["delta_subscription_impl.cc"],
     hdrs = ["delta_subscription_impl.h"],
     deps = [
-        ":delta_subscription_state_lib",
         ":grpc_stream_lib",
+        ":new_grpc_mux_lib",
         ":utility_lib",
         "//include/envoy/config:subscription_interface",
         "//include/envoy/grpc:async_client_interface",
@@ -90,6 +90,7 @@ envoy_cc_library(
     srcs = ["delta_subscription_state.cc"],
     hdrs = ["delta_subscription_state.h"],
     deps = [
+        ":pausable_ack_queue_lib",
         "//include/envoy/config:subscription_interface",
         "//include/envoy/event:dispatcher_interface",
         "//source/common/common:assert_lib",
@@ -125,7 +126,6 @@ envoy_cc_library(
         ":utility_lib",
         "//include/envoy/config:grpc_mux_interface",
         "//include/envoy/config:subscription_interface",
-        "//include/envoy/config:xds_grpc_context_interface",
         "//include/envoy/grpc:async_client_interface",
         "//include/envoy/upstream:cluster_manager_interface",
         "//source/common/common:backoff_lib",
@@ -217,6 +217,20 @@ envoy_cc_library(
     ],
 )
 
+envoy_cc_library(
+    name = "new_grpc_mux_lib",
+    srcs = ["new_grpc_mux_impl.cc"],
+    hdrs = ["new_grpc_mux_impl.h"],
+    deps = [
+        ":delta_subscription_state_lib",
+        ":grpc_stream_lib",
+        ":pausable_ack_queue_lib",
+        ":watch_map_lib",
+        "//include/envoy/event:dispatcher_interface",
+        "//include/envoy/grpc:async_client_interface",
+    ],
+)
+
 envoy_cc_library(
     name = "http_subscription_lib",
     srcs = ["http_subscription_impl.cc"],
@@ -261,6 +275,16 @@ envoy_cc_library(
     ],
 )
 
+envoy_cc_library(
+    name = "pausable_ack_queue_lib",
+    srcs = ["pausable_ack_queue.cc"],
+    hdrs = ["pausable_ack_queue.h"],
+    deps = [
+        "//source/common/common:assert_lib",
+        "@envoy_api//envoy/api/v2:discovery_cc",
+    ],
+)
+
 envoy_cc_library(
     name = "protobuf_link_hacks",
     hdrs = ["protobuf_link_hacks.h"],
@@ -300,12 +324,6 @@ envoy_cc_library(
     ],
 )
 
-envoy_cc_library(
-    name = "resources_lib",
-    hdrs = ["resources.h"],
-    deps = ["//source/common/singleton:const_singleton"],
-)
-
 envoy_cc_library(
     name = "remote_data_fetcher_lib",
     srcs = ["remote_data_fetcher.cc"],
@@ -320,6 +338,12 @@ envoy_cc_library(
     ],
 )
 
+envoy_cc_library(
+    name = "resources_lib",
+    hdrs = ["resources.h"],
+    deps = ["//source/common/singleton:const_singleton"],
+)
+
 envoy_cc_library(
     name = "runtime_utility_lib",
     srcs = ["runtime_utility.cc"],