implement rds api

docs/configuration/cluster_manager/cds.rst

@@ -11,19 +11,19 @@ clusters depending on what is required.
 
   {
     "cluster": "{...}",
-    "refresh_interval_ms": "{...}"
+    "refresh_delay_ms": "..."
   }
 
 :ref:`cluster <config_cluster_manager_cluster>`
   *(required, object)* A standard definition of an upstream cluster that hosts the cluster
   discovery service. The cluster must run a REST service that implements the :ref:`CDS HTTP API
   <config_cluster_manager_cds_api>`.
 
-refresh_interval_ms
-  *(optional, integer)* The delay, in milliseconds, between fetches to the CDS API for each
-  configured CDS cluster. Envoy will add an additional random jitter to the delay that is between
-  zero and *refresh_interval_ms* milliseconds. Thus the longest possible refresh delay is
-  2 \* *refresh_interval_ms*. Default value is 30000ms (30 seconds).
+refresh_delay_ms
+  *(optional, integer)* The delay, in milliseconds, between fetches to the CDS API. Envoy will add
+  an additional random jitter to the delay that is between zero and *refresh_delay_ms*
+  milliseconds. Thus the longest possible refresh delay is 2 \* *refresh_delay_ms*. Default value
+  is 30000ms (30 seconds).
 
 .. _config_cluster_manager_cds_api:
 

docs/configuration/http_conn_man/http_conn_man.rst

@@ -14,6 +14,7 @@ HTTP connection manager
     "config": {
       "codec_type": "...",
       "stat_prefix": "...",
+      "rds": "{...}",
       "route_config": "{...}",
       "filters": [],
       "add_user_agent": "...",
@@ -54,9 +55,16 @@ stat_prefix
   connection manager. See the :ref:`statistics <config_http_conn_man_stats>` documentation
   for more information.
 
+:ref:`rds <config_http_conn_man_rds>`
+  *(sometimes required, object)* The connection manager configuration must specify one of *rds* or
+  *route_config*. If *rds* is specified, the connection manager's route table will be dynamically
+  loaded via the RDS API. See the :ref:`documentation <config_http_conn_man_rds>` for more
+  information.
+
 :ref:`route_config <config_http_conn_man_route_table>`
-  *(required, object)* The :ref:`route table <arch_overview_http_routing>` for the connection
-  manager. All connection managers must have a route table, even if it is empty.
+  *(sometimes required, object)* The connection manager configuration must specify one of *rds* or
+  *route_config*. If *route_config* is specified, the :ref:`route table <arch_overview_http_routing>`
+  for the connection manager is static and is specified in this property.
 
 :ref:`filters <config_http_conn_man_filters>`
   *(required, array)* A list of individual :ref:`HTTP filters <arch_overview_http_filters>` that
@@ -74,7 +82,7 @@ add_user_agent
 :ref:`tracing <config_http_conn_man_tracing>`
   *(optional, object)* Presence of the object defines whether the connection manager
   emits :ref:`tracing <arch_overview_tracing>` data to the :ref:`configured tracing provider <config_tracing>`.
-  
+
 .. _config_http_conn_man_http_codec_options:
 
 http_codec_options
@@ -145,3 +153,4 @@ generate_request_id
   headers
   stats
   runtime
+  rds

docs/configuration/http_conn_man/rds.rst

@@ -0,0 +1,79 @@
+.. _config_http_conn_man_rds:
+
+Route discovery service
+=======================
+
+The route discovery service (RDS) API is an optional API that Envoy will call to dynamically fetch
+:ref:`route configurations <config_http_conn_man_route_table>`. A route configuration includes both
+HTTP header modifications, virtual hosts, and the individual route entries contained within each
+virtual host. Each :ref:`HTTP connection manager filter <config_http_conn_man>` can independently
+fetch its own route configuration via the API.
+
+.. code-block:: json
+
+  {
+    "cluster": "{...}",
+    "route_config_name": "...",
+    "refresh_delay_ms": "..."
+  }
+
+cluster
+  *(required, string)* The name of an upstream :ref:`cluster <config_cluster_manager_cluster>` that
+  hosts the route discovery service. The cluster must run a REST service that implements the
+  :ref:`RDS HTTP API <config_http_conn_man_rds_api>`.
+
+route_config_name
+  *(required, string)* The name of the route configuration. This name will be passed to the
+  :ref:`RDS HTTP API <config_http_conn_man_rds_api>`. This allows an Envoy configuration with
+  multiple HTTP listeners (and associated HTTP connection manager filters) to use different route
+  configurations.
+
+refresh_delay_ms
+  *(optional, integer)* The delay, in milliseconds, between fetches to the RDS API. Envoy will add
+  an additional random jitter to the delay that is between zero and *refresh_delay_ms*
+  milliseconds. Thus the longest possible refresh delay is 2 \* *refresh_delay_ms*. Default
+  value is 30000ms (30 seconds).
+
+.. _config_http_conn_man_rds_api:
+
+REST API
+--------
+
+.. http:get:: /v1/routes/(string: route_config_name)/(string: service_cluster)/(string: service_node)
+
+Asks the discovery service to return the route configuration for a particular `route_config_name`,
+`service_cluster`, and `service_node`. `route_config_name` corresponds to the RDS configuration
+parameter above. `service_cluster` corresponds to the :option:`--service-cluster` CLI option.
+`service_node` corresponds to the :option:`--service-node` CLI option. Responses are a single JSON
+object that contains a route configuration as defined in the :ref:`route configuration documentation
+<config_http_conn_man_route_table>`.
+
+A new route configuration will be gracefully swapped in such that existing requests are not
+affected. This means that when a request starts, it sees a consistent snapshot of the route
+configuration that does not change for the duration of the request. Thus, if an update changes a
+timeout for example, only new requests will use the updated timeout value.
+
+As a performance optimization, Envoy hashes the route configuration it receives from the RDS API and
+will only perform a full reload if the hash value changes.
+
+.. attention::
+
+  Route configurations that are loaded via RDS are *not* checked to see if referenced clusters are
+  known to the :ref:`cluster manager <config_cluster_manager>`. The RDS API has been designed to
+  work alongside the :ref:`CDS API <config_cluster_manager_cds>` such that Envoy assumes eventually
+  consistent updates. If a route references an unknown cluster a 404 response will be returned by
+  the router filter.
+
+Statistics
+----------
+
+RDS has a statistics tree rooted at *http.<stat_prefix>.rds.* with the following statistics:
+
+.. csv-table::
+  :header: Name, Type, Description
+  :widths: 1, 1, 2
+
+  config_reload, Counter, Total API fetches that resulted in a config reload due to a different config
+  update_attempt, Counter, Total API fetches attempted
+  update_success, Counter, Total API fetches completed successfully
+  update_failure, Counter, Total API fetches that failed (either network or schema errors)

docs/configuration/network_filters/client_ssl_auth_filter.rst

@@ -13,7 +13,7 @@ Client TLS authentication filter :ref:`architecture overview <arch_overview_ssl_
     "config": {
       "auth_api_cluster": "...",
       "stat_prefix": "...",
-      "refresh_interval_ms": "...",
+      "refresh_delay_ms": "...",
       "ip_white_list": []
     }
   }
@@ -28,10 +28,10 @@ stat_prefix
   *(required, string)* The prefix to use when emitting :ref:`statistics
   <config_network_filters_client_ssl_auth_stats>`.
 
-refresh_interval_ms
+refresh_delay_ms
   *(optional, integer)* Time in milliseconds between principal refreshes from the authentication
   service. Default is 60000 (60s). The actual fetch time will be this value plus a random jittered
-  value between 0-refresh_interval_ms milliseconds.
+  value between 0-refresh_delay_ms milliseconds.
 
 ip_white_list
   *(optional, array)* An optional list of IP address and subnet masks that should be white listed

include/envoy/router/rds.h

@@ -0,0 +1,24 @@
+#pragma once
+
+#include "envoy/router/router.h"
+
+namespace Router {
+
+/**
+ * A provider for constant route configurations.
+ */
+class RouteConfigProvider {
+public:
+  virtual ~RouteConfigProvider() {}
+
+  /**
+   * @return Router::ConfigPtr a route configuration for use during a single request. The returned
+   *         config may be different on a subsequent call, so a new config should be acquired for
+   *         each request flow.
+   */
+  virtual Router::ConfigPtr config() PURE;
+};
+
+typedef std::unique_ptr<RouteConfigProvider> RouteConfigProviderPtr;
+
+} // Router

include/envoy/router/router.h

@@ -264,6 +264,6 @@ class Config {
   virtual bool usesRuntime() const PURE;
 };
 
-typedef std::unique_ptr<Config> ConfigPtr;
+typedef std::shared_ptr<const Config> ConfigPtr;
 
 } // Router

source/common/CMakeLists.txt

@@ -90,6 +90,7 @@ add_library(
   redis/conn_pool_impl.cc
   redis/proxy_filter.cc
   router/config_impl.cc
+  router/rds_impl.cc
   router/retry_state_impl.cc
   router/router.cc
   router/router_ratelimit.cc

source/common/filter/auth/client_ssl.cc

@@ -18,7 +18,7 @@ Config::Config(const Json::Object& config, ThreadLocal::Instance& tls, Upstream:
                Event::Dispatcher& dispatcher, Stats::Store& stats_store,
                Runtime::RandomGenerator& random)
     : RestApiFetcher(cm, config.getString("auth_api_cluster"), dispatcher, random,
-                     std::chrono::milliseconds(config.getInteger("refresh_interval_ms", 60000))),
+                     std::chrono::milliseconds(config.getInteger("refresh_delay_ms", 60000))),
       tls_(tls), tls_slot_(tls.allocateSlot()), ip_white_list_(config, "ip_white_list"),
       stats_(generateStats(stats_store, config.getString("stat_prefix"))) {
 

source/common/http/conn_manager_impl.cc

@@ -287,8 +287,9 @@ void ConnectionManagerImpl::chargeTracingStats(const Tracing::Reason& tracing_re
 
 ConnectionManagerImpl::ActiveStream::ActiveStream(ConnectionManagerImpl& connection_manager)
     : connection_manager_(connection_manager),
-      stream_id_(ConnectionManagerUtility::generateStreamId(
-          connection_manager.config_.routeConfig(), connection_manager.random_generator_)),
+      snapped_route_config_(connection_manager.config_.routeConfigProvider().config()),
+      stream_id_(ConnectionManagerUtility::generateStreamId(*snapped_route_config_,
+                                                            connection_manager.random_generator_)),
       request_timer_(connection_manager_.stats_.named_.downstream_rq_time_.allocateSpan()),
       request_info_(connection_manager_.codec_->protocol()) {
   connection_manager_.stats_.named_.downstream_rq_total_.inc();
@@ -434,7 +435,7 @@ void ConnectionManagerImpl::ActiveStream::decodeHeaders(HeaderMapPtr&& headers,
 
   ConnectionManagerUtility::mutateRequestHeaders(
       *request_headers_, connection_manager_.read_callbacks_->connection(),
-      connection_manager_.config_, connection_manager_.random_generator_,
+      connection_manager_.config_, *snapped_route_config_, connection_manager_.random_generator_,
       connection_manager_.runtime_);
 
   // Check if tracing is enabled at all.
@@ -582,7 +583,7 @@ void ConnectionManagerImpl::ActiveStream::encodeHeaders(ActiveStreamEncoderFilte
   connection_manager_.config_.dateProvider().setDateHeader(headers);
   headers.insertServer().value(connection_manager_.config_.serverName());
   ConnectionManagerUtility::mutateResponseHeaders(headers, *request_headers_,
-                                                  connection_manager_.config_);
+                                                  *snapped_route_config_);
 
   // See if we want to drain/close the connection. Send the go away frame prior to encoding the
   // header block.
@@ -820,8 +821,8 @@ AccessLog::RequestInfo& ConnectionManagerImpl::ActiveStreamFilterBase::requestIn
 
 Router::RoutePtr ConnectionManagerImpl::ActiveStreamFilterBase::route() {
   if (!parent_.cached_route_.valid()) {
-    parent_.cached_route_.value(parent_.connection_manager_.config_.routeConfig().route(
-        *parent_.request_headers_, parent_.stream_id_));
+    parent_.cached_route_.value(
+        parent_.snapped_route_config_->route(*parent_.request_headers_, parent_.stream_id_));
   }
 
   return parent_.cached_route_.value();

source/common/http/conn_manager_impl.h

@@ -10,6 +10,7 @@
 #include "envoy/network/connection.h"
 #include "envoy/network/drain_decision.h"
 #include "envoy/network/filter.h"
+#include "envoy/router/rds.h"
 #include "envoy/runtime/runtime.h"
 #include "envoy/stats/stats_macros.h"
 #include "envoy/tracing/http_tracer.h"
@@ -159,9 +160,10 @@ class ConnectionManagerConfig {
   virtual const Optional<std::chrono::milliseconds>& idleTimeout() PURE;
 
   /**
-   * @return const Router::Config& the route configuration for all connection manager requests.
+   * @return Router::RouteConfigProvider& the configuration provider used to acquire a route
+   *         config for each request flow.
    */
-  virtual const Router::Config& routeConfig() PURE;
+  virtual Router::RouteConfigProvider& routeConfigProvider() PURE;
 
   /**
    * @return const std::string& the server name to write into responses.
@@ -396,6 +398,7 @@ class ConnectionManagerImpl : Logger::Loggable<Logger::Id::http>,
     };
 
     ConnectionManagerImpl& connection_manager_;
+    Router::ConfigPtr snapped_route_config_;
     Tracing::SpanPtr active_span_;
     const uint64_t stream_id_;
     StreamEncoder* response_encoder_{};

source/common/http/conn_manager_utility.cc

@@ -25,6 +25,7 @@ uint64_t ConnectionManagerUtility::generateStreamId(const Router::Config& route_
 void ConnectionManagerUtility::mutateRequestHeaders(Http::HeaderMap& request_headers,
                                                     Network::Connection& connection,
                                                     ConnectionManagerConfig& config,
+                                                    const Router::Config& route_config,
                                                     Runtime::RandomGenerator& random,
                                                     Runtime::Loader& runtime) {
   // Clean proxy headers.
@@ -80,7 +81,7 @@ void ConnectionManagerUtility::mutateRequestHeaders(Http::HeaderMap& request_hea
     request_headers.removeEnvoyExpectedRequestTimeoutMs();
     request_headers.removeEnvoyForceTrace();
 
-    for (const Http::LowerCaseString& header : config.routeConfig().internalOnlyHeaders()) {
+    for (const Http::LowerCaseString& header : route_config.internalOnlyHeaders()) {
       request_headers.remove(header);
     }
   }
@@ -123,16 +124,16 @@ void ConnectionManagerUtility::mutateRequestHeaders(Http::HeaderMap& request_hea
 
 void ConnectionManagerUtility::mutateResponseHeaders(Http::HeaderMap& response_headers,
                                                      const Http::HeaderMap& request_headers,
-                                                     ConnectionManagerConfig& config) {
+                                                     const Router::Config& route_config) {
   response_headers.removeConnection();
   response_headers.removeTransferEncoding();
 
-  for (const Http::LowerCaseString& to_remove : config.routeConfig().responseHeadersToRemove()) {
+  for (const Http::LowerCaseString& to_remove : route_config.responseHeadersToRemove()) {
     response_headers.remove(to_remove);
   }
 
   for (const std::pair<Http::LowerCaseString, std::string>& to_add :
-       config.routeConfig().responseHeadersToAdd()) {
+       route_config.responseHeadersToAdd()) {
     response_headers.addStatic(to_add.first, to_add.second);
   }
 

source/common/http/conn_manager_utility.h

@@ -17,11 +17,12 @@ class ConnectionManagerUtility {
 
   static void mutateRequestHeaders(Http::HeaderMap& request_headers,
                                    Network::Connection& connection, ConnectionManagerConfig& config,
+                                   const Router::Config& route_config,
                                    Runtime::RandomGenerator& random, Runtime::Loader& runtime);
 
   static void mutateResponseHeaders(Http::HeaderMap& response_headers,
                                     const Http::HeaderMap& request_headers,
-                                    ConnectionManagerConfig& config);
+                                    const Router::Config& route_config);
 
 private:
   // NOTE: This is used for stable randomness in the case where the route table does not use any

source/common/http/filter/fault_filter.h

@@ -4,7 +4,6 @@
 #include "envoy/runtime/runtime.h"
 #include "envoy/stats/stats_macros.h"
 
-#include "common/json/json_loader.h"
 #include "common/router/config_impl.h"
 
 namespace Http {

source/common/json/config_schemas.cc

@@ -69,6 +69,23 @@ const std::string Json::Schema::CLIENT_SSL_NETWORK_FILTER_SCHEMA(R"EOF(
   }
   )EOF");
 
+const std::string Json::Schema::RDS_CONFIGURATION_SCHEMA(R"EOF(
+  {
+    "$schema": "http://json-schema.org/schema#",
+    "properties" : {
+      "cluster" : {"type": "string"},
+      "route_config_name" : {"type": "string"},
+      "refresh_delay_ms" : {
+        "type" : "integer",
+        "minimum" : 0,
+        "exclusiveMinimum" : true
+      }
+    },
+    "required" : ["cluster", "route_config_name"],
+    "additionalProperties" : false
+  }
+  )EOF");
+
 const std::string Json::Schema::HTTP_CONN_NETWORK_FILTER_SCHEMA(R"EOF(
   {
     "$schema": "http://json-schema.org/schema#",
@@ -201,6 +218,7 @@ const std::string Json::Schema::HTTP_CONN_NETWORK_FILTER_SCHEMA(R"EOF(
         "enum" : ["http1", "http2", "auto"]
       },
       "stat_prefix" : {"type" : "string"},
+      "rds" : {"type": "object"},
       "route_config" : {"type": "object"},
       "filters" : {
         "type" : "array",
@@ -242,7 +260,7 @@ const std::string Json::Schema::HTTP_CONN_NETWORK_FILTER_SCHEMA(R"EOF(
       "use_remote_address" : {"type" : "boolean"},
       "generate_request_id" : {"type" : "boolean"}
     },
-    "required" : ["codec_type", "stat_prefix", "route_config", "filters"],
+    "required" : ["codec_type", "stat_prefix", "filters"],
     "additionalProperties" : false
   }
   )EOF");
@@ -698,7 +716,7 @@ const std::string Json::Schema::CLUSTER_MANAGER_SCHEMA(R"EOF(
         "type" : "object",
         "properties" : {
           "cluster" : {"type" : "object"},
-          "refresh_interval_ms" : {
+          "refresh_delay_ms" : {
             "type" : "integer",
             "minimum" : 0,
             "exclusiveMinimum" : true