Overload Manager: Reset expensive streams

docs/root/configuration/operations/overload_manager/overload_manager.rst

@@ -96,6 +96,10 @@ The following overload actions are supported:
     - Envoy will reduce the waiting period for a configured set of timeouts. See
       :ref:`below <config_overload_manager_reducing_timeouts>` for details on configuration.
 
+  * - envoy.overload_actions.reset_streams
+    - Envoy will reset expensive streams to terminate them. See
+      :ref:`below <config_overload_manager_reset_streams>` for details on configuration.
+
 .. _config_overload_manager_reducing_timeouts:
 
 Reducing timeouts
@@ -163,6 +167,47 @@ all listeners.
 
 An example configuration can be found in the :ref:`edge best practices document <best_practices_edge>`.
 
+.. _config_overload_manager_reset_streams:
+
+Reset Streams
+^^^^^^^^^^^^^^^^^
+
+The ``envoy.overload_actions.reset_streams`` overload action will reset
+expensive streams. This works in conjuction with the
+``envoy.reloadable_features.per_stream_buffer_accounting`` flag which enables
+per stream buffer accounting.
+
+As an example, here is a single overload action entry that enables reset streams:
+
+.. code-block:: yaml
+
+  name: "envoy.overload_actions.reset_streams"
+  triggers:
+    - name: "envoy.resource_monitors.fixed_heap"
+      scaled:
+        scaling_threshold: 0.85
+        saturation_threshold: 0.95
+
+It configures the overload manager to reset certain streams depending on the
+heap size.  When the heap usage is less than 85%, no streams will be reset.
+When heap usage is at or above 85%, we start to reset certain memory classes
+(e.g. streams using memory within a power of two range). There are 8 buckets,
+with the last bucket capturing all of the streams using :math:`>= 128 *
+minimum_threshold_for_tracking`.  The `minimum_threshold_for_tracking` can be
+configured via :ref:`buffer_factory_config
+<envoy_v3_api_field_config.bootstrap.v3.Bootstrap.buffer_factory_config>`.
+
+Given that there are only 8 buckets, we partition the space with a gradation of
+:math:`gradation = (saturation_threshold - scaling_threshold)/8`. Hence at 85%
+we reset streams in the last bucket. At :math:`85% + 1 * gradation` we reset
+streams in the second to last, and last bucket. And so forth as memory pressure
+is higher.
+
+It's expected that the first few gradations shouldn't trigger anything, unless
+there's something seriously wrong e.g. the existence of streams using :math:`>=
+128 * minimum_threshold_for_tracking`.
+
+
 Statistics
 ----------
 

docs/root/version_history/current.rst

@@ -62,6 +62,55 @@ New Features
 * listener: new listener metric `downstream_cx_transport_socket_connect_timeout` to track transport socket timeouts.
 * rbac: added :ref:`destination_port_range <envoy_v3_api_field_config.rbac.v3.Permission.destination_port_range>` for matching range of destination ports.
 
+* access_log: added the new response flag for :ref:`overload manager termination <envoy_v3_api_field_data.accesslog.v3.ResponseFlags.overload_manager>`. The response flag will be set when the http stream is terminated by overload manager.
+* admission control: added :ref:`admission control <envoy_v3_api_field_extensions.filters.http.admission_control.v3alpha.AdmissionControl.rps_threshold>` option that when average RPS of the sampling window is below this threshold, the filter will not throttle requests. Added :ref:`admission control <envoy_v3_api_field_extensions.filters.http.admission_control.v3alpha.AdmissionControl.max_rejection_probability>` option to set an upper limit on the probability of rejection.
+* bandwidth_limit: added new :ref:`HTTP bandwidth limit filter <config_http_filters_bandwidth_limit>`.
+* bootstrap: added :ref:`dns_resolution_config <envoy_v3_api_field_config.bootstrap.v3.Bootstrap.dns_resolution_config>` to aggregate all of the DNS resolver configuration in a single message. By setting one such configuration option ``no_default_search_domain`` as true the DNS resolver will not use the default search domains. And by setting the configuration ``resolvers`` we can specify the external DNS servers to be used for external DNS query.
+* cluster: added :ref:`dns_resolution_config <envoy_v3_api_field_config.cluster.v3.Cluster.dns_resolution_config>` to aggregate all of the DNS resolver configuration in a single message. By setting one such configuration option ``no_default_search_domain`` as true the DNS resolver will not use the default search domains.
+* cluster: added :ref:`host_rewrite_literal <envoy_v3_api_field_config.route.v3.WeightedCluster.ClusterWeight.host_rewrite_literal>` to WeightedCluster.
+* cluster: added :ref:`wait_for_warm_on_init <envoy_v3_api_field_config.cluster.v3.Cluster.wait_for_warm_on_init>`, which allows cluster readiness to not block on cluster warm-up. It is true by default, which preserves existing behavior. Currently, only applicable for DNS-based clusters.
+* composite filter: can now be used with filters that also add an access logger, such as the WASM filter.
+* config: added stat :ref:`config_reload_time_ms <subscription_statistics>`.
+* connection_limit: added new :ref:`Network connection limit filter <config_network_filters_connection_limit>`.
+* crash support: restore crash context when continuing to processing requests or responses as a result of an asynchronous callback that invokes a filter directly. This is unlike the call stacks that go through the various network layers, to eventually reach the filter. For a concrete example see: ``Envoy::Extensions::HttpFilters::Cache::CacheFilter::getHeaders`` which posts a callback on the dispatcher that will invoke the filter directly.
+* dns cache: added :ref:`preresolve_hostnames <envoy_v3_api_field_extensions.common.dynamic_forward_proxy.v3.DnsCacheConfig.preresolve_hostnames>` option to the DNS cache config. This option allows hostnames to be preresolved into the cache upon cache creation. This might provide performance improvement, in the form of cache hits, for hostnames that are going to be resolved during steady state and are known at config load time.
+* dns cache: added :ref:`dns_query_timeout <envoy_v3_api_field_extensions.common.dynamic_forward_proxy.v3.DnsCacheConfig.dns_query_timeout>` option to the DNS cache config. This option allows explicitly controlling the timeout of underlying queries independently of the underlying DNS platform implementation. Coupled with success and failure retry policies the use of this timeout will lead to more deterministic DNS resolution times.
+* dns resolver: added ``DnsResolverOptions`` protobuf message to reconcile all of the DNS lookup option flags. By setting the configuration option :ref:`use_tcp_for_dns_lookups <envoy_v3_api_field_config.core.v3.DnsResolverOptions.use_tcp_for_dns_lookups>` as true we can make the underlying dns resolver library to make only TCP queries to the DNS servers and by setting the configuration option :ref:`no_default_search_domain <envoy_v3_api_field_config.core.v3.DnsResolverOptions.no_default_search_domain>` as true the DNS resolver library will not use the default search domains.
+* dns resolver: added ``DnsResolutionConfig`` to combine :ref:`dns_resolver_options <envoy_v3_api_field_config.core.v3.DnsResolutionConfig.dns_resolver_options>` and :ref:`resolvers <envoy_v3_api_field_config.core.v3.DnsResolutionConfig.resolvers>` in a single protobuf message. The field ``resolvers`` can be specified with a list of DNS resolver addresses. If specified, DNS client library will perform resolution via the underlying DNS resolvers. Otherwise, the default system resolvers (e.g., /etc/resolv.conf) will be used.
+* dns_filter: added :ref:`dns_resolution_config <envoy_v3_api_field_extensions.filters.udp.dns_filter.v3alpha.DnsFilterConfig.ClientContextConfig.dns_resolution_config>` to aggregate all of the DNS resolver configuration in a single message. By setting the configuration option ``use_tcp_for_dns_lookups`` to true we can make dns filter's external resolvers to answer queries using TCP only, by setting the configuration option ``no_default_search_domain`` as true the DNS resolver will not use the default search domains. And by setting the configuration ``resolvers`` we can specify the external DNS servers to be used for external DNS query which replaces the pre-existing alpha api field ``upstream_resolvers``.
+* dynamic_forward_proxy: added :ref:`dns_resolution_config <envoy_v3_api_field_extensions.common.dynamic_forward_proxy.v3.DnsCacheConfig.dns_resolution_config>` option to the DNS cache config in order to aggregate all of the DNS resolver configuration in a single message. By setting one such configuration option ``no_default_search_domain`` as true the DNS resolver will not use the default search domains. And by setting the configuration ``resolvers`` we can specify the external DNS servers to be used for external DNS query instead of the system default resolvers.
+* http: a new field ``is_optional`` is added to ``extensions.filters.network.http_connection_manager.v3.HttpFilter``. When
+  value is ``true``, the unsupported http filter will be ignored by envoy. This is also same with unsupported http filter
+  in the typed per filter config. For more information, please reference
+  :ref:`HttpFilter <envoy_v3_api_field_extensions.filters.network.http_connection_manager.v3.HttpFilter.is_optional>`.
+* http: added :ref``scheme options <envoy_v3_api_field_extensions.filters.network.http_connection_manager.v3.HttpConnectionManager.scheme>` for adding or overwriting scheme.
+* http: added :ref:`stripping trailing host dot from host header <envoy_v3_api_field_extensions.filters.network.http_connection_manager.v3.HttpConnectionManager.strip_trailing_host_dot>` support.
+* http: added support for :ref:`original IP detection extensions <envoy_v3_api_field_extensions.filters.network.http_connection_manager.v3.HttpConnectionManager.original_ip_detection_extensions>`.
+  Two initial extensions were added, the :ref:`custom header <envoy_v3_api_msg_extensions.http.original_ip_detection.custom_header.v3.CustomHeaderConfig>` extension and the
+  :ref:`xff <envoy_v3_api_msg_extensions.http.original_ip_detection.xff.v3.XffConfig>` extension.
+* http: added a new option to upstream HTTP/2 :ref:`keepalive <envoy_v3_api_field_config.core.v3.Http2ProtocolOptions.connection_keepalive>` to send a PING ahead of a new stream if the connection has been idle for a sufficient duration.
+* http: added the ability to :ref:`unescape slash sequences <envoy_v3_api_field_extensions.filters.network.http_connection_manager.v3.HttpConnectionManager.path_with_escaped_slashes_action>` in the path. Requests with unescaped slashes can be proxied, rejected or redirected to the new unescaped path. By default this feature is disabled. The default behavior can be overridden through :ref:`http_connection_manager.path_with_escaped_slashes_action<config_http_conn_man_runtime_path_with_escaped_slashes_action>` runtime variable. This action can be selectively enabled for a portion of requests by setting the :ref:`http_connection_manager.path_with_escaped_slashes_action_sampling<config_http_conn_man_runtime_path_with_escaped_slashes_action_enabled>` runtime variable.
+* http: added upstream and downstream alpha HTTP/3 support! See :ref:`quic_options <envoy_v3_api_field_config.listener.v3.UdpListenerConfig.quic_options>` for downstream and the new http3_protocol_options in :ref:`http_protocol_options <envoy_v3_api_msg_extensions.upstreams.http.v3.HttpProtocolOptions>` for upstream HTTP/3.
+* input matcher: a new input matcher that :ref:`matches an IP address against a list of CIDR ranges <envoy_v3_api_file_envoy/extensions/matching/input_matchers/ip/v3/ip.proto>`.
+* jwt_authn: added support to fetch remote jwks asynchronously specified by :ref:`async_fetch <envoy_v3_api_field_extensions.filters.http.jwt_authn.v3.RemoteJwks.async_fetch>`.
+* jwt_authn: added support to add padding in the forwarded JWT payload specified by :ref:`pad_forward_payload_header <envoy_v3_api_field_extensions.filters.http.jwt_authn.v3.JwtProvider.pad_forward_payload_header>`.
+* listener: added ability to change an existing listener's address.
+* listener: added filter chain match support for :ref:`direct source address <envoy_v3_api_field_config.listener.v3.FilterChainMatch.direct_source_prefix_ranges>`.
+* local_rate_limit_filter: added suppoort for locally rate limiting http requests on a per connection basis. This can be enabled by setting the :ref:`local_rate_limit_per_downstream_connection <envoy_v3_api_field_extensions.filters.http.local_ratelimit.v3.LocalRateLimit.local_rate_limit_per_downstream_connection>` field to true.
+* metric service: added support for sending metric tags as labels. This can be enabled by setting the :ref:`emit_tags_as_labels <envoy_v3_api_field_config.metrics.v3.MetricsServiceConfig.emit_tags_as_labels>` field to true.
+* overload: add a new overload action that resets streams using a lot of memory. To enable the tracking of allocated bytes in buffers that a stream is using turn on ``envoy.reloadable_features.per_stream_buffer_accounting``. Only streams using above a minimum threshold of allocated bytes are tracked in various power of two sized buckets. The minimum threshold for tracking is tunable via :ref:`buffer_factory_config <envoy_v3_api_field_config.bootstrap.v3.Bootstrap.buffer_factory_config>`. If the overload action is triggered, we reset the most expensive stream first.
+* proxy protocol: added support for generating the header while using the :ref:`HTTP connection manager <config_http_conn_man>`. This is done using the using the :ref:`Proxy Protocol Transport Socket <extension_envoy.transport_sockets.upstream_proxy_protocol>` on upstream clusters.
+  This feature is currently affected by a memory leak `issue <https://github.com/envoyproxy/envoy/issues/16682>`_.
+* req_without_query: added access log formatter extension implementing command operator :ref:`REQ_WITHOUT_QUERY <envoy_v3_api_msg_extensions.formatter.req_without_query.v3.ReqWithoutQuery>` to log the request path, while excluding the query string.
+* router: added flag ``suppress_grpc_request_failure_code_stats`` to :ref:`key <envoy_v3_api_msg_extensions.filters.http.router.v3.Router>` to allow users to exclude incrementing HTTP status code stats on gRPC requests.
+* stats: added native :ref:`Graphite-formatted tag <envoy_v3_api_msg_extensions.stat_sinks.graphite_statsd.v3.GraphiteStatsdSink>` support.
+* tcp: added support for :ref:`preconnecting <v1.18.0:envoy_v3_api_msg_config.cluster.v3.Cluster.PreconnectPolicy>`. Preconnecting is off by default, but recommended for clusters serving latency-sensitive traffic.
+* thrift_proxy: added per upstream metrics within the :ref:`thrift router <envoy_v3_api_msg_extensions.filters.network.thrift_proxy.router.v3.Router>` for request and response size histograms.
+* thrift_proxy: added support for :ref:`outlier detection <arch_overview_outlier_detection>`.
+* tls: allow dual ECDSA/RSA certs via SDS. Previously, SDS only supported a single certificate per context, and dual cert was only supported via non-SDS.
+* udp_proxy: added :ref:`key <envoy_v3_api_msg_extensions.filters.udp.udp_proxy.v3.UdpProxyConfig.HashPolicy>` as another hash policy to support hash based routing on any given key.
+* windows container image: added user, EnvoyUser which is part of the Network Configuration Operators group to the container image.
+
 Deprecated
 ----------
 * cluster: :ref:`max_requests_per_connection <envoy_v3_api_field_config.cluster.v3.Cluster.max_requests_per_connection>` is deprecated in favor of :ref:`max_requests_per_connection <envoy_v3_api_field_config.core.v3.HttpProtocolOptions.max_requests_per_connection>`.

envoy/buffer/buffer.h

@@ -521,6 +521,15 @@ class WatermarkFactory {
    * @return a BufferMemoryAccountSharedPtr of the newly created account.
    */
   virtual BufferMemoryAccountSharedPtr createAccount(Http::StreamResetHandler& reset_handler) PURE;
+
+  /**
+   * Goes through the tracked accounts, resetting the accounts and their
+   * corresponding stream depending on the pressure.
+   *
+   * @param pressure scaled threshold pressure used to compute the buckets to
+   *  reset internally.
+   */
+  virtual void resetAccountsGivenPressure(float pressure) PURE;
 };
 
 using WatermarkFactoryPtr = std::unique_ptr<WatermarkFactory>;

envoy/server/overload/overload_manager.h

@@ -34,6 +34,9 @@ class OverloadActionNameValues {
 
   // Overload action to reduce some subset of configured timeouts.
   const std::string ReduceTimeouts = "envoy.overload_actions.reduce_timeouts";
+
+  // Overload action to reset streams using excessive memory.
+  const std::string ResetStreams = "envoy.overload_actions.reset_streams";
 };
 
 using OverloadActionNames = ConstSingleton<OverloadActionNameValues>;