[WIP] HdsDelegate can healthcheck an endpoint

.circleci/config.yml

@@ -1,6 +1,6 @@
 references:
   envoy-build-image: &envoy-build-image
-    envoyproxy/envoy-build:7f7f5666c72e00ac7c1909b4fc9a2121d772c859
+    envoyproxy/envoy-build:1ef23d481a4701ad4a414d1ef98036bd2ed322e7
 
 version: 2
 jobs:

.gitattributes

@@ -0,0 +1 @@
+/docs/root/intro/version_history.rst merge=union

.github/stale.yml

@@ -41,3 +41,4 @@ issues:
     Thank you for your contributions.
   exemptLabels:
     - help wanted
+    - no stalebot

.gitignore

@@ -15,3 +15,4 @@ SOURCE_VERSION
 .cache
 .vimrc
 .vscode
+.vs

CONTRIBUTING.md

@@ -90,7 +90,7 @@ maximize the chances of your PR being merged.
 * We expect that once a PR is opened, it will be actively worked on until it is merged or closed.
   We reserve the right to close PRs that are not making progress. This is generally defined as no
   changes for 7 days. Obviously PRs that are closed due to lack of activity can be reopened later.
-  Closing stale PRs helps us keep on top of all of the work currently in flight.
+  Closing stale PRs helps us to keep on top of all of the work currently in flight.
 * If a commit deprecates a feature, the commit message must mention what has been deprecated.
   Additionally, [DEPRECATED.md](DEPRECATED.md) must be updated as part of the commit.
 * Please consider joining the [envoy-dev](https://groups.google.com/forum/#!forum/envoy-dev)

DEPRECATED.md

@@ -8,13 +8,24 @@ A logged warning is expected for each deprecated item that is in deprecation win
 
 ## Version 1.8.0 (pending)
 
-* Use of the legacy 
+* Use of the v1 API is deprecated. See envoy-announce
+  [email](https://groups.google.com/forum/#!topic/envoy-announce/oPnYMZw8H4U).
+* Use of the legacy
   [ratelimit.proto](https://github.com/envoyproxy/envoy/blob/b0a518d064c8255e0e20557a8f909b6ff457558f/source/common/ratelimit/ratelimit.proto)
   is deprecated, in favor of the proto defined in
   [date-plane-api](https://github.com/envoyproxy/envoy/blob/master/api/envoy/service/ratelimit/v2/rls.proto)
   Prior to 1.8.0, Envoy can use either proto to send client requests to a ratelimit server with the use of the
   `use_data_plane_proto` boolean flag in the [ratelimit configuration](https://github.com/envoyproxy/envoy/blob/master/api/envoy/config/ratelimit/v2/rls.proto).
   However, when using the deprecated client a warning is logged.
+* Use of the --v2-config-only flag.
+* Use of both `use_websocket` and `websocket_config` in
+  [route.proto](https://github.com/envoyproxy/envoy/blob/master/api/envoy/api/v2/route/route.proto)
+  is deprecated. Please use the new `upgrade_configs` in the
+  [HttpConnectionManager](https://github.com/envoyproxy/envoy/blob/master/api/envoy/config/filter/network/http_connection_manager/v2/http_connection_manager.proto)
+  instead.
+* Setting hosts via `hosts` field in `Cluster` is deprecated. Use `load_assignment` instead.
+* Use of `response_headers_to_*` and `request_headers_to_add` are deprecated at the `RouteAction`
+  level. Please use the configuration options at the `Route` level.
 
 ## Version 1.7.0
 

README.md

@@ -14,7 +14,7 @@ involved and how Envoy plays a role, read the CNCF
 
 * [Official documentation](https://www.envoyproxy.io/)
 * [FAQ](https://www.envoyproxy.io/docs/envoy/latest/faq/overview)
-* [Unofficial Chinese documentation](https://github.com/lixiangyun/envoyproxy_doc_ZH_CN)
+* [Unofficial Chinese documentation](https://github.com/servicemesher/envoy/)
 * Watch [a video overview of Envoy](https://www.youtube.com/watch?v=RVZX4CwKhGE)
 ([transcript](https://www.microservices.com/talks/lyfts-envoy-monolith-service-mesh-matt-klein/))
 to find out more about the origin story and design philosophy of Envoy

REPO_LAYOUT.md

@@ -28,7 +28,7 @@ are:
 ## [source/](source/)
 
 * [common/](source/common/): Core Envoy code (not specific to extensions) that is also not
-  specific to a standalone server implementation. I.e., this is code that could be used if Envoy
+  specific to a standalone server implementation. I.e., this is the code that could be used if Envoy
   were eventually embedded as a library.
 * [docs/](source/docs/): Miscellaneous developer/design documentation that is not relevant for
   the public user documentation.

SECURITY_RELEASE_PROCESS.md

@@ -1,7 +1,7 @@
 # Security Release Process
 
 Envoy is a large growing community of volunteers, users, and vendors. The Envoy community has
-adopted this security disclosures and response policy to ensure we responsibly handle critical
+adopted this security disclosure and response policy to ensure we responsibly handle critical
 issues.
 
 ## Product Security Team (PST)
@@ -73,7 +73,7 @@ These steps should be completed within the 1-7 days of Disclosure.
 - The Fix Lead and the Fix Team will create a
   [CVSS](https://www.first.org/cvss/specification-document) using the [CVSS
   Calculator](https://www.first.org/cvss/calculator/3.0). The Fix Lead makes the final call on the
-  calculated CVSS; it is better to move quickly than make the CVSS perfect.
+  calculated CVSS; it is better to move quickly than making the CVSS perfect.
 - The Fix Team will notify the Fix Lead that work on the fix branch is complete once there are LGTMs
   on all commits in the private repo from one or more maintainers.
 
@@ -160,7 +160,7 @@ said issue, they must agree to the same terms and only find out information on a
 
 In the unfortunate event you share the information beyond what is allowed by this policy, you _must_
 urgently inform the envoy-security@googlegroups.com mailing list of exactly what information leaked
-and to whom. A retrospective will take place after the leak so we can assess how to not make the
+and to whom. A retrospective will take place after the leak so we can assess how to prevent making the
 same mistake in the future.
 
 If you continue to leak information and break the policy outlined here, you will be removed from the

STYLE.md

@@ -1,6 +1,6 @@
 # C++ coding style
 
-* The Envoy source code is formatted using clang-format. Thus all white space, etc.
+* The Envoy source code is formatted using clang-format. Thus all white spaces, etc.
   issues are taken care of automatically. The Travis tests will automatically check
   the code format and fail. There are make targets that can both check the format
   (check_format) as well as fix the code format for you (fix_format).
@@ -96,7 +96,7 @@ A few general notes on our error handling philosophy:
   silently be ignored and should crash the process either via the C++ allocation error exception, an
   explicit `RELEASE_ASSERT` following a third party library call, or an obvious crash on a subsequent
   line via null pointer dereference. This rule is again based on the philosophy that the engineering
-  costs of properly handling these cases is not worth it. Time is better spent designing proper system
+  costs of properly handling these cases are not worth it. Time is better spent designing proper system
   controls that shed load if resource usage becomes too high, etc.
 * The "less is more" error handling philosophy described in the previous two points is primarily
   based on the fact that restarts are designed to be fast, reliable and cheap.

api/STYLE.md

@@ -131,3 +131,6 @@ the build system to prevent circular dependency formation. Package group
 `//envoy/api/v2:friends` selects consumers of the core API package (services and configs)
 and is the default visibility for the core API packages. The default visibility
 for services and configs should be `//docs` (proto documentation tool).
+
+Extensions should use the regular hierarchy. For example, configuration for network filters belongs
+in a package under `envoy.config.filter.network`.

api/XDS_PROTOCOL.md

@@ -147,7 +147,7 @@ management server will provide the complete state of the LDS/CDS resources in
 each response. An absent `Listener` or `Cluster` will be deleted.
 
 For EDS/RDS, the management server does not need to supply every requested
-resource and may also supply additional, unrequested resources, `resource_names`
+resource and may also supply additional, unrequested resources. `resource_names`
 is only a hint. Envoy will silently ignore any superfluous resources. When a
 requested resource is missing in a RDS or EDS update, Envoy will retain the last
 known value for this resource. The management server may be able to infer all
@@ -166,7 +166,7 @@ For EDS/RDS, Envoy may either generate a distinct stream for each resource of a
 given type (e.g. if each `ConfigSource` has its own distinct upstream cluster
 for a management server), or may combine together multiple resource requests for
 a given resource type when they are destined for the same management server.
-This is left to implementation specifics, management servers should be capable
+While this is left to implementation specifics, management servers should be capable
 of handling one or more `resource_names` for a given resource type in each
 request. Both sequence diagrams below are valid for fetching two EDS resources
 `{foo, bar}`:
@@ -285,6 +285,51 @@ admin:
 
 ```
 
+### Incremental xDS
+
+Incremental xDS is a separate xDS endpoint available for ADS, CDS and RDS that
+allows:
+
+  * Incremental updates of the list of tracked resources by the xDS client.
+    This supports Envoy on-demand / lazily requesting additional resources. For
+    example, this may occur when a request corresponding to an unknown cluster
+    arrives.
+  * The xDS server can incremetally update the resources on the client.
+    This supports the goal of scalability of xDS resources. Rather than deliver
+    all 100k clusters when a single cluster is modified, the management server
+    only needs to deliver the single cluster that changed.
+
+An xDS incremental session is always in the context of a gRPC bidirectional
+stream. This allows the xDS server to keep track of the state of xDS clients
+connected to it. There is no REST version of Incremental xDS.
+
+In incremental xDS the nonce field is required and used to pair a
+[`IncrementalDiscoveryResponse`](https://www.envoyproxy.io/docs/envoy/latest/api-v2/api/v2/discovery.proto#discoveryrequest)
+to a [`IncrementalDiscoveryRequest`](https://www.envoyproxy.io/docs/envoy/latest/api-v2/api/v2/discovery.proto#discoveryrequest)
+ACK or NACK.
+Optionally, a response message level system_version_info is present for
+debugging purposes only.
+
+`IncrementalDiscoveryRequest` can be sent in 3 situations:
+  1. Initial message in a xDS bidirectional gRPC stream.
+  2. As an ACK or NACK response to a previous `IncrementalDiscoveryResponse`.
+     In this case the `response_nonce` is set to the nonce value in the Response.
+     ACK or NACK is determined by the absence or presence of `error_detail`.
+  3. Spontaneous `IncrementalDiscoveryRequest` from the client.
+     This can be done to dynamically add or remove elements from the tracked
+     `resource_names` set. In this case `response_nonce` must be omitted.
+
+In this first example the client connects and receives a first update that it
+ACKs. The second update fails and the client NACKs the update. Later the xDS
+client spontaneously requests the "wc" resource.
+
+![Incremental session example](diagrams/incremental.svg)
+
+On reconnect the xDS Incremental client may tell the server of its known resources
+to avoid resending them over the network.
+
+![Incremental reconnect example](diagrams/incremental-reconnect.svg)
+
 ## REST-JSON polling subscriptions
 
 Synchronous (long) polling via REST endpoints is also available for the xDS

api/bazel/api_build_system.bzl

@@ -1,23 +1,22 @@
 load("@com_google_protobuf//:protobuf.bzl", "py_proto_library")
 load("@com_lyft_protoc_gen_validate//bazel:pgv_proto_library.bzl", "pgv_cc_proto_library")
-load("@io_bazel_rules_go//proto:def.bzl", "go_proto_library", "go_grpc_library")
+load("@io_bazel_rules_go//proto:def.bzl", "go_grpc_library", "go_proto_library")
 load("@io_bazel_rules_go//go:def.bzl", "go_test")
 
-_PY_SUFFIX="_py"
-_CC_SUFFIX="_cc"
-_GO_PROTO_SUFFIX="_go_proto"
-_GO_GRPC_SUFFIX="_go_grpc"
-_GO_IMPORTPATH_PREFIX="github.com/envoyproxy/data-plane-api/api/"
+_PY_SUFFIX = "_py"
+_CC_SUFFIX = "_cc"
+_GO_PROTO_SUFFIX = "_go_proto"
+_GO_GRPC_SUFFIX = "_go_grpc"
+_GO_IMPORTPATH_PREFIX = "github.com/envoyproxy/data-plane-api/api/"
 
 def _Suffix(d, suffix):
-  return d + suffix
+    return d + suffix
 
 def _LibrarySuffix(library_name, suffix):
-  # Transform //a/b/c to //a/b/c:c in preparation for suffix operation below.
-  if library_name.startswith("//") and ":" not in library_name:
-      library_name += ":" + Label(library_name).name
-  return _Suffix(library_name, suffix)
-
+    # Transform //a/b/c to //a/b/c:c in preparation for suffix operation below.
+    if library_name.startswith("//") and ":" not in library_name:
+        library_name += ":" + Label(library_name).name
+    return _Suffix(library_name, suffix)
 
 # TODO(htuch): has_services is currently ignored but will in future support
 # gRPC stub generation.
@@ -32,6 +31,7 @@ def api_py_proto_library(name, srcs = [], deps = [], has_services = 0):
         protoc = "@com_google_protobuf//:protoc",
         deps = [_LibrarySuffix(d, _PY_SUFFIX) for d in deps] + [
             "@com_lyft_protoc_gen_validate//validate:validate_py",
+            "@googleapis//:api_httpbody_protos_py",
             "@googleapis//:http_api_protos_py",
             "@googleapis//:rpc_status_protos_py",
             "@com_github_gogo_protobuf//:gogo_proto_py",
@@ -54,7 +54,7 @@ def api_go_proto_library(name, proto, deps = []):
             "@com_github_golang_protobuf//ptypes/any:go_default_library",
             "@com_lyft_protoc_gen_validate//validate:go_default_library",
             "@googleapis//:rpc_status_go_proto",
-        ]
+        ],
     )
 
 def api_go_grpc_library(name, proto, deps = []):
@@ -71,9 +71,19 @@ def api_go_grpc_library(name, proto, deps = []):
             "@com_github_golang_protobuf//ptypes/any:go_default_library",
             "@com_lyft_protoc_gen_validate//validate:go_default_library",
             "@googleapis//:http_api_go_proto",
-        ]
+        ],
     )
 
+# This is api_proto_library plus some logic internal to //envoy/api.
+def api_proto_library_internal(visibility = ["//visibility:private"], **kwargs):
+    # //envoy/docs/build.sh needs visibility in order to generate documents.
+    if visibility == ["//visibility:private"]:
+        visibility = ["//docs"]
+    elif visibility != ["//visibility:public"]:
+        visibility = visibility + ["//docs"]
+
+    api_proto_library(visibility = visibility, **kwargs)
+
 # TODO(htuch): has_services is currently ignored but will in future support
 # gRPC stub generation.
 # TODO(htuch): Automatically generate go_proto_library and go_grpc_library
@@ -86,11 +96,6 @@ def api_proto_library(name, visibility = ["//visibility:private"], srcs = [], de
     # it can play well with the PGV plugin and (2) other language support that
     # can make use of native proto_library.
 
-    if visibility == ["//visibility:private"]:
-      visibility = ["//docs"]
-    elif visibility != ["//visibility:public"]:
-      visibility = visibility + ["//docs"]
-
     native.proto_library(
         name = name,
         srcs = srcs,
@@ -102,13 +107,15 @@ def api_proto_library(name, visibility = ["//visibility:private"], srcs = [], de
             "@com_google_protobuf//:struct_proto",
             "@com_google_protobuf//:timestamp_proto",
             "@com_google_protobuf//:wrappers_proto",
+            "@googleapis//:api_httpbody_protos_proto",
             "@googleapis//:http_api_protos_proto",
             "@googleapis//:rpc_status_protos_lib",
             "@com_github_gogo_protobuf//:gogo_proto",
             "@com_lyft_protoc_gen_validate//validate:validate_proto",
         ],
         visibility = visibility,
     )
+
     # Under the hood, this is just an extension of the Protobuf library's
     # bespoke cc_proto_library. It doesn't consume proto_library as a proto
     # provider. Hopefully one day we can move to a model where this target and
@@ -126,7 +133,7 @@ def api_proto_library(name, visibility = ["//visibility:private"], srcs = [], de
         visibility = ["//visibility:public"],
     )
     if (require_py == 1):
-      api_py_proto_library(name, srcs, deps, has_services)
+        api_py_proto_library(name, srcs, deps, has_services)
 
 def api_cc_test(name, srcs, proto_deps):
     native.cc_test(