Add access logging to tcp_proxy

docs/configuration/http_conn_man/access_log.rst → docs/configuration/access_log.rst

@@ -1,4 +1,4 @@
-.. _config_http_conn_man_access_log:
+.. _config_access_log:
 
 Access logging
 ==============
@@ -7,7 +7,7 @@ Configuration
 -------------------------
 
 Access logs are configured as part of the :ref:`HTTP connection manager config
-<config_http_conn_man>`.
+<config_http_conn_man>` or :ref:`TCP Proxy <config_network_filters_tcp_proxy>`.
 
 .. code-block:: json
 
@@ -21,71 +21,102 @@ Access logs are configured as part of the :ref:`HTTP connection manager config
     ]
   }
 
-.. _config_http_conn_man_access_log_path_param:
+.. _config_access_log_path_param:
 
 path
   *(required, string)* Path the access log is written to.
 
-.. _config_http_conn_man_access_log_format_param:
+.. _config_access_log_format_param:
 
 format
   *(optional, string)* Access log format. Envoy supports :ref:`custom access log formats
-  <config_http_con_manager_access_log_format>` as well as a :ref:`default format
-  <config_http_con_manager_access_log_default_format>`.
+  <config_access_log_format>` as well as a :ref:`default format
+  <config_access_log_default_format>`.
 
-.. _config_http_conn_man_access_log_filter_param:
+.. _config_access_log_filter_param:
 
 filter
   *(optional, object)* :ref:`Filter <config_http_con_manager_access_log_filters>` which is used to
   determine if the access log needs to be written.
 
-.. _config_http_con_manager_access_log_format:
+.. _config_access_log_format:
 
 Format rules
 ------------
 
 The access log format string contains either command operators or other characters interpreted as a
 plain string. The access log formatter does not make any assumptions about a new line separator, so one
 has to specified as part of the format string.
-See the :ref:`default format <config_http_con_manager_access_log_default_format>` for an example.
+See the :ref:`default format <config_access_log_default_format>` for an example.
 Note that the access log line will contain a '-' character for every not set/empty value.
 
+The same format strings are used by different types of access logs (such as HTTP and TCP).  Some
+fields may have slightly different meanings, depending on what type of log it is.  Differences
+are noted.
+
 The following command operators are supported:
 
 %START_TIME%
-  Request start time including milliseconds.
+  HTTP
+    Request start time including milliseconds.
+
+  TCP
+    Downstream connection start time including milliseconds.
 
 %BYTES_RECEIVED%
-  Body bytes received.
+  HTTP
+    Body bytes received.
+
+  TCP
+    Downstream bytes received on connection.
 
 %PROTOCOL%
-  Protocol. Currently either *HTTP/1.1* or *HTTP/2*.
+  HTTP
+    Protocol. Currently either *HTTP/1.1* or *HTTP/2*.
+
+  TCP
+    Not implemented ("-").
 
 %RESPONSE_CODE%
-  HTTP response code. Note that a response code of '0' means that the server never sent the
-  beginning of a response. This generally means that the (downstream) client disconnected.
+  HTTP
+    HTTP response code. Note that a response code of '0' means that the server never sent the
+    beginning of a response. This generally means that the (downstream) client disconnected.
+
+  TCP
+    Not implemented ("-").
 
 %BYTES_SENT%
-  Body bytes sent.
+  HTTP
+    Body bytes sent.
+
+  TCP
+    Downstream bytes sent on connection.
 
 %DURATION%
-  Total duration in milliseconds of the request from the start time to the last byte out.
+  HTTP
+    Total duration in milliseconds of the request from the start time to the last byte out.
+
+  TCP
+    Total duration of the downstream connection.
 
 %RESPONSE_FLAGS%
-  Additional details about the response, if any. Possible values are:
-
-  * **LH**: Local service failed :ref:`health check request <arch_overview_health_checking>` in addition to 503 response code.
-  * **UH**: No healthy upstream hosts in upstream cluster in addition to 503 response code.
-  * **UT**: Upstream request timeout in addition to 504 response code.
-  * **LR**: Connection local reset in addition to 503 response code.
-  * **UR**: Upstream remote reset in addition to 503 response code.
-  * **UF**: Upstream connection failure in addition to 503 response code.
-  * **UC**: Upstream connection termination in addition to 503 response code.
-  * **UO**: Upstream overflow (:ref:`circuit breaking <arch_overview_circuit_break>`) in addition to 503 response code.
-  * **NR**: No :ref:`route configured <arch_overview_http_routing>` for a given request in addition to 404 response code.
-  * **DI**: The request processing was delayed for a period specified via :ref:`fault injection <config_http_filters_fault_injection>`.
-  * **FI**: The request was aborted with a response code specified via :ref:`fault injection <config_http_filters_fault_injection>`.
-  * **RL**: The request was ratelimited locally by the :ref:`HTTP rate limit filter <config_http_filters_rate_limit>` in addition to 429 response code.
+  Additional details about the response or connection, if any. For TCP connections, the specified response code
+  does not apply.  Possible values are:
+
+  HTTP and TCP
+    * **UH**: No healthy upstream hosts in upstream cluster in addition to 503 response code.
+    * **UF**: Upstream connection failure in addition to 503 response code.
+    * **UO**: Upstream overflow (:ref:`circuit breaking <arch_overview_circuit_break>`) in addition to 503 response code.
+    * **NR**: No :ref:`route configured <arch_overview_http_routing>` for a given request in addition to 404 response code.
+  HTTP only
+    * **LH**: Local service failed :ref:`health check request <arch_overview_health_checking>` in addition to 503 response code.
+    * **UT**: Upstream request timeout in addition to 504 response code.
+    * **LR**: Connection local reset in addition to 503 response code.
+    * **UR**: Upstream remote reset in addition to 503 response code.
+    * **UC**: Upstream connection termination in addition to 503 response code.
+    * **DI**: The request processing was delayed for a period specified via :ref:`fault injection <config_http_filters_fault_injection>`.
+    * **FI**: The request was aborted with a response code specified via :ref:`fault injection <config_http_filters_fault_injection>`.
+    * **RL**: The request was ratelimited locally by the :ref:`HTTP rate limit filter <config_http_filters_rate_limit>` in addition to 429 response code.
 
 %UPSTREAM_HOST%
   Upstream host URL (e.g., tcp://ip:port for TCP connections).
@@ -94,15 +125,24 @@ The following command operators are supported:
   Upstream cluster to which the upstream host belongs to.
 
 %REQ(X?Y):Z%
-  An HTTP request header where X is the main HTTP header, Y is the alternative one, and Z is an
-  optional parameter denoting string truncation up to Z characters long. The value is taken from the
-  HTTP request header named X first and if it's not set, then request header Y is used. If none of
-  the headers are present '-' symbol will be in the log.
+  HTTP
+    An HTTP request header where X is the main HTTP header, Y is the alternative one, and Z is an
+    optional parameter denoting string truncation up to Z characters long. The value is taken from
+    the HTTP request header named X first and if it's not set, then request header Y is used. If
+    none of the headers are present '-' symbol will be in the log.
+
+  TCP
+    Not implemented ("-").
 
 %RESP(X?Y):Z%
-  Same as **%REQ(X?Y):Z%** but taken from HTTP response headers.
+  HTTP
+    Same as **%REQ(X?Y):Z%** but taken from HTTP response headers.
+
+  TCP
+    Not implemented ("-").
+
 
-.. _config_http_con_manager_access_log_default_format:
+.. _config_access_log_default_format:
 
 Default format
 --------------

docs/configuration/configuration.rst

@@ -13,4 +13,5 @@ Configuration reference
   http_conn_man/http_conn_man
   http_filters/http_filters
   cluster_manager/cluster_manager
+  access_log
   tools/router_check

docs/configuration/http_conn_man/headers.rst

@@ -203,7 +203,7 @@ is one of the few areas where a thin client library is needed to perform this du
 is out of scope for this documentation. If *x-request-id* is propagated across all hosts, the
 following features are available:
 
-* Stable :ref:`access logging <config_http_conn_man_access_log>` via the
+* Stable :ref:`access logging <config_access_log>` via the
   :ref:`runtime filter<config_http_con_manager_access_log_filters_runtime>`.
 * Stable tracing when performing random sampling via the :ref:`tracing.random_sampling
   <config_http_conn_man_runtime_random_sampling>` runtime setting or via forced tracing using the

docs/configuration/http_conn_man/http_conn_man.rst

@@ -167,8 +167,8 @@ drain_timeout_ms
   when a connection hits the idle timeout or during general server draining. The default grace
   period is 5000 milliseconds (5 seconds) if this option is not specified.
 
-:ref:`access_log <config_http_conn_man_access_log>`
-  *(optional, array)* Configuration for :ref:`HTTP access logs <arch_overview_http_access_logs>`
+:ref:`access_log <config_access_log>`
+  *(optional, array)* Configuration for :ref:`HTTP access logs <arch_overview_access_logs>`
   emitted by the connection manager.
 
 .. _config_http_conn_man_use_remote_address:
@@ -218,7 +218,6 @@ generate_request_id
 
   route_config/route_config
   filters
-  access_log
   tracing
   headers
   header_sanitizing

docs/configuration/network_filters/tcp_proxy_filter.rst

@@ -12,6 +12,7 @@ TCP proxy :ref:`architecture overview <arch_overview_tcp_proxy>`.
     "config": {
       "stat_prefix": "...",
       "route_config": "{...}"
+      "access_log": "[]"
     }
   }
 
@@ -23,6 +24,10 @@ stat_prefix
   *(required, string)* The prefix to use when emitting :ref:`statistics
   <config_network_filters_tcp_proxy_stats>`.
 
+:ref:`access_log <config_access_log>`
+  *(optional, array)* Configuration for :ref:`access logs <arch_overview_access_logs>`
+  emitted by the this tcp_proxy.
+
 .. _config_network_filters_tcp_proxy_route_config:
 
 Route Configuration

docs/intro/arch_overview/access_logging.rst

@@ -0,0 +1,18 @@
+.. _arch_overview_access_logs:
+
+Access logging
+===================
+
+The HTTP connection manager and tcp proxy supports extensible access logging with the following
+features:
+
+* Any number of access logs per connection manager or tcp proxy.
+* Asynchronous IO flushing architecture. Access logging will never block the main network processing
+  threads.
+* Customizable access log formats using predefined fields as well as arbitrary HTTP request and
+  response headers.
+* Customizable access log filters that allow different types of requests and responses to be written
+  to different access logs.
+
+Access log :ref:`configuration <config_access_log>`.
+

docs/intro/arch_overview/arch_overview.rst

@@ -26,6 +26,7 @@ Architecture overview
   runtime
   tracing
   tcp_proxy
+  access_logging
   mongo
   dynamo
   redis

docs/intro/arch_overview/http_connection_management.rst

@@ -8,7 +8,7 @@ large amount of HTTP specific functionality. Envoy has a built in network level
 :ref:`HTTP connection manager <config_http_conn_man>`. This filter translates raw bytes into HTTP
 level messages and events (e.g., headers received, body data received, trailers received, etc.). It
 also handles functionality common to all HTTP connections and requests such as :ref:`access logging
-<arch_overview_http_access_logs>`, :ref:`request ID generation and tracing <arch_overview_tracing>`,
+<arch_overview_access_logs>`, :ref:`request ID generation and tracing <arch_overview_tracing>`,
 :ref:`request/response header manipulation <config_http_conn_man_headers>`, :ref:`route table
 <arch_overview_http_routing>` management, and :ref:`statistics <config_http_conn_man_stats>`.
 
@@ -34,23 +34,6 @@ HTTP header sanitizing
 The HTTP connection manager performs various :ref:`header sanitizing
 <config_http_conn_man_header_sanitizing>` actions for security reasons.
 
-.. _arch_overview_http_access_logs:
-
-HTTP access logging
--------------------
-
-The HTTP connection manager supports extensible access logging with the following features:
-
-* Any number of access logs per connection manager.
-* Asynchronous IO flushing architecture. Access logging will never block the main network processing
-  threads.
-* Customizable access log formats using predefined fields as well as arbitrary HTTP request and
-  response headers.
-* Customizable access log filters that allow different types of requests and responses to be written
-  to different access logs.
-
-HTTP access log :ref:`configuration <config_http_conn_man_access_log>`.
-
 Route table configuration
 -------------------------
 

docs/operations/cli.rst

@@ -98,7 +98,7 @@ following are the command line options that Envoy supports.
   This setting is used during file creation to determine the duration between flushes
   of buffers to files. The buffer will flush every time it gets full, or every time
   the interval has elapsed, whichever comes first. Adjusting this setting is useful
-  when tailing :ref:`access logs <arch_overview_http_access_logs>` in order to
+  when tailing :ref:`access logs <arch_overview_access_logs>` in order to
   get more (or less) immediate flushing.
 
 .. option:: --drain-time-s <integer>

include/envoy/access_log/access_log.h

@@ -88,7 +88,7 @@ class RequestInfo {
    * @return duration from request start to when the entire request was received from the
    * downstream client in microseconds. Note: if unset, will return 0 microseconds.
    */
-  virtual std::chrono::microseconds requestReceivedDuration() const PURE;
+  virtual Optional<std::chrono::microseconds> requestReceivedDuration() const PURE;
 
   /**
    * Set the duration from request start to when the entire request was received from the
@@ -101,7 +101,7 @@ class RequestInfo {
    * @return the duration from request start to when the entire response was received from the
    * upstream host in microseconds. Note: if unset, will return 0 microseconds.
    */
-  virtual std::chrono::microseconds responseReceivedDuration() const PURE;
+  virtual Optional<std::chrono::microseconds> responseReceivedDuration() const PURE;
 
   /**
    * Set the duration from request start to when the entire response was received from the
@@ -118,7 +118,7 @@ class RequestInfo {
   /**
    * @return the protocol of the request.
    */
-  virtual Http::Protocol protocol() const PURE;
+  virtual Optional<Http::Protocol> protocol() const PURE;
 
   /**
    * Set the request's protocol.