Skip to content
Merged
Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
53 commits
Select commit Hold shift + click to select a range
f85d68d
router: add start timestamp header config
dio May 2, 2018
b0b741b
docs: update release notes
dio May 2, 2018
4c80a6d
review: add comment
dio May 8, 2018
3110ce0
review: test that the x-request-start header is not present before th…
dio May 8, 2018
42bd291
review: use header manipulation facility instead
dio May 9, 2018
8fcc471
Merge remote-tracking branch 'upstream/master'
dio May 9, 2018
4e74570
Add a test sample to TestParseInternal
dio May 9, 2018
79a3320
tsan: check RatelimitIntegrationTest failure
dio May 9, 2018
f94e020
Rename it to START_TIME_SINCE_EPOCH
dio May 9, 2018
07dc56f
Fix format
dio May 10, 2018
91fb3da
Merge remote-tracking branch 'upstream/master'
dio May 14, 2018
aef2ed9
Allow header parser to accept START_TIME(ARG)
dio May 14, 2018
1970913
Add replacers
dio May 14, 2018
9004414
Use %f as subsecond formatter
dio May 15, 2018
dd15262
Merge remote-tracking branch 'upstream/master'
dio May 15, 2018
ce1d3a0
Add test in accesss_log_formatter
dio May 16, 2018
e98f1a4
Retry ipv6_test
dio May 16, 2018
edbc437
Add a parser to initialize the format_string_
dio May 24, 2018
8919c50
Add a benchmark for date formatter with subseconds
dio May 24, 2018
908fc61
Merge remote-tracking branch 'upstream/master'
dio May 24, 2018
0f417da
Remove unused typedef
dio May 24, 2018
106c3e2
Use shared ptr to cache the format
dio May 24, 2018
cc6c7c9
Try to fix asan build take 1
dio May 24, 2018
4dcfeb0
Try to fix asan build take 2
dio May 24, 2018
7d53b28
Try to fix asan build take 3
dio May 25, 2018
e6cb04e
Copy the ns value
dio May 25, 2018
96eb0c6
Update release notes
dio May 25, 2018
f475839
review: simplify parsing step
dio May 29, 2018
724a203
review: reuse access log's formatter
dio May 29, 2018
24518ef
Use map to cache access log formatter
dio May 29, 2018
0f0b1bf
review: update docs
dio May 29, 2018
f0fa285
Merge remote-tracking branch 'upstream/master'
dio May 29, 2018
f78e7ad
Fix additional specifiers table format
dio May 29, 2018
7aeb767
Use [1-9] instead of {:digit}
dio May 29, 2018
6310447
Fix doc's indentations
dio May 29, 2018
8a1571f
No more find()
dio May 31, 2018
78e8430
Merge remote-tracking branch 'upstream/master'
dio May 31, 2018
325fdc5
Move subsecond offsets computation
dio Jun 2, 2018
4b9fa9e
Merge remote-tracking branch 'upstream/master'
dio Jun 2, 2018
9aba1e9
Move the offsets calc to fromTimeAndPrepareSubsecondOffsets
dio Jun 3, 2018
94e3add
Cleanup unused deps and vars
dio Jun 3, 2018
612f829
Infer seconds_length only when it is needed
dio Jun 3, 2018
96f915e
Precompute the string segments between specifiers
dio Jun 5, 2018
d5f082b
Try to fix asan on using stack after scope: digits
dio Jun 5, 2018
b85b275
Stamp per cache item.
dio Jun 5, 2018
513d4bf
Fix removing an item of a map while iterating it
dio Jun 5, 2018
9a16090
Remove const PLACEHOLDER
dio Jun 6, 2018
e5642ba
Add comments to CachedTime struct
dio Jun 6, 2018
bda6762
Remove last_segment_ and streamline replacement
dio Jun 6, 2018
ec35f6c
Add test with shorter segments
dio Jun 6, 2018
ea5e62f
Remove % since it is diff on osx vs. linux
dio Jun 6, 2018
22a94a4
Use unordered_map instead
dio Jun 6, 2018
3f01c64
Comment on the possibility to have negative offset value
dio Jun 6, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions docs/root/configuration/http_conn_man/headers.rst
Original file line number Diff line number Diff line change
Expand Up @@ -480,3 +480,6 @@ Supported variable names are:
found, or if the selected value is not a supported type, then no header is emitted. The
namespace and key(s) are specified as a JSON array of strings. Finally, percent symbols in the
parameters **do not** need to be escaped by doubling them.

%START_TIMESTAMP%

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We already have START_TIME in access logs. I wonder, could we just use START_TIME and allow a custom format specifier that is effectively milliseconds since epoch? Then we could use the same code in both places. If we can't do that what about START_TIME_SINCE_EPOCH or something? Though I like the former better. Thoughts?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mattklein123 do you think specifically handling since_epoch_ms in start time formatting makes sense?

i.e. the start time formatting is either:

  1. Using std::put_time's fmt string for %START_TIME(%Y/%m/%dT%H:%M:%S%z %s)%, or
  2. Using fmt::format for %START_TIME(since_epoch_ms)%

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

However, I think for now we probably can go with START_TIME_SINCE_EPOCH for header formatter, since currently the way AccessLogFormatParser build the formatter is a bit isolated to itself. WDYT?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is somewhat related to #3256. It would be nice to support custom formatters apart from what is currently supported. cc @bplotnick.

I think if we did this we could support time since epoch and some other things and I think this is a better long term direction, though more work. Are you interested in picking up #3256 as well and working on both?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mattklein123 sure let me take a look at the possibilities.

The timestamp when the first byte of a request is received. The timestamp unit is in milliseconds.
2 changes: 2 additions & 0 deletions docs/root/intro/version_history.rst
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,8 @@ Version history
* logger: added the ability to optionally set the log format via the :option:`--log-format` option.
* logger: all :ref:`logging levels <operations_admin_interface_logging>` can be configured
at run-time: trace debug info warning error critical.
* router: added `START_TIMESTAMP` as one of supported variables in :ref:`header
formatters <config_http_conn_man_headers_custom_request_headers>`. The timestamp unit is in milliseconds.
* sockets: added :ref:`capture transport socket extension <operations_traffic_capture>` to support
recording plain text traffic and PCAP generation.
* sockets: added `IP_FREEBIND` socket option support for :ref:`listeners
Expand Down
7 changes: 7 additions & 0 deletions source/common/router/header_formatter.cc
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,13 @@ RequestInfoHeaderFormatter::RequestInfoHeaderFormatter(absl::string_view field_n
return RequestInfo::Utility::formatDownstreamAddressNoPort(
*request_info.downstreamLocalAddress());
};
} else if (field_name == "START_TIMESTAMP") {
field_extractor_ = [](const RequestInfo::RequestInfo& request_info) {
// Returns the start timestamp in milliseconds.
return fmt::format("{}", std::chrono::duration_cast<std::chrono::milliseconds>(
request_info.startTime().time_since_epoch())
.count());
};
} else if (field_name.find_first_of("UPSTREAM_METADATA") == 0) {
field_extractor_ =
parseUpstreamMetadataField(field_name.substr(STATIC_STRLEN("UPSTREAM_METADATA")));
Expand Down
31 changes: 31 additions & 0 deletions test/common/router/header_formatter_test.cc
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
#include "gtest/gtest.h"

using testing::NiceMock;
using testing::Return;
using testing::ReturnPointee;
using testing::ReturnRef;

Expand Down Expand Up @@ -251,6 +252,12 @@ TEST(HeaderParserTest, TestParseInternal) {
absl::optional<std::string> expected_exception_;
};

time_t start_time_epoch = 1522280158;
SystemTime start_time = std::chrono::system_clock::from_time_t(start_time_epoch);
const std::string timestamp = fmt::format(
"{}",
std::chrono::duration_cast<std::chrono::milliseconds>(start_time.time_since_epoch()).count());

static const TestCase test_cases[] = {
// Valid inputs
{"%PROTOCOL%", {"HTTP/1.1"}, {}},
Expand All @@ -270,6 +277,7 @@ TEST(HeaderParserTest, TestParseInternal) {
{"%UPSTREAM_METADATA([\"ns\", \t \"key\"])%", {"value"}, {}},
{"%UPSTREAM_METADATA([\"ns\", \n \"key\"])%", {"value"}, {}},
{"%UPSTREAM_METADATA( \t [ \t \"ns\" \t , \t \"key\" \t ] \t )%", {"value"}, {}},
{"%START_TIMESTAMP%", {timestamp}, {}},

// Unescaped %
{"%", {}, {"Invalid header configuration. Un-escaped % at position 0"}},
Expand Down Expand Up @@ -404,6 +412,8 @@ TEST(HeaderParserTest, TestParseInternal) {
)EOF");
ON_CALL(*host, metadata()).WillByDefault(ReturnRef(metadata));

ON_CALL(request_info, startTime()).WillByDefault(Return(start_time));

for (const auto& test_case : test_cases) {
Protobuf::RepeatedPtrField<envoy::api::v2::core::HeaderValueOption> to_add;
envoy::api::v2::core::HeaderValueOption* header = to_add.Add();
Expand Down Expand Up @@ -595,6 +605,10 @@ TEST(HeaderParserTest, EvaluateHeadersWithAppendFalse) {
{
"key": "x-client-ip",
"value": "%CLIENT_IP%"
},
{
"key": "x-request-start",
"value": "%START_TIMESTAMP%"
}
]
}
Expand All @@ -604,19 +618,30 @@ TEST(HeaderParserTest, EvaluateHeadersWithAppendFalse) {
envoy::api::v2::route::RouteAction route_action = parseRouteFromJson(json).route();
route_action.mutable_request_headers_to_add(0)->mutable_append()->set_value(false);
route_action.mutable_request_headers_to_add(1)->mutable_append()->set_value(false);
route_action.mutable_request_headers_to_add(2)->mutable_append()->set_value(false);

HeaderParserPtr req_header_parser =
Router::HeaderParser::configure(route_action.request_headers_to_add());
Http::TestHeaderMapImpl headerMap{
{":method", "POST"}, {"static-header", "old-value"}, {"x-client-ip", "0.0.0.0"}};

NiceMock<Envoy::RequestInfo::MockRequestInfo> request_info;
time_t start_time_epoch = 1522280158;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: const here and below

SystemTime start_time = std::chrono::system_clock::from_time_t(start_time_epoch);
EXPECT_CALL(request_info, startTime()).WillOnce(Return(start_time));

req_header_parser->evaluateHeaders(headerMap, request_info);
EXPECT_TRUE(headerMap.has("static-header"));
EXPECT_EQ("static-value", headerMap.get_("static-header"));
EXPECT_TRUE(headerMap.has("x-client-ip"));
EXPECT_EQ("127.0.0.1", headerMap.get_("x-client-ip"));

uint64_t timestamp;
StringUtil::atoul(headerMap.get_("x-request-start").c_str(), timestamp);
EXPECT_EQ(
timestamp,
std::chrono::duration_cast<std::chrono::milliseconds>(start_time.time_since_epoch()).count());

typedef std::map<std::string, int> CountMap;
CountMap counts;
headerMap.iterate(
Expand All @@ -635,6 +660,7 @@ TEST(HeaderParserTest, EvaluateHeadersWithAppendFalse) {

EXPECT_EQ(1, counts["static-header"]);
EXPECT_EQ(1, counts["x-client-ip"]);
EXPECT_EQ(1, counts["x-request-start"]);
}

TEST(HeaderParserTest, EvaluateResponseHeaders) {
Expand All @@ -647,6 +673,10 @@ match: { prefix: "/new_endpoint" }
key: "x-client-ip"
value: "%CLIENT_IP%"
append: true
- header:
key: "x-request-start"
value: "%START_TIMESTAMP%"
append: true
response_headers_to_remove: ["x-nope"]
)EOF";

Expand All @@ -657,6 +687,7 @@ match: { prefix: "/new_endpoint" }
NiceMock<Envoy::RequestInfo::MockRequestInfo> request_info;
resp_header_parser->evaluateHeaders(headerMap, request_info);
EXPECT_TRUE(headerMap.has("x-client-ip"));
EXPECT_TRUE(headerMap.has("x-request-start"));
EXPECT_TRUE(headerMap.has("x-safe"));
EXPECT_FALSE(headerMap.has("x-nope"));
}
Expand Down