-
Notifications
You must be signed in to change notification settings - Fork 5.5k
access log: add the detected protocol set #10524
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -61,12 +61,15 @@ FormatterPtr AccessLogFormatUtils::defaultAccessLogFormatter() { | |
| return FormatterPtr{new FormatterImpl(DEFAULT_FORMAT)}; | ||
| } | ||
|
|
||
| const std::string& | ||
| AccessLogFormatUtils::protocolToString(const absl::optional<Http::Protocol>& protocol) { | ||
| if (protocol) { | ||
| const std::string AccessLogFormatUtils::protocolToString(absl::Span<const std::string> protocols) { | ||
| auto protocol = Http::Utility::getProtocol(protocols); | ||
| if (protocol.has_value()) { | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Why do we need to do the string to enum to string dance here? Can't we just return the protocol string directly? Or this is for HTTP back compat? If for back compat can you add comments and also maybe skip the final
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Mostly for backwards compatibility. I can change the interface to return both the enum and the string.
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think trying to mix strings and the protocol enum is going to be confusing and it already involves a lot of converting back and forth between the string and enum representation. Perhaps a better approach would be to rename the existing Otherwise I think we'd need to work out a proper registry. I think that's feasible as long as we the identifiers are not exposed outside a given Envoy process (e.g. the X extension's ID may change from build-to-build but as long as the id is kept internal it shouldn't matter). We might even be able to arrange registrations of core protocols in a stable order so their ids wouldn't change from build-to-build.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Got it. I'll preserve protocol() for HTTP as-is and add two more stubs to deal with the layered protocols. Thank you! I can also do the rename separately (maybe there's some clang tool to automate that) to make reviews easier.
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. +1 this sounds like a good plan. Thank you! |
||
| return Http::Utility::getProtocolString(protocol.value()); | ||
| } | ||
| return UnspecifiedValueString; | ||
| if (protocols.empty()) { | ||
| return UnspecifiedValueString; | ||
| } | ||
| return absl::StrJoin(protocols, ","); | ||
| } | ||
|
|
||
| const std::string AccessLogFormatUtils::getHostname() { | ||
|
|
@@ -558,7 +561,7 @@ StreamInfoFormatter::StreamInfoFormatter(const std::string& field_name) { | |
| } else if (field_name == "PROTOCOL") { | ||
| field_extractor_ = std::make_unique<StreamInfoStringFieldExtractor>( | ||
| [](const StreamInfo::StreamInfo& stream_info) { | ||
| return AccessLogFormatUtils::protocolToString(stream_info.protocol()); | ||
| return AccessLogFormatUtils::protocolToString(stream_info.protocols()); | ||
| }); | ||
| } else if (field_name == "RESPONSE_CODE") { | ||
| field_extractor_ = std::make_unique<StreamInfoUInt64FieldExtractor>( | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -357,7 +357,9 @@ Network::FilterStatus ConnectionManagerImpl::onData(Buffer::Instance& data, bool | |
| } | ||
|
|
||
| Network::FilterStatus ConnectionManagerImpl::onNewConnection() { | ||
| if (!read_callbacks_->connection().streamInfo().protocol()) { | ||
| read_callbacks_->connection().streamInfo().addProtocol( | ||
| StreamInfo::ProtocolStrings::get().HttpString); | ||
| if (!Http::Utility::getProtocol(read_callbacks_->connection().streamInfo().protocols())) { | ||
| // For Non-QUIC traffic, continue passing data to filters. | ||
| return Network::FilterStatus::Continue; | ||
| } | ||
|
|
@@ -539,7 +541,7 @@ ConnectionManagerImpl::ActiveStream::ActiveStream(ConnectionManagerImpl& connect | |
| request_response_timespan_(new Stats::HistogramCompletableTimespanImpl( | ||
| connection_manager_.stats_.named_.downstream_rq_time_, connection_manager_.timeSource())), | ||
| stream_info_(connection_manager_.codec_->protocol(), connection_manager_.timeSource(), | ||
| connection_manager.filterState()), | ||
| connection_manager_.read_callbacks_->connection().streamInfo()), | ||
| upstream_options_(std::make_shared<Network::Socket::Options>()) { | ||
| ASSERT(!connection_manager.config_.isRoutable() || | ||
| ((connection_manager.config_.routeConfigProvider() == nullptr && | ||
|
|
@@ -814,7 +816,9 @@ void ConnectionManagerImpl::ActiveStream::decodeHeaders(RequestHeaderMapPtr&& he | |
| // HTTP/1.1. | ||
| // | ||
| // The protocol may have shifted in the HTTP/1.0 case so reset it. | ||
| stream_info_.protocol(protocol); | ||
| if (Http::Utility::getProtocol(stream_info_.protocols()) != protocol) { | ||
| stream_info_.addProtocol(Http::Utility::getProtocolString(protocol)); | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is going to change the behavior, right? Meaning we will now have multiple HTTP protocol strings?
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It is always appended and |
||
| } | ||
| if (!connection_manager_.config_.http1Settings().accept_http_10_) { | ||
| // Send "Upgrade Required" if HTTP/1.0 support is not explicitly configured on. | ||
| sendLocalReply(false, Code::UpgradeRequired, "", nullptr, state_.is_head_request_, | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -102,9 +102,6 @@ class ConnectionManagerImpl : Logger::Loggable<Logger::Id::http>, | |
|
|
||
| TimeSource& timeSource() { return time_source_; } | ||
|
|
||
| // Return a reference to the shared_ptr so that it can be lazy created on demand. | ||
| std::shared_ptr<StreamInfo::FilterState>& filterState() { return filter_state_; } | ||
|
|
||
| private: | ||
| struct ActiveStream; | ||
|
|
||
|
|
@@ -787,7 +784,6 @@ class ConnectionManagerImpl : Logger::Loggable<Logger::Id::http>, | |
| const Server::OverloadActionState& overload_stop_accepting_requests_ref_; | ||
| const Server::OverloadActionState& overload_disable_keepalive_ref_; | ||
| TimeSource& time_source_; | ||
| std::shared_ptr<StreamInfo::FilterState> filter_state_; | ||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is the filter state that was (lazily) created by a network filter. I changed it to use the filter state from the connection stream info. Otherwise, it's invisible to access loggers at the connection level. |
||
| }; | ||
|
|
||
| } // namespace Http | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -636,18 +636,34 @@ bool Utility::sanitizeConnectionHeader(Http::RequestHeaderMap& headers) { | |
| const std::string& Utility::getProtocolString(const Protocol protocol) { | ||
| switch (protocol) { | ||
| case Protocol::Http10: | ||
| return Headers::get().ProtocolStrings.Http10String; | ||
| return StreamInfo::ProtocolStrings::get().Http10String; | ||
| case Protocol::Http11: | ||
| return Headers::get().ProtocolStrings.Http11String; | ||
| return StreamInfo::ProtocolStrings::get().Http11String; | ||
| case Protocol::Http2: | ||
| return Headers::get().ProtocolStrings.Http2String; | ||
| return StreamInfo::ProtocolStrings::get().Http2String; | ||
| case Protocol::Http3: | ||
| return Headers::get().ProtocolStrings.Http3String; | ||
| return StreamInfo::ProtocolStrings::get().Http3String; | ||
| } | ||
|
|
||
| NOT_REACHED_GCOVR_EXCL_LINE; | ||
| } | ||
|
|
||
| absl::optional<const Protocol> Utility::getProtocol(absl::Span<const std::string> protocols) { | ||
| for (auto it = protocols.rbegin(); it != protocols.rend(); it++) { | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I assume you are going in reverse for a reason? Comment?
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Sure, sorry if it was not clear but it's basically append-only history of discovered protocols, so looking from the back is right. |
||
| if (*it == StreamInfo::ProtocolStrings::get().Http10String) { | ||
| return Protocol::Http10; | ||
| } else if (*it == StreamInfo::ProtocolStrings::get().Http11String) { | ||
| return Protocol::Http11; | ||
| } else if (*it == StreamInfo::ProtocolStrings::get().Http2String) { | ||
| return Protocol::Http2; | ||
| } else if (*it == StreamInfo::ProtocolStrings::get().Http3String) { | ||
| return Protocol::Http3; | ||
| } | ||
| } | ||
|
|
||
| return {}; | ||
| } | ||
|
|
||
| void Utility::extractHostPathFromUri(const absl::string_view& uri, absl::string_view& host, | ||
| absl::string_view& path) { | ||
| /** | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -10,6 +10,7 @@ | |
|
|
||
| #include "common/common/assert.h" | ||
| #include "common/common/dump_state_utils.h" | ||
| #include "common/http/utility.h" | ||
| #include "common/stream_info/filter_state_impl.h" | ||
|
|
||
| namespace Envoy { | ||
|
|
@@ -23,17 +24,22 @@ struct StreamInfoImpl : public StreamInfo { | |
|
|
||
| StreamInfoImpl(Http::Protocol protocol, TimeSource& time_source) | ||
| : time_source_(time_source), start_time_(time_source.systemTime()), | ||
| start_time_monotonic_(time_source.monotonicTime()), protocol_(protocol), | ||
| start_time_monotonic_(time_source.monotonicTime()), | ||
| protocols_({Http::Utility::getProtocolString(protocol)}), | ||
| filter_state_(std::make_shared<FilterStateImpl>(FilterState::LifeSpan::FilterChain)) {} | ||
|
|
||
| StreamInfoImpl(Http::Protocol protocol, TimeSource& time_source, | ||
| std::shared_ptr<FilterState>& parent_filter_state) | ||
| StreamInfoImpl(Http::Protocol protocol, TimeSource& time_source, StreamInfo& parent) | ||
| : time_source_(time_source), start_time_(time_source.systemTime()), | ||
| start_time_monotonic_(time_source.monotonicTime()), protocol_(protocol), | ||
| start_time_monotonic_(time_source.monotonicTime()), | ||
| filter_state_(std::make_shared<FilterStateImpl>( | ||
| FilterStateImpl::LazyCreateAncestor(parent_filter_state, | ||
| FilterStateImpl::LazyCreateAncestor(parent.mutableFilterState(), | ||
| FilterState::LifeSpan::DownstreamConnection), | ||
| FilterState::LifeSpan::FilterChain)) {} | ||
| FilterState::LifeSpan::FilterChain)) { | ||
| for (const auto& protocol : parent.protocols()) { | ||
| protocols_.push_back(protocol); | ||
| } | ||
| protocols_.push_back(Http::Utility::getProtocolString(protocol)); | ||
| } | ||
|
|
||
| SystemTime startTime() const override { return start_time_; } | ||
|
|
||
|
|
@@ -108,9 +114,11 @@ struct StreamInfoImpl : public StreamInfo { | |
|
|
||
| uint64_t bytesReceived() const override { return bytes_received_; } | ||
|
|
||
| absl::optional<Http::Protocol> protocol() const override { return protocol_; } | ||
| absl::Span<const std::string> protocols() const override { return protocols_; } | ||
|
|
||
| void protocol(Http::Protocol protocol) override { protocol_ = protocol; } | ||
| void addProtocol(absl::string_view protocol) override { | ||
| protocols_.push_back(std::string(protocol)); | ||
| } | ||
|
|
||
| absl::optional<uint32_t> responseCode() const override { return response_code_; } | ||
|
|
||
|
|
@@ -216,6 +224,7 @@ struct StreamInfoImpl : public StreamInfo { | |
| (*metadata_.mutable_filter_metadata())[name].MergeFrom(value); | ||
| }; | ||
|
|
||
| FilterStateSharedPtr& mutableFilterState() override { return filter_state_; } | ||
| const FilterStateSharedPtr& filterState() override { return filter_state_; } | ||
| const FilterState& filterState() const override { return *filter_state_; } | ||
|
|
||
|
|
@@ -248,7 +257,8 @@ struct StreamInfoImpl : public StreamInfo { | |
|
|
||
| void dumpState(std::ostream& os, int indent_level = 0) const { | ||
| const char* spaces = spacesForLevel(indent_level); | ||
| os << spaces << "StreamInfoImpl " << this << DUMP_OPTIONAL_MEMBER(protocol_) | ||
| os << spaces << "StreamInfoImpl " << this | ||
| << DUMP_OPTIONAL_MEMBER(Http::Utility::getProtocol(protocols_)) | ||
| << DUMP_OPTIONAL_MEMBER(response_code_) << DUMP_OPTIONAL_MEMBER(response_code_details_) | ||
| << DUMP_MEMBER(health_check_request_) << DUMP_MEMBER(route_name_) << "\n"; | ||
| } | ||
|
|
@@ -271,7 +281,7 @@ struct StreamInfoImpl : public StreamInfo { | |
| absl::optional<MonotonicTime> last_downstream_tx_byte_sent_; | ||
| absl::optional<MonotonicTime> final_time_; | ||
|
|
||
| absl::optional<Http::Protocol> protocol_; | ||
| std::vector<std::string> protocols_; | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I would recommend an InlinedVector here with a small number of protocols.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks. I am relying on inlined strings to save some space. |
||
| absl::optional<uint32_t> response_code_; | ||
| absl::optional<std::string> response_code_details_; | ||
| uint64_t response_flags_{}; | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It doesn't thrill me to expose a reference to the shared_ptr on the interface? Can you explain why this is needed?
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
HTTP request stream info takes TCP stream info filter state by a mutable shared_ptr. I think an additional filter state was created before that was used for HTTP parent filter state, which does not feel right to me. I'm not sure why HTTP stream info is allowed to modify TCP stream info filter state.