-
Notifications
You must be signed in to change notification settings - Fork 5.5k
proxy protocol: new feature auto-detect proxy protocol #18951
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 16 commits
278ce7e
121e87d
2d43ae3
de6b7f2
3a36695
409f94c
ae19e1f
ed4fb54
bcc7aef
8c87bcd
4aa520e
91c9a1b
c45ecc7
4259f1a
2b4a89a
f105c3d
6a0d49b
f47232f
fa9746a
a37ad43
6f19883
0927000
b5cbfc1
a7ef68f
9495853
ffb56d1
ac2982f
2a66a01
c9e086f
af6f88d
7f17660
59206da
b416242
4e4638a
354a365
86ad288
899f199
887d0bc
a23ac5f
6075b67
4d6f75f
2cb4526
1667461
a0c977f
85706e6
c86edef
6031151
a09bb1b
cfeda5a
5a660a0
e92a8a4
84cdc32
f3d65e9
b2d467e
45e9c24
056f74f
39e7b50
392124e
307df1c
5c50e23
36bbfa2
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -40,4 +40,14 @@ message ProxyProtocol { | |
|
|
||
| // The list of rules to apply to requests. | ||
| repeated Rule rules = 1; | ||
|
|
||
| // Allow requests through that don't use proxy protocol. Defaults to false. | ||
| // | ||
| // .. attention:: | ||
| // | ||
| // Only enable if ALL traffic to the listener comes from a trusted source. | ||
| // For more information on the security implications of this feature, see | ||
| // https://www.haproxy.org/download/1.9/doc/proxy-protocol.txt | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. In the doc comment a couple lines up, it references version 2.1's copy of the doc. Please use the same url as that one (or update both of them), so the user isn't left wondering if there's a significant difference between the two links.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. addressed in fa9746a |
||
| // | ||
| bool allow_requests_without_proxy_protocol = 2; | ||
| } | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -65,6 +65,7 @@ New Features | |
| * listener: added support for :ref:`MPTCP <envoy_v3_api_field_config.listener.v3.Listener.enable_mptcp>` (multipath TCP). | ||
| * oauth filter: added :ref:`cookie_names <envoy_v3_api_field_extensions.filters.http.oauth2.v3.OAuth2Credentials.cookie_names>` to allow overriding (default) cookie names (``BearerToken``, ``OauthHMAC``, and ``OauthExpires``) set by the filter. | ||
| * oauth filter: setting IdToken and RefreshToken cookies if they are provided by Identity provider along with AccessToken. | ||
| * proxy protocol: added support for allowing requests without proxy protocol on the listener from trusted downstreams as an opt-in flag. | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Please add a link to the new config setting.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. addressed in 392124e |
||
| * tcp: added a :ref:`FilterState <envoy_v3_api_msg_type.v3.HashPolicy.FilterState>` :ref:`hash policy <envoy_v3_api_msg_type.v3.HashPolicy>`, used by :ref:`TCP proxy <envoy_v3_api_field_extensions.filters.network.tcp_proxy.v3.TcpProxy.hash_policy>` to allow hashing load balancer algorithms to hash on objects in filter state. | ||
| * tcp_proxy: added support to populate upstream http connect header values from stream info. | ||
| * thrift_proxy: add header to metadata filter for turning headers into dynamic metadata. | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -46,7 +46,8 @@ namespace ProxyProtocol { | |
| Config::Config( | ||
| Stats::Scope& scope, | ||
| const envoy::extensions::filters::listener::proxy_protocol::v3::ProxyProtocol& proto_config) | ||
| : stats_{ALL_PROXY_PROTOCOL_STATS(POOL_COUNTER(scope))} { | ||
| : stats_{ALL_PROXY_PROTOCOL_STATS(POOL_COUNTER(scope))}, | ||
| allow_requests_without_proxy_protocol_(proto_config.allow_requests_without_proxy_protocol()) { | ||
| for (const auto& rule : proto_config.rules()) { | ||
| tlv_types_[0xFF & rule.tlv_type()] = rule.on_tlv_present(); | ||
| } | ||
|
|
@@ -63,6 +64,10 @@ const KeyValuePair* Config::isTlvTypeNeeded(uint8_t type) const { | |
|
|
||
| size_t Config::numberOfNeededTlvTypes() const { return tlv_types_.size(); } | ||
|
|
||
| bool Config::allowRequestsWithoutProxyProtocol() const { | ||
| return allow_requests_without_proxy_protocol_; | ||
| } | ||
|
|
||
| Network::FilterStatus Filter::onAccept(Network::ListenerFilterCallbacks& cb) { | ||
| ENVOY_LOG(debug, "proxy_protocol: new connection accepted"); | ||
| Network::ConnectionSocket& socket = cb.socket(); | ||
|
|
@@ -77,11 +82,19 @@ Network::FilterStatus Filter::onAccept(Network::ListenerFilterCallbacks& cb) { | |
| return Network::FilterStatus::StopIteration; | ||
| } | ||
|
|
||
| void Filter::resetAndContinue(Network::IoHandle& io_handle) { | ||
| // Release the file event so that we do not interfere with the connection read events. | ||
| io_handle.resetFileEvents(); | ||
| cb_->continueFilterChain(true); | ||
| } | ||
|
|
||
| void Filter::onRead() { | ||
| const ReadOrParseState read_state = onReadWorker(); | ||
| if (read_state == ReadOrParseState::Error) { | ||
| config_->stats_.downstream_cx_proxy_proto_error_.inc(); | ||
| cb_->continueFilterChain(false); | ||
| } else if (read_state == ReadOrParseState::SkipFilterError) { | ||
| resetAndContinue(cb_->socket().ioHandle()); | ||
| } | ||
| } | ||
|
|
||
|
|
@@ -135,9 +148,7 @@ ReadOrParseState Filter::onReadWorker() { | |
| proxy_protocol_header_.value().remote_address_); | ||
| } | ||
|
|
||
| // Release the file event so that we do not interfere with the connection read events. | ||
| socket.ioHandle().resetFileEvents(); | ||
| cb_->continueFilterChain(true); | ||
| resetAndContinue(socket.ioHandle()); | ||
| return ReadOrParseState::Done; | ||
| } | ||
|
|
||
|
|
@@ -309,7 +320,7 @@ ReadOrParseState Filter::parseExtensions(Network::IoHandle& io_handle, uint8_t* | |
| const auto recv_result = io_handle.recv(buf, to_read, 0); | ||
| if (!recv_result.ok()) { | ||
| if (recv_result.err_->getErrorCode() == Api::IoError::IoErrorCode::Again) { | ||
| return ReadOrParseState::TryAgainLater; | ||
| return ReadOrParseState::TryAgainLaterError; | ||
| } | ||
| ENVOY_LOG(debug, "failed to read proxy protocol (no bytes avail)"); | ||
| return ReadOrParseState::Error; | ||
|
|
@@ -424,7 +435,7 @@ ReadOrParseState Filter::readProxyHeader(Network::IoHandle& io_handle) { | |
|
|
||
| if (!result.ok()) { | ||
| if (result.err_->getErrorCode() == Api::IoError::IoErrorCode::Again) { | ||
| return ReadOrParseState::TryAgainLater; | ||
| return ReadOrParseState::TryAgainLaterError; | ||
| } | ||
| ENVOY_LOG(debug, "failed to read proxy protocol (no bytes read)"); | ||
| return ReadOrParseState::Error; | ||
|
|
@@ -434,6 +445,13 @@ ReadOrParseState Filter::readProxyHeader(Network::IoHandle& io_handle) { | |
| if (nread < 1) { | ||
| ENVOY_LOG(debug, "failed to read proxy protocol (no bytes read)"); | ||
| return ReadOrParseState::Error; | ||
| } else if (nread < PROXY_PROTO_V2_HEADER_LEN && | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't think this handles the case where there is no PROXY header, but the number of bytes read is larger than
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. good catch, thanks! addressed in fa9746a i went ahead and wrote a test that failed (the new one), then removed the faulty logic. now there are two tests (for small and large requests that get passed through the filter) that hit the exact same code path. happy to remove either one, what do you think?
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Please leave both tests in |
||
| config_.get()->allowRequestsWithoutProxyProtocol()) { | ||
| if (nread < PROXY_PROTO_V1_SIGNATURE_LEN || | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Although it is rare case, there still have the chance of receiving the partial v1 signature, like just a 'P' char, the left of the signature received by the second we could return But if we only receive very short data from the client (short than both v1 and v2 signature), then I feel we only can waiting for a timeout of listener filter. But that is bad for the usecase of issue. Another idea is if the
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @soulxu can you update the link from
to a link to main / a commit and where you're proposing this change? I think with the recent commits to this PR that the link provided is dated, I'm not sure where you're proposing we return the error
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. also related #18951 (comment) |
||
| memcmp(buf_, PROXY_PROTO_V1_SIGNATURE, PROXY_PROTO_V1_SIGNATURE_LEN)) { | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. it should be
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. this is actually fine as is. the first if statement in this block ensures that by the time we get here we know if it helps with readability I'm happy to add
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. sorry, you are right. I was based on the logic we want to exam more bytes. |
||
| ENVOY_LOG(debug, "need more bytes to determine if we have a proxy protocol header"); | ||
| return ReadOrParseState::SkipFilterError; | ||
| } | ||
| } | ||
|
|
||
| if (buf_off_ + nread >= PROXY_PROTO_V2_HEADER_LEN) { | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -60,15 +60,22 @@ class Config : public Logger::Loggable<Logger::Id::filter> { | |
| */ | ||
| size_t numberOfNeededTlvTypes() const; | ||
|
|
||
| /** | ||
| * Filter configuration that determines if we should pass-through requests without | ||
| * proxy protocol. Should only be configured to true for trusted downstreams. | ||
| */ | ||
| bool allowRequestsWithoutProxyProtocol() const; | ||
|
|
||
| private: | ||
| absl::flat_hash_map<uint8_t, KeyValuePair> tlv_types_; | ||
| bool allow_requests_without_proxy_protocol_{}; | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
| }; | ||
|
|
||
| using ConfigSharedPtr = std::shared_ptr<Config>; | ||
|
|
||
| enum ProxyProtocolVersion { Unknown = -1, InProgress = -2, V1 = 1, V2 = 2 }; | ||
|
|
||
| enum class ReadOrParseState { Done, TryAgainLater, Error }; | ||
| enum class ReadOrParseState { Done, TryAgainLaterError, Error, SkipFilterError }; | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Why did you rename I think
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. addressed in fa9746a |
||
|
|
||
| /** | ||
| * Implementation the PROXY Protocol listener filter | ||
|
|
@@ -98,7 +105,7 @@ class Filter : public Network::ListenerFilter, Logger::Loggable<Logger::Id::filt | |
| /** | ||
| * Helper function that attempts to read the proxy header | ||
| * (delimited by \r\n if V1 format, or with length if V2) | ||
| * @return bool true valid header, false if more data is needed or socket errors occurred. | ||
| * @return ReadOrParseState Done if successfully parsed, or the error type. | ||
| */ | ||
| ReadOrParseState readProxyHeader(Network::IoHandle& io_handle); | ||
|
|
||
|
|
@@ -118,6 +125,8 @@ class Filter : public Network::ListenerFilter, Logger::Loggable<Logger::Id::filt | |
| bool parseV2Header(char* buf); | ||
| absl::optional<size_t> lenV2Address(char* buf); | ||
|
|
||
| void resetAndContinue(Network::IoHandle& io_handle); | ||
|
|
||
| Network::ListenerFilterCallbacks* cb_{}; | ||
|
|
||
| // The offset in buf_ that has been fully read | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -222,6 +222,19 @@ TEST_P(ProxyProtocolTest, V1Basic) { | |
| disconnect(); | ||
| } | ||
|
|
||
| TEST_P(ProxyProtocolTest, AllowNoProxyProtocol) { | ||
| // allows request through even though it doesn't use proxy protocol | ||
| envoy::extensions::filters::listener::proxy_protocol::v3::ProxyProtocol proto_config; | ||
| proto_config.set_allow_requests_without_proxy_protocol(true); | ||
| connect(true, &proto_config); | ||
|
|
||
| write("more data"); | ||
|
|
||
| expectData("more data"); | ||
|
|
||
| disconnect(); | ||
| } | ||
|
|
||
| TEST_P(ProxyProtocolTest, V1Minimal) { | ||
| connect(); | ||
| write("PROXY UNKNOWN\r\nmore data"); | ||
|
|
@@ -927,6 +940,30 @@ TEST_P(ProxyProtocolTest, PartialRead) { | |
| disconnect(); | ||
| } | ||
|
|
||
| TEST_P(ProxyProtocolTest, PartialV1ReadWithAllowNoProxyProtocol) { | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Also worth testing a case that we have correct v1 or v2 header signature, but invalid data in the rest of header, then ensure we disconnect the connection, and not let the connection passthrough.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. addressed in 6031151 |
||
|
|
||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. nit, useless space line
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. addressed in 1667461 |
||
| envoy::extensions::filters::listener::proxy_protocol::v3::ProxyProtocol proto_config; | ||
| proto_config.set_allow_requests_without_proxy_protocol(true); | ||
| connect(true, &proto_config); | ||
|
|
||
| write("PROXY TCP4"); | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We should test the case of the first write is not full v1 protocol signature, like just one char 'P'
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Addressed in 6a0d49b
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm actually not sure this is possible simultaneously with the fragmented/partial protocol without the possibility for false positive, e.g. TEST_P(ProxyProtocolTest, TinyPartialV1ReadWithAllowNoProxyProtocolThisFails) {
envoy::extensions::filters::listener::proxy_protocol::v3::ProxyProtocol proto_config;
proto_config.set_allow_requests_without_proxy_protocol(true);
connect(true, &proto_config);
write("P"); // matches the beginning of v1 proxy protocol
dispatcher_->run(Event::Dispatcher::RunType::NonBlock);
write("BOGUS");
expectData("PBOGUS"); // we will consume `P` and upstream gets `BOGUS` not `PBOGUS`
disconnect();
}this is why in the initial implementation I required that the initial read from We can leave the PR as it is (or perhaps preferably throw a hard error if we consumed some bytes and then realize we had a false positive?) with the understanding that there could be false positives on identifying proxy protocol, additionally with the understanding that this is quite rare and will not likely happen in practice (except perhaps for single byte reads of Also willing to go back to my original version (before 6a0d49b which addresses the initial concern with partial reads) which addresses the potential false positive here
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm not following. How would the filter consume the
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Running the test above, we do Then before we continue the loop, we read that byte for real (no The only way I see to avoid false positives identifying the header and always send the correct response up the filter chain is to require that the initial
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Oh, I see. I think you need to fix this case. You may have to restructure the code such that no bytes are read (without MSG_PEEK) until we've decided 100% whether this is a proxy header or not. I think it's probably good enough to look for either
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. |
||
|
|
||
| dispatcher_->run(Event::Dispatcher::RunType::NonBlock); | ||
|
|
||
| write(" 254.254.2"); | ||
| write("54.254 1.2"); | ||
| write(".3.4 65535"); | ||
| write(" 1234\r\n..."); | ||
|
|
||
| expectData("..."); | ||
|
|
||
| EXPECT_EQ(server_connection_->connectionInfoProvider().remoteAddress()->ip()->addressAsString(), | ||
| "254.254.254.254"); | ||
| EXPECT_TRUE(server_connection_->connectionInfoProvider().localAddressRestored()); | ||
|
|
||
| disconnect(); | ||
| } | ||
|
|
||
| TEST_P(ProxyProtocolTest, V2PartialRead) { | ||
| // A well-formed ipv4/tcp header, delivered with part of the signature, | ||
| // part of the header, rest of header + body | ||
|
|
||
Uh oh!
There was an error while loading. Please reload this page.