xff: add support for configuring a list of trusted CIDRs #31831

jamesog · 2024-01-15T19:25:10Z

Commit Message: xff: add support for configuring a list of trusted CIDRs

The original client IP address can be determined from the x-forwarded-for header either by a fixed number of trusted hops, or by evaluating the client IP address against a list of trusted addresses.

This adds support for configuring a list of CIDRs in the xff original IP detection extension. The remote IP address is evaluated against these, and optionally recurses through XFF to find the last non-trusted address.

Additional Description:
This feature is generally used by people with a CDN in front of their edge proxy to ensure that XFF is only parsed when the remote connection comes from a CDN server.

The behaviour of the new functionality should be the same as Nginx's realip module.

Disclaimer: This is my first time writing C++ so I'm not certain my changes are completely idiomatic, but I've tried to stick with existing style in the codebase. Feedback very welcome!

Risk Level: Medium
Testing: Unit tests, manual tests
Docs Changes: Updates to HTTP Connection Manager header manipulation docs, and proto docs.
Release Notes: Added to changelogs/current.yaml
Platform Specific Features: None
Fixes #21639
Relates to #31296

X-Forwarded-For parsing can either trust a number of hops, as Envoy does today, or it can trust by IP address. This enables configuration to add a set of CIDRs to the XFF config, and a recurse option. When recurse is not enabled, HCM should trust a remote IP that is covered by one of the configured CIDRs and use the last IP in XFF as the remote IP. When recurse is enabled, if the remote IP is covered by one of the CIDRs, HCM should also walk backwards through XFF to find the first IP not covered by the trusted CIDR list. Signed-off-by: James O'Gorman <[email protected]>

remoteAddressIsTrustedProxy checks the remote IP against a list of CIDRs. If it is covered by any CIDR the remote is considered a trusted proxy. getLastNonTrustedAddressFromXFF walks backwards through XFF to find the first IP that is not covered by a list of trusted CIDRs. This IP is considered the "true" remote IP. Signed-off-by: James O'Gorman <[email protected]>

If a list of trusted CIDRs is set, rather than checking for a set number of trusted hops, instead check the remote IP against the trusted CIDR list. If recurse is set, XFF is also evaluated to find the last IP in the list that is not covered by the trusted CIDRs. Signed-off-by: James O'Gorman <[email protected]>

Signed-off-by: James O'Gorman <[email protected]>

repokitteh-read-only · 2024-01-15T19:25:15Z

Hi @jamesog, welcome and thank you for your contribution.

We will try to review your Pull Request as quickly as possible.

In the meantime, please take a look at the contribution guidelines if you have not done so already.

🐱

Caused by: #31831 was opened by jamesog.

see: more, trace.

repokitteh-read-only · 2024-01-15T19:25:20Z

CC @envoyproxy/api-shepherds: Your approval is needed for changes made to (api/envoy/|docs/root/api-docs/).
envoyproxy/api-shepherds assignee is @markdroth
CC @envoyproxy/api-watchers: FYI only for changes made to (api/envoy/|docs/root/api-docs/).

🐱

Caused by: #31831 was opened by jamesog.

see: more, trace.

Signed-off-by: James O'Gorman <[email protected]>

source/extensions/http/original_ip_detection/xff/xff.cc

api/envoy/extensions/http/original_ip_detection/xff/v3/xff.proto

Override xff_num_trusted_hops in the constructor instead of validating in the proto message when xff_trusted_cidrs is set. Signed-off-by: James O'Gorman <[email protected]>

Signed-off-by: James O'Gorman <[email protected]>

markdroth · 2024-01-17T17:29:32Z

/lgtm api

alyssawilk

So sorry for the review delay. Thanks for the well documented and tested first pass - my thoughts below.
/wait

alyssawilk · 2024-01-23T16:25:40Z

api/envoy/extensions/http/original_ip_detection/xff/v3/xff.proto

  // is not specified. See the documentation for
  // :ref:`config_http_conn_man_headers_x-forwarded-for` for more information.
+  //
+  // If ``xff_trusted_cidrs`` is configured, this field is not used.


can we disallow it being set and reject config if it is?

I originally had this change as using a oneof encapsulating the original xff_num_trusted_hops and the new xff_trusted_cidrs fields, but Mark told me that would be a breaking API change and to do it this way instead. Do you know if there's another way to handle rejecting it without it being a breaking change?

I don't think we want to disallow if both the old and new fields are set, since that makes it hard for control planes to start using this new feature.

If we reject when both fields are set, then a control plane must know whether any given client supports the new field. If it sets only the new field, old clients will not work properly, and if it sets both, new clients will break. So the only way to transition from the old field to the new field is to either (a) wait for all clients to be upgraded before starting to use the new field or (b) have some way for the control plane to know which client supports the new field and which clients support the old field so that it can send the right config to each one -- and doing that in a way that does not break cacheability requires some dances with dynamic parameter support, which means that the clients have to be configured to tell the control plane whether or not they support the new field.

In contrast, if we allow both fields to be set and specify the precedence order, then this becomes much easier. Control planes can simply set both fields. Old clients will use the old field, and new clients will prefer the new field.

I think the latter approach is much cleaner and should be preferred any time we add this kind of new field in xDS.

How do you feel about the change I made since Alyssa's feedback? d976354

She suggested it should either log or reject; I chose to reject by throwing an error with a message that clearly informs that only one field can be set.

What I'm saying is that I don't think that's the behavior we want. I think the way you had this originally was the right thing to do.

In that case I'm getting conflicting advice from two different maintainers :-)

@alyssawilk perhaps you can help decide which way I should take this based on Mark's feedback?

alyssawilk · 2024-01-23T16:27:41Z

api/envoy/extensions/http/original_ip_detection/xff/v3/xff.proto

+  //
+  // If ``recurse`` is set the :ref:`config_http_conn_man_headers_x-forwarded-for`
+  // header is also evaluated against the trusted CIDR list, and the last non-trusted
+  // address is used as the original client address.


ooc when would we not want to recurse?

It's a good question! I added some commentary to the docs in 1f76584.

api/envoy/extensions/http/original_ip_detection/xff/v3/xff.proto

alyssawilk · 2024-01-23T16:33:46Z

source/common/http/utility.cc

+    }
+    return {addr, true};
+  }
+  return {nullptr, false};


if every address in the XFF is trusted don't we want to return the last address in the list?

This one made me think! Originally I was assuming this might be some kind of undefined behaviour and so it was safer to return null, but after talking about this with a colleague, and looking over what other proxy servers do, I implemented your suggestion in a016a6c. Thanks for questioning my logic :-)

When parsing XFF using a list of trusted CIDRs, if all IP in XFF match the trusted CIDRs (there are no untrusted IPs), return the first IP in XFF (the last evaluated) - and a test to confirm the behaviour. Signed-off-by: James O'Gorman <[email protected]>

Signed-off-by: James O'Gorman <[email protected]>

jamesog · 2024-08-23T11:34:08Z

/retest

wbpcode · 2024-08-31T15:16:07Z

Sorry for the delay. Some times I may ignore the notification from tools. It's OK to ping me directly on slack.

wbpcode · 2024-08-31T15:16:57Z

/wait

The API should provide a consistent default value, and we cannot set this to false without causing a breaking change. Instead we have a warning in the API docs recommending to disable this option. Signed-off-by: James O'Gorman <[email protected]>

Signed-off-by: James O'Gorman <[email protected]>

This reverts c69df1c and changes to use throwEnvoyExceptionOrPanic, which will cause a config error in Envoy, and a panic in Envoy Mobile. Signed-off-by: James O'Gorman <[email protected]>

Signed-off-by: James O'Gorman <[email protected]>

wbpcode

Thanks for the great contribution. Only last comment, I think.

wbpcode · 2024-09-04T03:47:26Z

source/common/http/utility.cc

+bool Utility::remoteAddressIsTrustedProxy(
+    const Envoy::Network::Address::InstanceConstSharedPtr remote,


See, basically we only use the smart point as input parameter when we wants to change the ownership. But in this case, if we ensure this remote always be valid, then const Envoy::Network::Address::Instance& should be used.

If we can ensure this remote always be valid, then OptRef<const Envoy::Network::Address::Instance> should be used.

I think there should be the first case.

I think I understood you correctly. This is getting beyond my knowledge here so I wasn't quite sure. Please see the latest commit that now uses const Network::Address:Instance&.

Signed-off-by: James O'Gorman <[email protected]>

wbpcode

LGTM. Thanks.

…r in CustomHeaderIPDetection (#37194) ## Description This PR restores the original behavior prior to #31831, by explicitly setting `skip_xff_append` to `true`, as appending the `XFF` header from the custom header extension is not required. See #37171 for more details. --- **Commit Message:** original_ip_detection: revert unintended XFF header appending behavior in CustomHeaderIPDetection **Additional Description:** This PR restores the pre #31831 behavior by explicitly setting `skip_xff_append` to `true`, as appending the `XFF` header from the custom header extension is not required. **Risk Level:** Low **Testing:** Added Unit Tests **Docs Changes:** N/A **Release Notes:** Added Signed-off-by: Rohit Agrawal <[email protected]>

…r in CustomHeaderIPDetection (envoyproxy#37194) This PR restores the original behavior prior to envoyproxy#31831, by explicitly setting `skip_xff_append` to `true`, as appending the `XFF` header from the custom header extension is not required. See envoyproxy#37171 for more details. --- **Commit Message:** original_ip_detection: revert unintended XFF header appending behavior in CustomHeaderIPDetection **Additional Description:** This PR restores the pre envoyproxy#31831 behavior by explicitly setting `skip_xff_append` to `true`, as appending the `XFF` header from the custom header extension is not required. **Risk Level:** Low **Testing:** Added Unit Tests **Docs Changes:** N/A **Release Notes:** Added Signed-off-by: Rohit Agrawal <[email protected]>

…r in CustomHeaderIPDetection (#37194) This PR restores the original behavior prior to #31831, by explicitly setting `skip_xff_append` to `true`, as appending the `XFF` header from the custom header extension is not required. See #37171 for more details. --- **Commit Message:** original_ip_detection: revert unintended XFF header appending behavior in CustomHeaderIPDetection **Additional Description:** This PR restores the pre #31831 behavior by explicitly setting `skip_xff_append` to `true`, as appending the `XFF` header from the custom header extension is not required. **Risk Level:** Low **Testing:** Added Unit Tests **Docs Changes:** N/A **Release Notes:** Added Signed-off-by: Rohit Agrawal <[email protected]>

…oyproxy#31831)" This reverts commit fbc6ee2.

jamesog added 5 commits January 15, 2024 12:34

docs: Note about xff_trusted_cidrs.

831057e

Signed-off-by: James O'Gorman <[email protected]>

changelog: Document xff trusted CIDR addition

b32b10d

Signed-off-by: James O'Gorman <[email protected]>

jamesog requested review from alyssawilk and mattklein123 as code owners January 15, 2024 19:25

repokitteh-read-only bot added the api label Jan 15, 2024

repokitteh-read-only bot assigned markdroth Jan 15, 2024

jamesog added 2 commits January 15, 2024 19:53

Format fixes

b37899d

Signed-off-by: James O'Gorman <[email protected]>

Fix spell check issues

34e7d33

Signed-off-by: James O'Gorman <[email protected]>

jamesog commented Jan 15, 2024

View reviewed changes

source/extensions/http/original_ip_detection/xff/xff.cc Outdated Show resolved Hide resolved

jamesog mentioned this pull request Jan 16, 2024

Configurable filter of trusted proxies to handle x-forwarded-for header #21639

Closed

markdroth reviewed Jan 16, 2024

View reviewed changes

api/envoy/extensions/http/original_ip_detection/xff/v3/xff.proto Outdated Show resolved Hide resolved

api/envoy/extensions/http/original_ip_detection/xff/v3/xff.proto Outdated Show resolved Hide resolved

jamesog added 6 commits January 16, 2024 20:22

api: Remove use of oneof in XffConfig

61ade56

Override xff_num_trusted_hops in the constructor instead of validating in the proto message when xff_trusted_cidrs is set. Signed-off-by: James O'Gorman <[email protected]>

xff: Remove unused constructor

0cf5c6c

Signed-off-by: James O'Gorman <[email protected]>

Spelling: quote symbols

4c0ff3f

Signed-off-by: James O'Gorman <[email protected]>

Autoformat

d9f1a64

Signed-off-by: James O'Gorman <[email protected]>

Merge remote-tracking branch 'upstream/main' into xff_trusted_cidrs

5842f38

Signed-off-by: James O'Gorman <[email protected]>

api: Remove unused import

2c95c9f

Signed-off-by: James O'Gorman <[email protected]>

repokitteh-read-only bot removed the api label Jan 17, 2024

ravenblackx assigned alyssawilk Jan 19, 2024

alyssawilk reviewed Jan 23, 2024

View reviewed changes

repokitteh-read-only bot added the waiting label Jan 23, 2024

jamesog added 3 commits January 24, 2024 17:18

api: Note about n*m operation with xff_trusted_cidrs

eb2f7ff

Signed-off-by: James O'Gorman <[email protected]>

api: Clarify when recurse may want to be used

1f76584

Signed-off-by: James O'Gorman <[email protected]>

jamesog added 2 commits August 21, 2024 20:47

hcm: Address feedback

7721581

Signed-off-by: James O'Gorman <[email protected]>

Merge remote-tracking branch 'upstream/main' into xff_trusted_cidrs

837cd5f

Signed-off-by: James O'Gorman <[email protected]>

repokitteh-read-only bot removed the waiting label Aug 21, 2024

repokitteh-read-only bot added the waiting label Aug 31, 2024

jamesog added 4 commits August 31, 2024 18:49

hcm: Use absl::InlinedVector

aec2db5

Signed-off-by: James O'Gorman <[email protected]>

xff: Revert c69df1c

57baab7

This reverts c69df1c and changes to use throwEnvoyExceptionOrPanic, which will cause a config error in Envoy, and a panic in Envoy Mobile. Signed-off-by: James O'Gorman <[email protected]>

Merge remote-tracking branch 'upstream/main' into xff_trusted_cidrs

7344aa8

Signed-off-by: James O'Gorman <[email protected]>

repokitteh-read-only bot removed the waiting label Aug 31, 2024

Merge remote-tracking branch 'upstream/main' into xff_trusted_cidrs

277fefd

Signed-off-by: James O'Gorman <[email protected]>

wbpcode reviewed Sep 4, 2024

View reviewed changes

repokitteh-read-only bot added the waiting label Sep 4, 2024

xff: Don't use smart pointer

91903c9

Signed-off-by: James O'Gorman <[email protected]>

repokitteh-read-only bot removed the waiting label Sep 4, 2024

wbpcode approved these changes Sep 9, 2024

View reviewed changes

repokitteh-read-only bot removed the api label Sep 9, 2024

wbpcode merged commit fbc6ee2 into envoyproxy:main Sep 9, 2024

larhauga mentioned this pull request Sep 18, 2024

Support envoy X-Forwarded-For trusted CIDRs istio/istio#53185

Open

robd003 mentioned this pull request Oct 21, 2024

Allow setting of trusted CIDR blocks for X-Forwarded-For envoyproxy/gateway#4489

Closed

agrawroh added a commit to agrawroh/envoy that referenced this pull request Dec 2, 2024

Revert "xff: add support for configuring a list of trusted CIDRs (env…

5674069

…oyproxy#31831)" This reverts commit fbc6ee2.

adisuissa mentioned this pull request Feb 18, 2025

Conflicting XFF and xff_trusted_cidr documentation #38462

Closed

		bool Utility::remoteAddressIsTrustedProxy(
		const Envoy::Network::Address::InstanceConstSharedPtr remote,

xff: add support for configuring a list of trusted CIDRs #31831

xff: add support for configuring a list of trusted CIDRs #31831

Uh oh!

Conversation

jamesog commented Jan 15, 2024

Uh oh!

repokitteh-read-only bot commented Jan 15, 2024

Uh oh!

repokitteh-read-only bot commented Jan 15, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

markdroth commented Jan 17, 2024

Uh oh!

alyssawilk left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jamesog commented Aug 23, 2024

Uh oh!

wbpcode commented Aug 31, 2024

Uh oh!

wbpcode commented Aug 31, 2024

Uh oh!

wbpcode left a comment

Choose a reason for hiding this comment

Uh oh!

wbpcode Sep 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wbpcode left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

wbpcode Sep 4, 2024 •

edited

Loading