Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add IPv6 support to iptables proxier #48551

Merged
merged 1 commit into from
Sep 24, 2017

Conversation

leblancd
Copy link

@leblancd leblancd commented Jul 6, 2017

Add IPv6 support to iptables proxier

The following changes are proposed for the iptables proxier:

  • There are three places where a string specifying IP:port is parsed
    using something like this:
    if index := strings.Index(e.endpoint, ":"); index != -1 {
    This will fail for IPv6 since V6 addresses contain colons. Also,
    the V6 address is expected to be surrounded by square brackets
    (i.e. []:). Fix this by replacing call to Index with
    call to LastIndex() and stripping out square brackets.
  • The String() method for the localPort struct should put square brackets
    around IPv6 addresses.
  • The logging in the merge() method for proxyServiceMap should put brackets
    around IPv6 addresses.
  • There are several places where filterRules destination is hardcoded to
    /32. This should be a /128 for IPv6 case.
  • Add IPv6 unit test cases

Note: I've left out most of the UT test cases that I had included in my original version of this
PR because the number of lines of code change were much too large for a single review.
I'm including a minimum of UT with this current version of the PR.

fixes #48550

What this PR does / why we need it:
This PR addresses several issues in the iptables proxier for handling IPv6 addresses
that were found via visual code inspection, including:

  • There are three places where a string specifying IP:port using something like the following:
    if index := strings.Index(e.endpoint, ":"); index != -1 {
    This will fail for IPv6 since V6 addresses contains many colons, and the V6 address is expected
    to be enclosed in square brackets when followed by :.
  • The String() method for the localPort struct should put square brackets around IPv6 addresses.
  • The logging in the merge() method for proxyServiceMap should put brackets around IPv6
    addresses.
  • There are several places where filterRules destination is hardcoded to /32.
    Should be a /128 for IPv6 case.
  • More IPv6 unit test cases are needed.

Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #48550

Special notes for your reviewer:

Release note:

NONE

@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jul 6, 2017
@k8s-ci-robot
Copy link
Contributor

Hi @leblancd. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@k8s-github-robot k8s-github-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. release-note-label-needed labels Jul 6, 2017
@cmluciano
Copy link

@k8s-bot ok to test

@k8s-ci-robot k8s-ci-robot removed the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jul 6, 2017
Copy link
Contributor

@pmichali pmichali left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bunch of questions in-line. Mostly minor stuff.

if index := strings.LastIndex(s, ":"); index != -1 {
ip := s[0:index]
// Strip off any surrounding brackets.
re := regexp.MustCompile(`\[(.*)\]`)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would a less compute intensive solution be to just check for a '[' at position zero and then take slice w/o first and last character?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, regex seems overpowered for this

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed.

// ipPort returns a string of the form "<ip>:<port>" (for IPv4)
// or "[<ip>]:<port>" (for IPv6) for a given IP address and port
func ipPort(ipStr string, port int) string {
ip := net.ParseIP(ipStr)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we know that ipStr will always be an IP address (and not a name)? Is it guaranteed to always be valid?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

net.JoinHostPort() seems to do exactly what this does, if we just fmt.Sprintf("%d", port) or stconv.Itoa(port)

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@thockin: Yes, that will do the trick, thanks!

// IPv4 and <ip-address>/128 for IPv6
func hostAddress(ip net.IP) string {
len := 32
if ip.To4() == nil {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it always guaranteed that ip is a valid IP address? IOW no need to check for nil? Just wondering if all the callers using this are providing valid IP addresses.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The nil check isn't being done as a sanity check, it's being done to determine if the IP address is IPv4 or IPv6.

Copy link
Contributor

@pmichali pmichali left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would suggest adding additional test cases that exercise the new/modified methods directly (versus through some call chain of higher level function test). Especially as you can exercise error cases.

{
name: "V4 UDP glogged error",
epSvcPair: endpointServicePair{
endpoint: "10.240.0.5:80",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Should either name all the test case fields or none (consistency). For example, L232 vs L224

return "IPv6"
}

func hasJumpTest(t *testing.T, cidrs []string) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can the function name be more descriptive?

@@ -432,6 +505,36 @@ func hasJump(rules []iptablestest.Rule, destChain, destIP string, destPort int)
}

func TestHasJump(t *testing.T) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we make the test name more descriptive or add a docstring to indicate what is being tested?

@@ -442,18 +545,18 @@ func TestHasJump(t *testing.T) {
"case 1": {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of "case 1", "case 2", ... could these be meaningful names for the test "match 1st rule", "match 2nd rule", "match both dest IP and port",...? Just thinking we can document the failures better.

proto := strings.ToLower(string(api.ProtocolTCP))
fwChain := string(serviceFirewallChainName(svcPortName.String(), proto))
svcChain := string(servicePortChainName(svcPortName.String(), proto))
//lbChain := string(serviceLBChainName(svcPortName.String(), proto))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull dead code?

unnamedPort := func(ept *api.Endpoints) {
ept.Subsets = []api.EndpointSubset{{
Addresses: []api.EndpointAddress{{
IP: "1.1.1.1",
IP: testIP(1, 1),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I really think the indirection of the testIP function makes the tests harder to read.

@leblancd
Copy link
Author

leblancd commented Jul 7, 2017

/retest
(Looks like test failure was flake #48368)

@k8s-github-robot k8s-github-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 8, 2017
@cmluciano
Copy link

/assign @dcbw

// Returns just the IP part of an IP:port or IP endpoint string. If the IP
// part is an IPv6 address enclosed in brackets (e.g. "[fd00:1::5]:9999"),
// then the brackets are stripped as well.
func ipPart(s string) string {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Um, how about net.SplitHostPort() instead?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that seems to DTRT

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, net.SplitHostPort() will help, thanks! But we'll still need a little bit of processing here before it's called. SplitHostPort() only works with strings of the form IPv4-addr:port or [IPv6-addr]:port, so we'll have to first check if the string being processed is of the form IPv4, IPv6-addr, or [IPv6-addr] (and strip off brackets for the latter case) before calling SplitHostPort().

@dcbw
Copy link
Member

dcbw commented Jul 27, 2017

Also the pull-kubernetes-verify test failed; that's likely because of new imports to the testcases and proxy code. A good way to update that is:

docker run -it -v $(pwd):/go/src/k8s.io/kubernetes -w /go/src/k8s.io/kubernetes golang:1.8 /bin/bash -c "hack/update-bazel.sh"

And then git add any changed files and git commit --amend to add them to the existing commit.

Copy link
Member

@thockin thockin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would be nice to see this as individual tiny commits, especially as large as the test changes are.

if index := strings.LastIndex(s, ":"); index != -1 {
ip := s[0:index]
// Strip off any surrounding brackets.
re := regexp.MustCompile(`\[(.*)\]`)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, regex seems overpowered for this

// Returns just the IP part of an IP:port or IP endpoint string. If the IP
// part is an IPv6 address enclosed in brackets (e.g. "[fd00:1::5]:9999"),
// then the brackets are stripped as well.
func ipPart(s string) string {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that seems to DTRT

// ipPort returns a string of the form "<ip>:<port>" (for IPv4)
// or "[<ip>]:<port>" (for IPv6) for a given IP address and port
func ipPort(ipStr string, port int) string {
ip := net.ParseIP(ipStr)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

net.JoinHostPort() seems to do exactly what this does, if we just fmt.Sprintf("%d", port) or stconv.Itoa(port)

leblancd pushed a commit to leblancd/kubernetes that referenced this pull request Aug 10, 2017
For iptables save and restore operations, kube-proxy currently uses
the IPv4 versions of the iptables save and restore utilities
(iptables-save and iptables-restore, respectively). For IPv6 operation,
the IPv6 versions of these utilities needs to be used
(ip6tables-save and ip6tables-restore, respectively).

Both this change and PR kubernetes#48551 are needed to get Kubernetes services
to work in an IPv6-only Kubernetes cluster (along with setting
'--bind-address ::0' on the kube-proxy command line. This change
was alluded to in a discussion on services for issue kubernetes#1443.

fixes kubernetes#50474
@thockin
Copy link
Member

thockin commented Aug 17, 2017

ping me when ready for review.

@thockin
Copy link
Member

thockin commented Aug 29, 2017

We have O(days) to get this into v1.8

@leblancd
Copy link
Author

@thockin Thanks. I think this will have to wait until v1.9. I'm seeing some issues in my functional tests with this code. It might be that the code is okay, but I'm driving it incorrectly. Sorry for the delay on this.

@k8s-github-robot
Copy link

Adding do-not-merge/release-note-label-needed because the release note process has not been followed.
One of the following labels is required "release-note", "release-note-action-required", "release-note-experimental" or "release-note-none".
Please see: https://github.com/kubernetes/community/blob/master/contributors/devel/pull-requests.md#write-release-notes-if-needed.

@k8s-github-robot k8s-github-robot added the do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. label Aug 30, 2017
@k8s-github-robot k8s-github-robot added release-note-none Denotes a PR that doesn't merit a release note. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. release-note-label-needed labels Sep 14, 2017
@leblancd
Copy link
Author

I'll post a rebase of the operational (non-unit-test) code with changes for review comments. For UT, I need to figure out how to split up the test cases into several, smaller-sized reviews (and add minimum set of test cases back into this review).

@k8s-ci-robot k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Sep 14, 2017
@k8s-github-robot k8s-github-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 14, 2017
@k8s-ci-robot k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Sep 15, 2017
@leblancd leblancd force-pushed the v6_new_proxier branch 2 times, most recently from 1d198e1 to 58c6f49 Compare September 15, 2017 21:32
@leblancd
Copy link
Author

/assign thockin
/unassign leblancd

@k8s-ci-robot k8s-ci-robot assigned thockin and unassigned leblancd Sep 15, 2017
Copy link
Member

@thockin thockin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a couple minor questions, I think.

// IP address without brackets
return s
}
if s[0] == '[' && s[len(s)-1] == ']' {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this a legit notation? Why wouldn't ParseIP handle it correctly?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right, this is not legit. The brackets should only appear for IPv6:port or when IPv6 address appears in a URL. I'll delete these lines.

nsn := svcPortName.NamespacedName
if localIPs[nsn] == nil {
localIPs[nsn] = sets.NewString()
if ip := ep.IPPart(); ip != "" {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why would this case happen? If legit, leave a comment?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suppose this depends on how defensive we want to be with our coding. Here ep.IPPart() could return a null string if net.SplitHostPort() detects bad syntax and returns a null string (along with an error). This can happen (theoretically) if the string in ep.endpoint is not quite right. For example, if the ep.endpoint has too many colons, e.g. "1.2.3.4::9999", or brackets that are out of place, e.g. "1.2.3.4:[0]", then net.SplitHostPort() returns a null string and an error. It's probably very unlikely that we'd end up with a bad format in ep.endpoint at this point, with sanity checks that are done elsewhere. I'll leave this null-string check in for now and add a comment, let me know if you feel this is being too defensive.

The following changes are proposed for the iptables proxier:

* There are three places where a string specifying IP:port is parsed
  using something like this:

      if index := strings.Index(e.endpoint, ":"); index != -1 {

  This will fail for IPv6 since V6 addresses contain colons. Also,
  the V6 address is expected to be surrounded by square brackets
  (i.e. []:). Fix this by replacing call to Index with
  call to LastIndex() and stripping out square brackets.
* The String() method for the localPort struct should put square brackets
  around IPv6 addresses.
* The logging in the merge() method for proxyServiceMap should put brackets
  around IPv6 addresses.
* There are several places where filterRules destination is hardcoded to
  /32. This should be a /128 for IPv6 case.
* Add IPv6 unit test cases

fixes kubernetes#48550
leblancd pushed a commit to leblancd/kubernetes that referenced this pull request Sep 18, 2017
For iptables save and restore operations, kube-proxy currently uses
the IPv4 versions of the iptables save and restore utilities
(iptables-save and iptables-restore, respectively). For IPv6 operation,
the IPv6 versions of these utilities needs to be used
(ip6tables-save and ip6tables-restore, respectively).

Both this change and PR kubernetes#48551 are needed to get Kubernetes services
to work in an IPv6-only Kubernetes cluster (along with setting
'--bind-address ::0' on the kube-proxy command line. This change
was alluded to in a discussion on services for issue kubernetes#1443.

fixes kubernetes#50474
leblancd pushed a commit to leblancd/kubernetes that referenced this pull request Sep 18, 2017
This change causes kube-proxy to supply the required "-f ipv6"
family flag whenever the conntrack utility is executed and the
associated service is using IPv6.

This change is required for IPv6-only operation.

Note that unit test coverage for the 2-line changes in
pkg/proxy/iptables/proxier.go and /pkg/proxy/ipvs/proxier.go will need
to be added after support for IPv6 service addresses is added to these
files. For pkg/proxy/iptables/proxier.go, this coverage will be added
either with PR kubernetes#48551.

fixes kubernetes#52027
@thockin
Copy link
Member

thockin commented Sep 20, 2017

/lgtm
/approve

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 20, 2017
@k8s-github-robot
Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: leblancd, thockin

Associated issue: 48550

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

@k8s-github-robot k8s-github-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 20, 2017
@k8s-github-robot
Copy link

Automatic merge from submit-queue (batch tested with PRs 50068, 52406, 52394, 48551, 52131). If you want to cherry-pick this change to another branch, please follow the instructions here..

@k8s-github-robot k8s-github-robot merged commit 414a3bd into kubernetes:master Sep 24, 2017
k8s-github-robot pushed a commit that referenced this pull request Oct 5, 2017
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Add required family flag for conntrack IPv6 operation

This change causes kube-proxy to supply the required "-f ipv6"
family flag whenever the conntrack utility is executed and the
associated service is using IPv6.

This change is required for IPv6-only operation.

Note that unit test coverage for the 2-line changes in
pkg/proxy/iptables/proxier.go and /pkg/proxy/ipvs/proxier.go will need
to be added after support for IPv6 service addresses is added to these
files. For pkg/proxy/iptables/proxier.go, this coverage will be added
either with PR #48551.

fixes #52027



**What this PR does / why we need it**:
Kube-proxy is currently not supplying the required "-f ipv6" family flag whenever it
calls the conntrack utility and the associated service is using an IPv6 service IP address.
This means that for IPv6-only operation, conntrack is not properly cleaning up
stale UDP connections, and this may be effecting ip6tables operation.

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes # 52027

**Special notes for your reviewer**:

**Release note**:

```release-note
NONE
```
k8s-github-robot pushed a commit that referenced this pull request Oct 11, 2017
Automatic merge from submit-queue (batch tested with PRs 52520, 52033, 53626, 50478). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Fix kube-proxy to use proper iptables commands for IPv6 operation

For iptables save and restore operations, kube-proxy currently uses
the IPv4 versions of the iptables save and restore utilities
(iptables-save and iptables-restore, respectively). For IPv6 operation,
the IPv6 versions of these utilities need to be used
(ip6tables-save and ip6tables-restore, respectively).

Both this change and PR #48551 are needed to get Kubernetes services
to work in an IPv6-only Kubernetes cluster (along with setting
'--bind-address ::0' on the kube-proxy command line. This change
was alluded to in a discussion on services for issue #1443.

fixes #50474



**What this PR does / why we need it**:
This change modifies kube-proxy so that it uses the proper commands for iptables save and
iptables restore for IPv6 operation. Currently kube-proxy uses 'iptables-save' and 'iptables-restore'
regardless of whether it is being used in IPv4 or IPv6 mode. This change fixes kube-proxy so
that it uses 'ip6tables-save' and 'ip6tables-restore' commands when kube-proxy is being run
in IPv6 mode.

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #50474

**Special notes for your reviewer**:

**Release note**:

```release-note NONE
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/ipv6 area/kube-proxy cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. release-note-none Denotes a PR that doesn't merit a release note. sig/network Categorizes an issue or PR as relevant to SIG Network. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Fix IPv6 address handling in iptables proxier
9 participants