Skip to content

Comments

[release-4.9] Bug 2050632: UPSTREAM: <drop>: Give warning when ipFamilyPolicy implicitly set#1170

Merged
openshift-merge-robot merged 1 commit intoopenshift:release-4.9from
andreaskaris:bz2045576-4.9
Apr 13, 2022
Merged

[release-4.9] Bug 2050632: UPSTREAM: <drop>: Give warning when ipFamilyPolicy implicitly set#1170
openshift-merge-robot merged 1 commit intoopenshift:release-4.9from
andreaskaris:bz2045576-4.9

Conversation

@andreaskaris
Copy link

@andreaskaris andreaskaris commented Feb 4, 2022

UPSTREAM: : Give warning when ipFamilyPolicy implicitly set

In kube 1.21 and 1.22 (OCP 4.8 and 4.9), the apiserver would default
the value of ipFamilyPolicy to RequireDualStack if you created a
Service with two ipFamilies or two clusterIPs but no explicitly
specified ipFamilyPolicy. In 1.23/4.10, you must explicitly specify
either PreferDualStack or RequireDualStack for DualStack services.
Emit a warning in 4.8 and 4.9 to raise awareness about the upcoming
API changes.

OpenShift 4.8 and 4.9 only, BZ 2047676

What type of PR is this?

What this PR does / why we need it:

Warn users about an upcoming breaking API change.

Which issue(s) this PR fixes:

Fixes BZ 2045576

Special notes for your reviewer:

I am proposing that we introduce a tiny change in openshift/kubernetes downstream and that we generate the warning directly inside the API server.

Does this PR introduce a user-facing change?

Yes, in a DualStack cluster:

[root@openshift-jumpserver-0 ~]# cat nginx-dualstack.yaml 
apiVersion: v1
kind: Service
metadata:
  name: nginx-dualstack
spec:
  type: ClusterIP
  selector:
    app: nginx
  ipFamilies:
         - IPv6
         - IPv4
  ports:
      # By default and for convenience, the `targetPort` is set to the same value as the `port` field.
    - port: 80
      targetPort: 80

[root@openshift-jumpserver-0 ~]# oc apply -f nginx-dualstack.yaml 
W0204 11:58:12.353512  112108 warnings.go:70] Setting DualStack Service.Spec.IPFamilies or DualStack Service.Spec.ClusterIPs without explicitly setting Service.Spec.IPFamilyPolicy is deprecated. This operation will fail starting with Red Hat OpenShift Platform 4.10. Make sure to set IPFamilyPolicy to PreferDualStack or RequireDualStack when configuring DualStack services.
service/nginx-dualstack created
[root@openshift-jumpserver-0 ~]# oc delete -f nginx-dualstack.yaml 
service "nginx-dualstack" deleted
[root@openshift-jumpserver-0 ~]# oc create -f nginx-dualstack.yaml 
W0204 11:58:27.983873  112139 warnings.go:70] Setting DualStack Service.Spec.IPFamilies or DualStack Service.Spec.ClusterIPs without explicitly setting Service.Spec.IPFamilyPolicy is deprecated. This operation will fail starting with Red Hat OpenShift Platform 4.10. Make sure to set IPFamilyPolicy to PreferDualStack or RequireDualStack when configuring DualStack services.
service/nginx-dualstack created
[root@openshift-jumpserver-0 ~]# oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.9.17    True        True          4h13m   Unable to apply 4.9.17: wait has exceeded 40 minutes for these operators: network
In Red Hat OpenShift Container Platform 4.8 and 4.9, the kube-apiserver will default the value of ipFamilyPolicy to RequireDualStack when such a service is created with two ipFamilies or two clusterIPs but no explicitly specified ipFamilyPolicy.
In Red Hat OpenShift Container Platform 4.10, administrators must explicitly specify either ipFamilyPolicy: PreferDualStack or ipFamilyPolicy: RequireDualStack for DualStack services.
Emit a warning in 4.8 and 4.9 to raise awareness about the upcoming API changes.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:


@openshift-ci-robot openshift-ci-robot added the backports/unvalidated-commits Indicates that not all commits come to merged upstream PRs. label Feb 4, 2022
@openshift-ci openshift-ci bot added the bugzilla/severity-low Referenced Bugzilla bug's severity is low for the branch this PR is targeting. label Feb 4, 2022
@openshift-ci
Copy link

openshift-ci bot commented Feb 4, 2022

@andreaskaris: This pull request references Bugzilla bug 2047676, which is invalid:

  • expected the bug to be open, but it isn't
  • expected the bug to be in one of the following states: NEW, ASSIGNED, ON_DEV, POST, POST, but it is CLOSED (NOTABUG) instead
  • expected dependent Bugzilla bug 2045576 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE), but it is POST instead
  • expected dependent Bugzilla bug 2045576 to target a release in 4.10.0, but it targets "4.9.z" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

[release-4.9] Bug 2047676: UPSTREAM: : Give warning when ipFamilyPolicy implicitly set

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci openshift-ci bot added the bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. label Feb 4, 2022
@openshift-ci-robot
Copy link

@andreaskaris: the contents of this pull request could not be automatically validated.

The following commits could not be validated and must be approved by a top-level approver:

Comment /validate-backports to re-evaluate validity of the upstream PRs, for example when they are merged upstream.

@andreaskaris
Copy link
Author

Continuation from: #1151

@openshift-ci
Copy link

openshift-ci bot commented Feb 4, 2022

@andreaskaris: This pull request references Bugzilla bug 2047676, which is invalid:

  • expected the bug to be open, but it isn't
  • expected the bug to be in one of the following states: NEW, ASSIGNED, ON_DEV, POST, POST, but it is CLOSED (NOTABUG) instead
  • expected dependent Bugzilla bug 2045576 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE), but it is POST instead
  • expected dependent Bugzilla bug 2045576 to target a release in 4.10.0, but it targets "4.9.z" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

[release-4.9] Bug 2047676: UPSTREAM: : Give warning when ipFamilyPolicy implicitly set

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci openshift-ci bot requested review from deads2k and mfojtik February 4, 2022 11:22
@openshift-ci
Copy link

openshift-ci bot commented Feb 4, 2022

@andreaskaris: This pull request references Bugzilla bug 2047676, which is invalid:

  • expected the bug to be open, but it isn't
  • expected the bug to be in one of the following states: NEW, ASSIGNED, ON_DEV, POST, POST, but it is CLOSED (NOTABUG) instead
  • expected dependent Bugzilla bug 2045576 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE), but it is POST instead
  • expected dependent Bugzilla bug 2045576 to target a release in 4.10.0, but it targets "4.9.z" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

[release-4.9] Bug 2047676: UPSTREAM: : Give warning when ipFamilyPolicy implicitly set

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@andreaskaris andreaskaris changed the title [release-4.9] Bug 2047676: UPSTREAM: <carry>: Give warning when ipFamilyPolicy implicitly set [release-4.9] Bug 2050632: UPSTREAM: <carry>: Give warning when ipFamilyPolicy implicitly set Feb 4, 2022
@openshift-ci openshift-ci bot added bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. and removed bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. labels Feb 4, 2022
@openshift-ci
Copy link

openshift-ci bot commented Feb 4, 2022

@andreaskaris: This pull request references Bugzilla bug 2050632, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

6 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (4.9.z) matches configured target release for branch (4.9.z)
  • bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)
  • dependent bug Bugzilla bug 2048645 is in the state CLOSED (CURRENTRELEASE), which is one of the valid states (VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE))
  • dependent Bugzilla bug 2048645 targets the "4.10.0" release, which is one of the valid target releases: 4.10.0
  • bug has dependents

Requesting review from QA contact:
/cc @wangke19

Details

In response to this:

[release-4.9] Bug 2050632: UPSTREAM: : Give warning when ipFamilyPolicy implicitly set

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci openshift-ci bot requested a review from wangke19 February 4, 2022 11:29
@aojea
Copy link

aojea commented Feb 4, 2022

/test k8s-e2e-gcp

@aojea
Copy link

aojea commented Feb 4, 2022

/hold
I was confused with other PR, I'll pick this up next week, I want to see if we can add some unit tests

@openshift-ci openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 4, 2022
@openshift-ci
Copy link

openshift-ci bot commented Feb 4, 2022

@andreaskaris: This pull request references Bugzilla bug 2050632, which is valid.

6 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (4.9.z) matches configured target release for branch (4.9.z)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)
  • dependent bug Bugzilla bug 2048645 is in the state CLOSED (CURRENTRELEASE), which is one of the valid states (VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE))
  • dependent Bugzilla bug 2048645 targets the "4.10.0" release, which is one of the valid target releases: 4.10.0
  • bug has dependents

Requesting review from QA contact:
/cc @wangke19

Details

In response to this:

[release-4.9] Bug 2050632: UPSTREAM: : Give warning when ipFamilyPolicy implicitly set

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

1 similar comment
@openshift-ci
Copy link

openshift-ci bot commented Feb 4, 2022

@andreaskaris: This pull request references Bugzilla bug 2050632, which is valid.

6 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (4.9.z) matches configured target release for branch (4.9.z)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)
  • dependent bug Bugzilla bug 2048645 is in the state CLOSED (CURRENTRELEASE), which is one of the valid states (VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE))
  • dependent Bugzilla bug 2048645 targets the "4.10.0" release, which is one of the valid target releases: 4.10.0
  • bug has dependents

Requesting review from QA contact:
/cc @wangke19

Details

In response to this:

[release-4.9] Bug 2050632: UPSTREAM: : Give warning when ipFamilyPolicy implicitly set

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@aojea
Copy link

aojea commented Feb 4, 2022

/assign

Comment on lines 22 to 25
Copy link

@aojea aojea Feb 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the message should be short

field.NewPath("service","spec", "ipFamilyPolicy").String() + "must be RequireDualStack or PreferDualStack when multiple 'ipFamilies' are specified, this operation will fail starting with Red Hat OpenShift Platform 4.10."

Comment on lines 16 to 18
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sttts the field population happens on the Rest Create and Update hooks that AFAIK run after the admission , so this statement is not true, am I wrong?

@aojea
Copy link

aojea commented Feb 9, 2022

I would add a test , you can complete this and drop it as a file in test/integration/dualstack/patch_warnings_test.go

/*
Copyright 2022 The Kubernetes Authors.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package service

import (
	"bytes"
	"context"
	"fmt"
	"strings"
	"testing"

	v1 "k8s.io/api/core/v1"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/apimachinery/pkg/util/intstr"
	utilfeature "k8s.io/apiserver/pkg/util/feature"
	clientset "k8s.io/client-go/kubernetes"
	restclient "k8s.io/client-go/rest"
	featuregatetesting "k8s.io/component-base/featuregate/testing"
	kubeapiservertesting "k8s.io/kubernetes/cmd/kube-apiserver/app/testing"
	"k8s.io/kubernetes/pkg/features"
	"k8s.io/kubernetes/test/integration/framework"
)

func Test_ServiceDualStackIPFamilyPolicy(t *testing.T) {
	// Create an IPv4IPv6 dual stack control-plane
	serviceCIDR := "10.0.0.0/16"
	secondaryServiceCIDR := "2001:db8:1::/112"
	defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.IPv6DualStack, true)()

	s := kubeapiservertesting.StartTestServerOrDie(t, nil, []string{"--service-cluster-ip-range", serviceCIDR + "," + secondaryServiceCIDR}, framework.SharedEtcd())
	defer s.TearDownFn()

	b := &bytes.Buffer{}
	warningWriter := restclient.NewWarningWriter(b, restclient.WarningWriterOptions{})
	s.ClientConfig.WarningHandler = warningWriter
	client := clientset.NewForConfigOrDie(s.ClientConfig)

	singleStack := v1.IPFamilyPolicySingleStack
	var testcases = []struct {
		name           string
		clusterIPs     []string
		ipFamilies     []v1.IPFamily
		ipFamilyPolicy *v1.IPFamilyPolicyType
		warnings       int
	}{
		{
			name:           "Single Stack - IPFamilyPolicy set",
			clusterIPs:     []string{},
			ipFamilies:     []v1.IPFamily{v1.IPv4Protocol},
			ipFamilyPolicy: &singleStack,
		},
		{
			name:           "Single Stack - IPFamilyPolicy nil",
			clusterIPs:     []string{},
			ipFamilies:     []v1.IPFamily{v1.IPv4Protocol},
			ipFamilyPolicy: nil,
		},
		{
			name:           "Dual Stack - IPFamilyPolicy nil",
			clusterIPs:     []string{},
			ipFamilies:     []v1.IPFamily{v1.IPv4Protocol, v1.IPv6Protocol},
			ipFamilyPolicy: nil,
			warnings:       1,
		},
		{
			// COMPLETE HERE WITH MORE VARIATIONS
		},
	}

	for i, tc := range testcases {
		tc := tc
		t.Run(tc.name, func(t *testing.T) {

			svc := &v1.Service{
				ObjectMeta: metav1.ObjectMeta{
					Name: fmt.Sprintf("svc-test-%d", i), // use different services for each test
				},
				Spec: v1.ServiceSpec{
					Type:           v1.ServiceTypeClusterIP,
					ClusterIPs:     tc.clusterIPs,
					IPFamilies:     tc.ipFamilies,
					IPFamilyPolicy: tc.ipFamilyPolicy,
					Ports: []v1.ServicePort{
						{
							Port:       443,
							TargetPort: intstr.FromInt(443),
						},
					},
				},
			}

			// create a service
 			_, err := client.CoreV1().Services(metav1.NamespaceDefault).Create(context.TODO(), svc, metav1.CreateOptions{})
			if err != nil {
				t.Fatal(err)
			}

			if tc.warnings > 0 {
				//expectedPatchWarning := field.NewPath("service", "spec", "ipFamilyPolicy").String() + "must be RequireDualStack or PreferDualStack when multiple 'ipFamilies' are specified, this operation will fail starting with Red Hat OpenShift Platform 4.10."
				assertWarningCount(t, warningWriter, 1)
				//assertWarningMessage(t, b, expectedPatchWarning)
			}

		})
	}
}

type warningCounter interface {
	WarningCount() int
}

func assertWarningCount(t *testing.T, counter warningCounter, expected int) {
	if counter.WarningCount() != expected {
		t.Errorf("unexpected warning count, expected: %v, got: %v", expected, counter.WarningCount())
	}
}

func assertWarningMessage(t *testing.T, b *bytes.Buffer, expected string) {
	defer b.Reset()
	actual := b.String()
	if len(expected) == 0 && len(actual) != 0 {
		t.Errorf("unexpected warning message, expected no warning, got: %v", actual)
	}
	if len(expected) == 0 {
		return
	}
	if !strings.Contains(actual, expected) {
		t.Errorf("unexpected warning message, expected: %v, got: %v", expected, actual)
	}
}

@aojea
Copy link

aojea commented Feb 9, 2022

not an expert on the backport process, should this not a UPSTREAM: <drop>: commit? we don't want to carry this to 4.10

@openshift-ci-robot
Copy link

@andreaskaris: the contents of this pull request could not be automatically validated.

The following commits could not be validated and must be approved by a top-level approver:

Comment /validate-backports to re-evaluate validity of the upstream PRs, for example when they are merged upstream.

In kube 1.21 and 1.22 (OCP 4.8 and 4.9), the apiserver would default
the value of `ipFamilyPolicy` to `RequireDualStack` if you created a
Service with two `ipFamilies` or two `clusterIPs` but no explicitly
specified `ipFamilyPolicy`. In 1.23/4.10, you must explicitly specify
either PreferDualStack or RequireDualStack for DualStack services.
Emit a warning in 4.8 and 4.9 to raise awareness about the upcoming
API changes.

OpenShift 4.8 and 4.9 only, BZ 2045576

Signed-off-by: Andreas Karis <ak.karis@gmail.com>
@openshift-bot
Copy link

/retest-required

Please review the full test history for this PR and help us cut down flakes.

24 similar comments
@openshift-bot
Copy link

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-ci
Copy link

openshift-ci bot commented Apr 13, 2022

@andreaskaris: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-agnostic-cmd ec1d95e link false /test e2e-agnostic-cmd

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@openshift-bot
Copy link

/retest-required

Please review the full test history for this PR and help us cut down flakes.

1 similar comment
@openshift-bot
Copy link

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-merge-robot openshift-merge-robot merged commit acf7a50 into openshift:release-4.9 Apr 13, 2022
@openshift-ci
Copy link

openshift-ci bot commented Apr 13, 2022

@andreaskaris: All pull requests linked via external trackers have merged:

Bugzilla bug 2050632 has been moved to the MODIFIED state.

Details

In response to this:

[release-4.9] Bug 2050632: UPSTREAM: : Give warning when ipFamilyPolicy implicitly set

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. backport-risk-assessed Indicates a PR to a release branch has been evaluated and considered safe to accept. bugzilla/severity-low Referenced Bugzilla bug's severity is low for the branch this PR is targeting. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. lgtm Indicates that a PR is ready to be merged.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants