OCPBUGS-6661, OCPBUGS-9464: Move mTLS CRL handling into the router, and fix accidental duplication of CRLs by rfredette · Pull Request #930 · openshift/cluster-ingress-operator

rfredette · 2023-05-11T04:37:20Z

This PR moves CRL lifecycle management to the router pod to avoid hitting the configmap max size (1MB), and fixes a bug that caused duplicate CRLs to be downloaded.

This PR Includes 2 new e2e tests:

TestMTLSWithCRLs creates an ingress controller with mTLS enabled and client CA certificates that include CRLs. It verifies that certificates that are signed by each of the client CAs are accepted, certificates revoked by each of the CAs are rejected, and certificates not signed by any of the CAs are also rejected.
TestCRLUpdate creates an ingress controller with mTLS enabled and client CA certificates that include CRLs. It verifies that the CRLs that are downloaded on startup match the CRLs specified in the CA bundle, and waits until all CRLs have expired once, making sure that each update only updates the required CRLs, and that all updates download the correct CRLs.

openshift-ci-robot · 2023-05-11T04:37:25Z

@rfredette: This pull request references Jira Issue OCPBUGS-9464, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.14.0) matches configured target version for branch (4.14.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @lihongan

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

This PR contains one new test with multiple test cases for the fix in openshift/router#472.

The new test creates an ingress controller with mTLS enabled and client CA certificates that include CRLs. It verifies that the CRLs that are downloaded on startup match the CRLs specified in the CA bundle, and waits until all CRLs have expired once, making sure that each update only updates the required CRLs, and that all updates download the correct CRLs.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

rfredette · 2023-05-17T18:06:52Z

/jira refresh

openshift-ci-robot · 2023-05-17T18:06:56Z

@rfredette: This pull request references Jira Issue OCPBUGS-9464, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.14.0) matches configured target version for branch (4.14.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @lihongan

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2023-05-17T18:13:27Z

@rfredette: This pull request references Jira Issue OCPBUGS-9464, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.14.0) matches configured target version for branch (4.14.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @lihongan

Details

In response to this:

This PR moves CRL lifecycle management to the router pod to avoid hitting the configmap max size (1MB), and fixes a bug that caused duplicate CRLs to be downloaded.

This PR Includes 2 new e2e tests:

TestMTLSWithCRLs creates an ingress controller with mTLS enabled and client CA certificates that include CRLs. It verifies that certificates that are signed by each of the client CAs are accepted, certificates revoked by each of the CAs are rejected, and certificates not signed by any of the CAs are also rejected.

TestCRLUpdate creates an ingress controller with mTLS enabled and client CA certificates that include CRLs. It verifies that the CRLs that are downloaded on startup match the CRLs specified in the CA bundle, and waits until all CRLs have expired once, making sure that each update only updates the required CRLs, and that all updates download the correct CRLs.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

rfredette · 2023-05-17T19:35:33Z

operator tests failed on unrelated flakes.
/retest-required

frobware · 2023-05-19T06:36:31Z

/retest-required

frobware · 2023-05-19T16:49:31Z

/retest-required

And a reminder to myself, [some of] the tests are failing because we're hard coding a reference to @rfredette's router image.

test/e2e/util_certgen.go

pkg/operator/controller/crl/crl_configmap.go

Leave a stub of the CRL controller to clean up any existing configmaps. The stub controller will need to be removed in a future release Use cluster-wide proxy for CRL downloads when available Add a test with several test cases to test CRL management

rfredette · 2023-05-22T18:24:28Z

Operator suites failed on TestMTLSWithCRLs and TestCRLUpdate, but they were kicked off before the router PR was merged, so likely they didn't have the required router version.
/retest-required

rfredette · 2023-05-22T21:04:58Z

This PR has passed aws-ovn-upgrade before, and I don't see any obvious reason it should be failing now.
/retest-required

frobware · 2023-05-23T09:57:40Z

/retest

frobware · 2023-05-23T11:26:34Z

/lgtm

openshift-ci · 2023-05-23T12:01:26Z

@rfredette: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

Miciah · 2023-05-23T14:49:26Z

/approve

openshift-ci · 2023-05-23T14:56:53Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Miciah

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [Miciah]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci-robot · 2023-05-23T15:00:49Z

@rfredette: Jira Issue OCPBUGS-9464: All pull requests linked via external trackers have merged:

Jira Issue OCPBUGS-9464 has been moved to the MODIFIED state.

Details

In response to this:

This PR moves CRL lifecycle management to the router pod to avoid hitting the configmap max size (1MB), and fixes a bug that caused duplicate CRLs to be downloaded.

This PR Includes 2 new e2e tests:

TestMTLSWithCRLs creates an ingress controller with mTLS enabled and client CA certificates that include CRLs. It verifies that certificates that are signed by each of the client CAs are accepted, certificates revoked by each of the CAs are rejected, and certificates not signed by any of the CAs are also rejected.

TestCRLUpdate creates an ingress controller with mTLS enabled and client CA certificates that include CRLs. It verifies that the CRLs that are downloaded on startup match the CRLs specified in the CA bundle, and waits until all CRLs have expired once, making sure that each update only updates the required CRLs, and that all updates download the correct CRLs.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

rfredette · 2023-05-23T15:08:26Z

/cherry-pick release-4.13
/cherry-pick release-4.12
/cherry-pick release-4.11

openshift-cherrypick-robot · 2023-05-23T15:09:09Z

@rfredette: #930 failed to apply on top of branch "release-4.13":

Applying: Remove CRL management code from ingress operator
Using index info to reconstruct a base tree...
M	go.mod
M	go.sum
M	vendor/modules.txt
Falling back to patching base and 3-way merge...
Auto-merging vendor/modules.txt
Removing vendor/github.com/operator-framework/api/pkg/operators/zz_generated.deepcopy.go
Removing vendor/github.com/operator-framework/api/pkg/operators/subscription_types.go
Removing vendor/github.com/operator-framework/api/pkg/operators/operatorgroup_types.go
Removing vendor/github.com/operator-framework/api/pkg/operators/installplan_types.go
Removing vendor/github.com/operator-framework/api/pkg/operators/clusterserviceversion_types.go
Removing vendor/github.com/operator-framework/api/pkg/operators/catalogsource_types.go
Removing vendor/github.com/blang/semver/sql.go
Removing vendor/github.com/blang/semver/sort.go
Removing vendor/github.com/blang/semver/semver.go
Removing vendor/github.com/blang/semver/range.go
Removing vendor/github.com/blang/semver/package.json
Removing vendor/github.com/blang/semver/json.go
Removing vendor/github.com/blang/semver/README.md
Removing vendor/github.com/blang/semver/LICENSE
Removing vendor/github.com/blang/semver/.travis.yml
Auto-merging go.sum
Auto-merging go.mod
CONFLICT (content): Merge conflict in go.mod
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 Remove CRL management code from ingress operator
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

Details

In response to this:

/cherry-pick release-4.13
/cherry-pick release-4.12
/cherry-pick release-4.11

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

sjenning · 2023-05-23T22:04:10Z

This broke hypershift

https://github.com/openshift/cluster-ingress-operator/pull/930/files#diff-03760f03eeac1174e38e1ec5fecadcab4cb38d05e00509f201d7ae64dc1c7c36R1044-R1048

New behavior CIO passing its own proxy config down to operands causing router pods to fail.

We are trying to gate CIO so this does happen openshift/release#39246

We do have a blocking job on the ci release stream so it is blocked atm.

Miciah · 2023-05-23T22:28:26Z

This broke hypershift

https://github.com/openshift/cluster-ingress-operator/pull/930/files#diff-03760f03eeac1174e38e1ec5fecadcab4cb38d05e00509f201d7ae64dc1c7c36R1044-R1048

New behavior CIO passing its own proxy config down to operands causing router pods to fail.

That is surprising. Are you saying that HyperShift always configures a cluster-wide egress proxy for cluster-ingress-operator? What connection is the router making that is erroneously using the proxy? Is there a bug report?

sjenning · 2023-05-24T01:51:32Z

Not sure, but after this PR, we see this in our router pods

  containers:
  - env:
...
    - name: HTTPS_PROXY
      value: socks5://127.0.0.1:8090
    - name: HTTP_PROXY
      value: socks5://127.0.0.1:8090
    - name: NO_PROXY
      value: kube-apiserver

https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift_hypershift/2586/pull-ci-openshift-hypershift-main-e2e-aws/1661043100476248064/artifacts/e2e-aws/run-e2e/artifacts/TestCreateCluster_PreTeardownClusterDump/hostedcluster-example-gh9h2/namespaces/openshift-ingress/pods/router-default-68dd6bbc7d-75j47/router-default-68dd6bbc7d-75j47.yaml

Those are the proxy envvars we set for the cluster-ingress-operator on the hosted control plane, not meant to be passed down to operands, which run on the hosted cluster nodes.

Before we did not see this

https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift_hypershift/2588/pull-ci-openshift-hypershift-main-e2e-aws/1661016707734441984/artifacts/e2e-aws/run-e2e/artifacts/TestCreateCluster_PreTeardownClusterDump/hostedcluster-example-wdh7c/namespaces/openshift-ingress/pods/router-default-84cd478cc6-5jd9b/router-default-84cd478cc6-5jd9b.yaml

sjenning · 2023-05-24T01:58:26Z

Are you saying that HyperShift always configures a cluster-wide egress proxy for cluster-ingress-operator?

We configure the ingress-operator with a sidecar that runs konnectivity so that it can communicate with things that run on the cluster nodes (in a different service network) and use proxy envvars to direct operator traffic through the konnectivity sidecar.

This was merged in openshift#930. In hypershift the operator pod runs in a different cluster than the router pod, and so they might have different proxy configs. We set ootb proxy settings for the operator to go through konnectivity. I think because this got introduced in this PR https://github.com/openshift/cluster-ingress-operator/pull/930/files\#diff-03760f03eeac1174e38e1ec5fecadcab4cb38d05e00509f201d7ae64dc1c7c36R1044-R1048 now those proxy settings are being propagated down to the router pod and so breaking hypershift. See an example on how we diferenciate in the cluster network operator https://github.com/openshift/hypershift/blob/f03113aec2545aa08f85ab0e2fa0a790f709aacb/control-plane-operator/controllers/hostedcontrolplane/cno/clusternetworkoperator.go\#L341-L347 https://github.com/openshift/cluster-network-operator/blob/7df32ec5213e2abfcff18cd22a63e7f20a5cccdc/pkg/network/cloud_network.go\#L98

enxebre · 2023-05-24T10:45:32Z

That is surprising. Are you saying that HyperShift always configures a cluster-wide egress proxy for cluster-ingress-operator? What connection is the router making that is erroneously using the proxy? Is there a bug report?

In hypershift the operator pod runs in a different cluster than the router pod, and so they might have different proxy configs.
We set ootb proxy settings for the operator to go through konnectivity.
I think because this got introduced in this PR https://github.com/openshift/cluster-ingress-operator/pull/930/files#diff-03760f03eeac1174e38e1ec5fecadcab4cb38d05e00509f201d7ae64dc1c7c36R1044-R1048
now those proxy settings are being propagated down to the router pod and so breaking hypershift.

See an example on how we diferenciate in the cluster network operator https://github.com/openshift/hypershift/blob/f03113aec2545aa08f85ab0e2fa0a790f709aacb/control-plane-operator/controllers/hostedcontrolplane/cno/clusternetworkoperator.go#L341-L347

https://github.com/openshift/cluster-network-operator/blob/7df32ec5213e2abfcff18cd22a63e7f20a5cccdc/pkg/network/cloud_network.go#L98

enxebre · 2023-05-24T10:46:50Z

I'd suggest we revert that particular change #937.
Then we merge openshift/release#39246 and gate on a green run.
Then we re-introduce the proxy change with a test driven safety check.

dgoodwin · 2023-05-24T11:57:15Z

Policy required us to open a full revert rather than partial, see #938 for details.

Miciah · 2023-05-24T13:55:08Z

Those are the proxy envvars we set for the cluster-ingress-operator on the hosted control plane, not meant to be passed down to operands, which run on the hosted cluster nodes.

According to the design of the cluster-wide egress proxy feature, operators should propagate proxy configuration to their operands so that the operands use the proxy when communicating externally. This PR fixes a critical customer bug, and it follows the design specifications for the cluster-wide egress proxy feature in doing so.

Can you provide more details about what was failing on HyperShift? Specifically, what failed when the proxy was configured on the router pods, what were the specific error messages, and what specific URLs were involved? Is HyperShift not setting NO_PROXY to an appropriate value?

If instead of copying the environment variables, cluster-ingress-operator read the cluster-wide egress proxy (using the injected kubeconfig for the hosted cluster) and injected environment variables based on those values, would that avoid the HyperShift problem (while still configuring the proxy on the operand as required by the cluster-wide egress proxy design)?

enxebre · 2023-05-24T14:07:57Z

According to the design of the cluster-wide egress proxy feature, operators should propagate proxy configuration to their operands so that the operands use the proxy when communicating externally

Hypershift makes this assumption obsolete. We need to come up with an updated one.

Can you provide more details about what was failing on HyperShift? Specifically, what failed when the proxy was configured on the router pods, what were the specific error messages, and what specific URLs were involved? Is HyperShift not setting NO_PROXY to an appropriate value?

You can see router pod log failures here which prevent the ingress CO from going available
https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift_hypershift/2595/pull-ci-openshift-hypershift-main-e2e-aws/1661091472633499648/artifacts/e2e-aws/run-e2e/artifacts/TestCreateCluster_PreTeardownClusterDump/hostedcluster-example-qpj62/namespaces/openshift-ingress/pods/router-default-794cf955c9-fjdxw/router/router/logs/current.log

If instead of copying the environment variables, cluster-ingress-operator read the cluster-wide egress proxy (using the injected kubeconfig for the hosted cluster) and injected environment variables based on those values, would that avoid the HyperShift problem (while still configuring the proxy on the operand as required by the cluster-wide egress proxy design)?

That sounds reasonable to me. I think this probably deserves a discussion and possibly an enhancement update to make sure we consolidate on the approach for all components that have operator/operands split. cc @csrwng @sjenning

Miciah · 2023-05-25T20:32:09Z

Hypershift makes this assumption obsolete. We need to come up with an updated one.

💥.

If instead of copying the environment variables, cluster-ingress-operator read the cluster-wide egress proxy (using the injected kubeconfig for the hosted cluster) and injected environment variables based on those values, would that avoid the HyperShift problem (while still configuring the proxy on the operand as required by the cluster-wide egress proxy design)?

That sounds reasonable to me. I think this probably deserves a discussion and possibly an enhancement update to make sure we consolidate on the approach for all components that have operator/operands split. cc @csrwng @sjenning

In my opinion, it would be better to amend the design before invalidating it, or find a way for HyperShift to meet its requirements without violating existing designs and API abstractions.

Miciah · 2023-05-25T20:43:29Z

You can see router pod log failures here which prevent the ingress CO from going available
https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift_hypershift/2595/pull-ci-openshift-hypershift-main-e2e-aws/1661091472633499648/artifacts/e2e-aws/run-e2e/artifacts/TestCreateCluster_PreTeardownClusterDump/hostedcluster-example-qpj62/namespaces/openshift-ingress/pods/router-default-794cf955c9-fjdxw/router/router/logs/current.log

The following logs are key:

2023-05-23T20:30:42.394914893Z W0523 20:30:42.394839       1 reflector.go:424] github.com/openshift/router/pkg/router/controller/factory/factory.go:124: failed to list *v1.Route: Get "https://172.31.0.1:443/apis/route.openshift.io/v1/routes?limit=500&resourceVersion=0": proxyconnect tcp: dial tcp 127.0.0.1:8090: connect: connection refused
2023-05-23T20:30:42.394914893Z W0523 20:30:42.394835       1 reflector.go:424] github.com/openshift/router/pkg/router/controller/factory/factory.go:124: failed to list *v1.EndpointSlice: Get "https://172.31.0.1:443/apis/discovery.k8s.io/v1/endpointslices?limit=500&resourceVersion=0": proxyconnect tcp: dial tcp 127.0.0.1:8090: connect: connection refused

The proxy config must not have the service network in noProxy, and so the router controller attempts to connect to the kubernetes.default.svc address through the proxy. Usually, cluster-network-operator would inject the service network(s) into noProxy: https://github.com/openshift/cluster-network-operator/blob//pkg/util/proxyconfig/no_proxy.go#L78-L80

openshift-ci-robot added the bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. label May 11, 2023

openshift-ci bot requested review from alebedev87, gcs278 and lihongan May 11, 2023 04:37

frobware self-assigned this May 17, 2023

rfredette changed the title ~~OCPBUGS-9464: Verify CRLs are updated correctly when they expire~~ OCPBUGS-6661, OCPBUGS-9464: Move mTLS CRL handling into the router, and fix accidental duplication of CRLs May 17, 2023

rfredette mentioned this pull request May 17, 2023

OCPBUGS-6661: Handle mTLS CRLs in the router #891

Closed

openshift-ci-robot added jira/severity-critical Referenced Jira bug's severity is critical for the branch this PR is targeting. and removed jira/severity-important Referenced Jira bug's severity is important for the branch this PR is targeting. labels May 17, 2023

rfredette force-pushed the ocpbugs-9464-duplicate-crls branch from 95eea14 to 67f6a68 Compare May 17, 2023 23:09

frobware reviewed May 19, 2023

View reviewed changes

test/e2e/util_certgen.go Outdated Show resolved Hide resolved

pkg/operator/controller/crl/crl_configmap.go Outdated Show resolved Hide resolved

rfredette added 2 commits May 19, 2023 15:19

Add test case to test CRL updates

210072a

rfredette force-pushed the ocpbugs-9464-duplicate-crls branch 2 times, most recently from 91040f9 to 210072a Compare May 22, 2023 16:32

openshift-ci-robot mentioned this pull request May 22, 2023

OCPBUGS-6661, OCPBUGS-9464: Handle mTLS CRLs, and fix accidental CRL duplication openshift/router#472

Merged

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label May 23, 2023

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 23, 2023

openshift-merge-robot merged commit 108acb3 into openshift:master May 23, 2023

rfredette mentioned this pull request May 23, 2023

[release-4.13] OCPBUGS-13967, OCPBUGS-13964: Handle mTLS CRLs, and fix accidental CRL duplication #935

Merged

This was referenced May 24, 2023

HOSTEDCP-966: Gate more Operators on Hypeshift e2e openshift/release#39246

Merged

HOSTEDCP-966: require hypershift e2e for operators in control plane openshift/release#39641

Closed

enxebre mentioned this pull request May 24, 2023

Stop propagating proxy env to router pod #937

Closed

dgoodwin mentioned this pull request May 24, 2023

Revert "OCPBUGS-6661, OCPBUGS-9464: Move mTLS CRL handling into the router, and fix accidental duplication of CRLs" #938

Merged

rfredette mentioned this pull request May 24, 2023

OCPBUGS-6661, OCPBUGS-9464: Move mTLS CRL handling into the router, and fix accidental duplication of CRLs #939

Merged

Conversation

rfredette commented May 11, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci-robot commented May 11, 2023

Uh oh!

rfredette commented May 17, 2023

Uh oh!

openshift-ci-robot commented May 17, 2023

Uh oh!

openshift-ci-robot commented May 17, 2023

Uh oh!

rfredette commented May 17, 2023

Uh oh!

frobware commented May 19, 2023

Uh oh!

frobware commented May 19, 2023

Uh oh!

Uh oh!

Uh oh!

rfredette commented May 22, 2023

Uh oh!

rfredette commented May 22, 2023

Uh oh!

frobware commented May 23, 2023

Uh oh!

frobware commented May 23, 2023

Uh oh!

openshift-ci bot commented May 23, 2023

Uh oh!

Miciah commented May 23, 2023

Uh oh!

openshift-ci bot commented May 23, 2023

Uh oh!

openshift-ci-robot commented May 23, 2023

Uh oh!

rfredette commented May 23, 2023

Uh oh!

openshift-cherrypick-robot commented May 23, 2023

Uh oh!

sjenning commented May 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Miciah commented May 23, 2023

Uh oh!

sjenning commented May 24, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sjenning commented May 24, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

enxebre commented May 24, 2023

Uh oh!

enxebre commented May 24, 2023

Uh oh!

dgoodwin commented May 24, 2023

Uh oh!

Miciah commented May 24, 2023

Uh oh!

enxebre commented May 24, 2023

Uh oh!

Miciah commented May 25, 2023

Uh oh!

Miciah commented May 25, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

rfredette commented May 11, 2023 •

edited

Loading

sjenning commented May 23, 2023 •

edited

Loading

sjenning commented May 24, 2023 •

edited

Loading

sjenning commented May 24, 2023 •

edited

Loading