✨ Remove ingress and egress rules from vpc default security group #4707

fiunchinho · 2023-12-20T12:38:32Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

We want to secure the VPC as much as possible.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #4675

Special notes for your reviewer:

Checklist:

Release note:

Remove ingress and egress rules from vpc default security group

k8s-ci-robot · 2023-12-20T12:38:35Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

fiunchinho · 2023-12-20T12:41:09Z

/test all

fiunchinho · 2023-12-20T13:22:04Z

/test all

fiunchinho · 2023-12-20T13:38:35Z

/test all

fiunchinho · 2023-12-20T13:51:56Z

/test all

fiunchinho · 2023-12-20T15:18:12Z

/test all

fiunchinho · 2023-12-21T11:35:35Z

api/v1beta1/awscluster_conversion.go

@@ -90,7 +90,7 @@ func (src *AWSCluster) ConvertTo(dstRaw conversion.Hub) error {
 		restoreIPAMPool(restored.Spec.NetworkSpec.VPC.IPv6.IPAMPool, dst.Spec.NetworkSpec.VPC.IPv6.IPAMPool)
 	}

-	dst.Spec.NetworkSpec.AdditionalControlPlaneIngressRules = restored.Spec.NetworkSpec.AdditionalControlPlaneIngressRules


This was already done on line 75

fiunchinho · 2023-12-21T11:36:27Z

api/v1beta2/network_types.go

+	// SecureDefaultVPCSecurityGroup specifies whether the default VPC security group ingress
+	// and egress rules should be removed.
+	// +optional
+	SecureDefaultVPCSecurityGroup bool `json:"secureDefaultVPCSecurityGroup,omitempty"`


I'd remove this field on the next major release containing breaking changes. And change CAPA behavior as if this field was always true. Should we add a @todo somewhere?

At least, we should add here in the description that the default will become true in the future, while currently it's false. Regarding a TODO, an issue could be opened for the next milestone.

It might be nice if this is true for new clusters by default, but it stays false for older clusters. The defaulting can happen if in the validation webhook if the vpc id of a managed VPC is already filled.

Part of me was also considering of not adding this flag at all, which can be confusing?

I'd prefer not adding the field/flag, because it makes things simpler and easier to understand. I don't think users should rely on this security group anyway. But someone in Slack suggested adding it to be backwards compatible, which also makes sense.

fiunchinho · 2023-12-21T17:28:06Z

pkg/cloud/services/securitygroup/securitygroups.go

+			},
+		}
+		err = s.revokeSecurityGroupIngressRules(defaultSecurityGroupID, ingressRules)
+		if awserrors.IsIgnorableSecurityGroupError(errors.Cause(err)) != nil {


The ingress and egress rules are removed successfully, but on the next reconciliations, I keep getting

I1221 17:25:07.051779 1 recorder.go:104] "events: Failed to revoke security group ingress rules [protocol=-1/range=[-1--1]/description=] for SecurityGroup \"sg-09b4f9143ee0aeda4\": InvalidPermission.NotFound: The specified rule does not exist in this security group.\n\tstatus code: 400, request id: df8bbb6a-5674-4f0c-a255-8a2da44eb3e0" type="Warning" object={"kind":"AWSCluster","namespace":"org-giantswarm","name":"jose4","uid":"87039375-5a6c-4e7e-bbce-b784c05085e2","apiVersion":"infrastructure.cluster.x-k8s.io/v1beta2","resourceVersion":"811993128"} reason="FailedRevokeSecurityGroupIngressRules"

What would be the best way to unwrap the error so that it's ignored? I figured that we don't want to do anything if the ingress or egress rules were already removed.

We can use the same logic as func (s *Service) ReconcileSecurityGroups: first read existing rules with s.getSecurityGroupIngressRules and then reconcile the differences.

I don't think it makes sense in this case, because all we want is to delete the existing rules. We don't care what's inside, or "the differences". We want the group to have no rules at all.

AndiDog · 2023-12-28T15:45:15Z

api/v1beta2/network_types.go

+	// SecureDefaultVPCSecurityGroup specifies whether the default VPC security group ingress
+	// and egress rules should be removed.
+	// +optional
+	SecureDefaultVPCSecurityGroup bool `json:"secureDefaultVPCSecurityGroup,omitempty"`


At least, we should add here in the description that the default will become true in the future, while currently it's false. Regarding a TODO, an issue could be opened for the next milestone.

AndiDog · 2023-12-28T15:47:30Z

pkg/cloud/services/securitygroup/securitygroups.go

+// The VPC default security group is created by AWS and cannot be deleted.
+// But we can revoke all ingress and egress rules from it to make it more secure. This security group is not used by CAPA.
+func (s *Service) revokeIngressAndEgressRulesFromVPCDefaultSecurityGroup() error {
+	if s.scope.VPC().SecureDefaultVPCSecurityGroup {


Please return early instead of indenting the whole function code. Or alternatively, you could move the condition to the caller.

AndiDog · 2023-12-28T15:49:49Z

pkg/cloud/services/securitygroup/securitygroups.go

+				filter.EC2.SecurityGroupName("default"),
+			},
+		})
+		if err != nil {


len(securityGroups) != 1 is also an error

The default security group can't be removed. When creating a new security group, group names must be unique, so there can't be two groups named default. I don't think len(securityGroups) can ever be != 1.

AndiDog · 2023-12-28T15:52:37Z

pkg/cloud/services/securitygroup/securitygroups.go

+			},
+		}
+		err = s.revokeSecurityGroupIngressRules(defaultSecurityGroupID, ingressRules)
+		if awserrors.IsIgnorableSecurityGroupError(errors.Cause(err)) != nil {


We can use the same logic as func (s *Service) ReconcileSecurityGroups: first read existing rules with s.getSecurityGroupIngressRules and then reconcile the differences.

AndiDog · 2023-12-28T15:55:40Z

pkg/cloud/services/securitygroup/securitygroups_test.go

@@ -90,6 +92,29 @@ func TestReconcileSecurityGroups(t *testing.T) {
 				},
 			},
 			expect: func(m *mocks.MockEC2APIMockRecorder) {


We should also have a test for reconciliation if the rules are already gone – in that case, no requests apart from DescribeSecurityGroups and listing its rules should be done.

Added a test case to cover that.

vincepri · 2024-01-10T17:14:54Z

api/v1beta2/network_types.go

+
+	// SecureDefaultVPCSecurityGroup specifies whether the default VPC security group ingress
+	// and egress rules should be removed.
+	// +optional
+	SecureDefaultVPCSecurityGroup bool `json:"secureDefaultVPCSecurityGroup,omitempty"`


Suggested change

// SecureDefaultVPCSecurityGroup specifies whether the default VPC security group ingress

// and egress rules should be removed.

// +optional

SecureDefaultVPCSecurityGroup bool `json:"secureDefaultVPCSecurityGroup,omitempty"`

// EmptyRoutesDefaultVPCSecurityGroup specifies whether the default VPC security group ingress

// and egress rules should be removed.

//

// By default, when creating a VPC, AWS creates a security group called `default` with ingress and egress

// rules that allow traffic from anywhere. The group could be used as a potential surface attack and

// it's generally suggested that the group rules are removed or modified appropriately.

//

// NOTE: This only applies when the VPC is managed by the Cluster API AWS controller.

//

// +optional

EmptyRoutesDefaultVPCSecurityGroup bool `json:"emptyRoutesDefaultVPCSecurityGroup,omitempty"`

AndiDog · 2024-01-15T09:18:24Z

pkg/cloud/services/securitygroup/securitygroups.go

+	}
+	err = s.revokeSecurityGroupIngressRules(defaultSecurityGroupID, ingressRules)
+	if awserrors.IsIgnorableSecurityGroupError(errors.Cause(err)) != nil {
+		return errors.Wrapf(err, "failed to revoke ingress rules from vpc default security group in vpc %q", s.scope.VPC().ID)


Suggested change

return errors.Wrapf(err, "failed to revoke ingress rules from vpc default security group in vpc %q", s.scope.VPC().ID)

return errors.Wrapf(err, "failed to revoke ingress rules from vpc default security group %q in VPC %q", defaultSecurityGroupID, s.scope.VPC().ID)

AndiDog · 2024-01-15T09:22:36Z

pkg/cloud/services/securitygroup/securitygroups.go

+		},
+	}
+	err = s.revokeSecurityGroupIngressRules(defaultSecurityGroupID, ingressRules)
+	if awserrors.IsIgnorableSecurityGroupError(errors.Cause(err)) != nil {


Are GroupNotFound/PermissionNotFound (which IsIgnorableSecurityGroupError handles) really errors that we want to ignore? The default group cannot be deleted, so such errors here should fail, I think. From the test, it seems that you found that AWS throws a permission error if the given rules do not exist. So we could instead use ec2:DescribeSecurityGroupRules (new permission which CAPA does not have yet) or handle the very specific error?

After deleting the ingress/egress rules from the default security group, the next time we try to delete them we'll get a PermissionNotFound error. I think this one can be ignored, because that's the state we want. To avoid swallowing the GroupNotFound error, that should not happen in this scenario, we could handle PermissionNotFound more specifically, like you mentioned. I'll add a new IsPermissionNotFoundError function. How does that sound?

AndiDog

/lgtm

AndiDog · 2024-01-16T09:31:20Z

@vincepri did you mean to /approve instead of GitHub "approve"?

vincepri

/approve

k8s-ci-robot · 2024-01-16T18:56:47Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: vincepri

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [vincepri]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

richardcase · 2024-01-23T14:47:45Z

/milestone v2.4.0

k8s-ci-robot requested review from cnmcavoy and dlipovetsky December 20, 2023 12:38

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Dec 20, 2023

fiunchinho changed the title ~~Remove ingress and egress rules from vpc default security group~~ ✨ Remove ingress and egress rules from vpc default security group Dec 20, 2023

fiunchinho force-pushed the secure-default-vpc-sg branch 2 times, most recently from c48b5d6 to 6f6e516 Compare December 20, 2023 13:21

fiunchinho force-pushed the secure-default-vpc-sg branch from 6f6e516 to 3435308 Compare December 20, 2023 13:36

fiunchinho force-pushed the secure-default-vpc-sg branch from 3435308 to 2544b32 Compare December 20, 2023 13:51

fiunchinho force-pushed the secure-default-vpc-sg branch from 2544b32 to a91b151 Compare December 20, 2023 15:17

fiunchinho mentioned this pull request Dec 20, 2023

CAPA Security Hardening giantswarm/roadmap#3015

Closed

3 tasks

fiunchinho marked this pull request as ready for review December 20, 2023 16:06

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 20, 2023

k8s-ci-robot requested a review from richardcase December 20, 2023 16:06

fiunchinho commented Dec 21, 2023

View reviewed changes

fiunchinho force-pushed the secure-default-vpc-sg branch 3 times, most recently from ec24551 to 5f19835 Compare December 21, 2023 17:01

fiunchinho commented Dec 21, 2023

View reviewed changes

AndiDog reviewed Dec 28, 2023

View reviewed changes

fiunchinho force-pushed the secure-default-vpc-sg branch from 5f19835 to 26208de Compare January 9, 2024 16:26

vincepri reviewed Jan 10, 2024

View reviewed changes

fiunchinho force-pushed the secure-default-vpc-sg branch 3 times, most recently from 9c6af09 to 30a9257 Compare January 11, 2024 15:22

fiunchinho requested a review from vincepri January 11, 2024 15:28

AndiDog reviewed Jan 15, 2024

View reviewed changes

fiunchinho force-pushed the secure-default-vpc-sg branch from 30a9257 to 50dc78d Compare January 15, 2024 10:09

Remove ingress and egress rules from vpc default security group

c066167

fiunchinho force-pushed the secure-default-vpc-sg branch from 50dc78d to c066167 Compare January 15, 2024 12:11

fiunchinho requested a review from AndiDog January 15, 2024 12:11

AndiDog reviewed Jan 15, 2024

View reviewed changes

k8s-ci-robot assigned AndiDog Jan 15, 2024

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 15, 2024

vincepri approved these changes Jan 15, 2024

View reviewed changes

vincepri approved these changes Jan 16, 2024

View reviewed changes

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 16, 2024

k8s-ci-robot merged commit c777c9d into kubernetes-sigs:main Jan 16, 2024
17 checks passed

k8s-ci-robot added this to the v2.4.0 milestone Jan 23, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

✨ Remove ingress and egress rules from vpc default security group #4707

✨ Remove ingress and egress rules from vpc default security group #4707

fiunchinho commented Dec 20, 2023 •

edited

Loading

k8s-ci-robot commented Dec 20, 2023

fiunchinho commented Dec 20, 2023

fiunchinho commented Dec 20, 2023

fiunchinho commented Dec 20, 2023

fiunchinho commented Dec 20, 2023

fiunchinho commented Dec 20, 2023

fiunchinho Dec 21, 2023

fiunchinho Dec 21, 2023

AndiDog Dec 28, 2023

vincepri Jan 2, 2024

fiunchinho Jan 8, 2024

fiunchinho Dec 21, 2023 •

edited

Loading

AndiDog Dec 28, 2023

fiunchinho Jan 8, 2024

AndiDog Dec 28, 2023

AndiDog Dec 28, 2023

fiunchinho Jan 9, 2024

AndiDog Dec 28, 2023

fiunchinho Jan 8, 2024

AndiDog Dec 28, 2023

AndiDog Dec 28, 2023

fiunchinho Jan 11, 2024

vincepri Jan 10, 2024

AndiDog Jan 15, 2024

AndiDog Jan 15, 2024

fiunchinho Jan 15, 2024

AndiDog left a comment

AndiDog commented Jan 16, 2024

vincepri left a comment

k8s-ci-robot commented Jan 16, 2024

richardcase commented Jan 23, 2024

	return errors.Wrapf(err, "failed to revoke ingress rules from vpc default security group in vpc %q", s.scope.VPC().ID)
	return errors.Wrapf(err, "failed to revoke ingress rules from vpc default security group %q in VPC %q", defaultSecurityGroupID, s.scope.VPC().ID)

✨ Remove ingress and egress rules from vpc default security group #4707

✨ Remove ingress and egress rules from vpc default security group #4707

Conversation

fiunchinho commented Dec 20, 2023 • edited Loading

k8s-ci-robot commented Dec 20, 2023

fiunchinho commented Dec 20, 2023

fiunchinho commented Dec 20, 2023

fiunchinho commented Dec 20, 2023

fiunchinho commented Dec 20, 2023

fiunchinho commented Dec 20, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fiunchinho Dec 21, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AndiDog left a comment

Choose a reason for hiding this comment

AndiDog commented Jan 16, 2024

vincepri left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Jan 16, 2024

richardcase commented Jan 23, 2024

fiunchinho commented Dec 20, 2023 •

edited

Loading

fiunchinho Dec 21, 2023 •

edited

Loading