Moving all security group rules to separate resources

[binlab]

• Refactored Security Groups (SG) and rules,
• Moved all rules to separate resources
• Removed two SG which are not needed now
• Changed description all GS and rules
• closes Move all security groups' rules to external resources #29

IMPORTANT NOTE:

after this PR for updating the version of the module need some tricks, due to some Terraform bugs and AWS limitation

Terraform output

...
module.vault.aws_security_group.node: Still destroying... [id=sg-0b29f7b5fd2babdf0, 9m0s elapsed]
module.vault.aws_security_group.node: Still destroying... [id=sg-0b29f7b5fd2babdf0, 9m10s elapsed]
module.vault.aws_security_group.node: Still destroying... [id=sg-0b29f7b5fd2babdf0, 9m20s elapsed]
module.vault.aws_security_group.node: Still destroying... [id=sg-0b29f7b5fd2babdf0, 9m30s elapsed]
module.vault.aws_security_group.node: Still destroying... [id=sg-0b29f7b5fd2babdf0, 9m40s elapsed]
module.vault.aws_security_group.node: Still destroying... [id=sg-0b29f7b5fd2babdf0, 9m50s elapsed]
module.vault.aws_security_group.node: Still destroying... [id=sg-0b29f7b5fd2babdf0, 10m0s elapsed]

Error: Error deleting security group: DependencyViolation: resource sg-0b29f7b5fd2babdf0 has a dependent object
	status code: 400, request id: b6cd4391-8972-498c-8f3e-d70cf189eaed

Terraform bugs

Terraform has a knows bug it can't remove inline ingress or egress from the aws_security_group resource, more about this issue:

• Cannot remove inline ingress or egress from aws_security_group hashicorp/terraform-provider-aws#20046
• Removing ingress rules from aws_security_group is not detected hashicorp/terraform-provider-aws#4399.

So all separate rules will conflict with existing inline rules, the only way to fix it currently is to change the Name of Description on Security Group which trigger recreating and truncate inline rules, but is connected to another problem - AWS limitation

AWS limitation

AWS has a couple of resources that are available to the user in view mode but managed by AWS and there is no way to control it by Terraform. In this case, this is a Network Interface that is assigned to Subnets. Each Network Interface has connectivity with a Security Group as dependencies, more about this issue:

• AWS security groups not being destroyed hashicorp/terraform-provider-aws#2445 (comment)

Solution

A possible way to solve it and update an existing cluster is re-creating a VPC. This will trigger a replacing ALL network resources but leave untouched other module resources. One of the ways to achieve this, temporary change the CIDR of the network which is assigned to the Vault cluster (after that you can return back after the next apply)