A security group with a combination of a CIDR, a SG, and self=true, applies but still shows changes #7425
Labels
bug
Addresses a defect in current functionality.
service/ec2
Issues and PRs that pertain to the ec2 service.
Comments
|
Hi @jbscare 👋 Does it work if you combine the two rules with matching ports and protocol? {
"provider": {
"aws": {
"__DEFAULT__": {
"assume_role": {
"role_arn": "arn:aws:iam::798732171007:role/admin"
},
"profile": "chargeup",
"region": "us-east-1",
"version": "~> 1.57.0"
}
}
},
"resource": {
"aws_security_group": {
"group-x": {
"description": "Managed by Terraform",
"egress": [],
"ingress": [],
"name": "group-x",
"vpc_id": "vpc-9bffd1fe"
},
"group-y": {
"description": "Managed by Terraform",
"egress": [],
"ingress": [
{
"from_port": 8989,
"protocol": "tcp",
"security_groups": [
"${aws_security_group.group-x.id}"
],
"to_port": 8990
},
{
"cidr_blocks": [
"10.192.22.52/32"
],
"self": true,
"from_port": 8989,
"protocol": "tcp",
"to_port": 8990
}
],
"name": "group-y",
"vpc_id": "vpc-9bffd1fe"
}
}
}
}For historical reasons to match the EC2 API, the Terraform AWS provider should try to combine rules in that manor. Can you let us know? Thanks! |
bflad
added
waiting-response
|
All three rules have the same ports and protocol. As it turns out, starting from the original configuration, if I combine:
... then |
ghost
removed
the
waiting-response
|
The But it doesn't actually do that, I guess? Because the provider combines them before it sends them off to AWS? So maybe the issue is that the provider needs to also want to combine them at plan time? |
|
I'm also still totally mystified as to why the original config behaves differently if you use And then a subsequent apply says no changes. So with a different IP address (or port numbers, or SG ID apparently), it's fine with having three rules, but with the original one, it thinks it needs to combine down to two? |
|
I noticed that the response from the AWS EC2 API seems to be the same regardless: +$ aws ec2 describe-security-groups --filters "Name=group-name,Values=group-y"
{
"SecurityGroups": [
{
"IpPermissionsEgress": [],
"Description": "Managed by Terraform",
"IpPermissions": [
{
"PrefixListIds": [],
"FromPort": 8989,
"IpRanges": [
{
"CidrIp": "10.192.22.52/32"
}
],
"ToPort": 8990,
"IpProtocol": "tcp",
"UserIdGroupPairs": [
{
"UserId": "798732171007",
"GroupId": "sg-05c9cc7e76b83e419"
},
{
"UserId": "798732171007",
"GroupId": "sg-0b5b97a5d904f9ca2"
}
],
"Ipv6Ranges": []
}
],
"GroupName": "group-y",
"VpcId": "vpc-9bffd1fe",
"OwnerId": "798732171007",
"GroupId": "sg-0b5b97a5d904f9ca2"
}
]
}So it looks like the AWS EC2 API is in fact representing these as one rule, whether the provider thinks of them that way or not. |
Given that, why does the provider ever think of them as multiple rules? That seems like the crux of this bug, unless there's some other good reason for it to do that. |
aeschright
added
the
needs-triage
|
We were able to work around this problem in our actual config when it came up before, but it's come up again in a way that we can't work around as easily. Any chance that someone could look into this? It's been labeled
I haven't looked at the code, but this still seems true to me: If the code is for some reason treating these as separate rules, that seems like it may be the cause of the problem. |
aeschright
added
bug
jbscare
changed the title
|
Updating the title because this isn't really about idempotence, it's about convergence. We've run into another couple of instances of this bug recently, and have been able to hack our way around all of them so far, by changing the ports or IP addresses that are permitted, but it's not always easy to do. Is there any chance someone could take a look at this at some point? |
|
I think I'm hitting this same or a very similar issue. I'm pulling cidr_blocks into a security group like this: The contents of the json file look like: The list contains 49 elements. After running Looks like the cidr_blocks list order might be changing, but I don't see what'd be changing the array order. |
|
We recently upgraded to Terraform 0.13, and with version 3.21 of the AWS provider, we still hit this bug from time to time. :^( |
|
Looks like I forgot to update my comment. In my case I found out that I had duplicate elements in the |
|
Alas, that's definitely not our problem. :^( The simple reproducer in the description only has one CIDR block, one security group, and the self=true rule. |
|
It's still happening with terraform 0.14.2 and provider aws 3.21.0, using the exact snippet as in the initial post, just changed vpc-id: |
|
Marking this issue as stale due to inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 30 days it will automatically be closed. Maintainers can also remove the stale label. If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you! |
github-actions
bot
added
the
stale
|
This is still a problem and it'd be nice if someone could take a look at it. |
github-actions
bot
removed
the
stale
|
Marking this issue as stale due to inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 30 days it will automatically be closed. Maintainers can also remove the stale label. If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you! |
github-actions
bot
added
the
stale
|
This is still still a problem and it'd still be nice if someone could take a look at it. |
github-actions
bot
removed
the
stale
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Community Note
Terraform Version
Affected Resource(s)
Terraform Configuration Files
{ "provider": { "aws": { "__DEFAULT__": { "assume_role": { "role_arn": "arn:aws:iam::798732171007:role/admin" }, "profile": "chargeup", "region": "us-east-1", "version": "~> 1.57.0" } } }, "resource": { "aws_security_group": { "group-x": { "description": "Managed by Terraform", "egress": [], "ingress": [], "name": "group-x", "vpc_id": "vpc-9bffd1fe" }, "group-y": { "description": "Managed by Terraform", "egress": [], "ingress": [ { "from_port": 8989, "protocol": "tcp", "security_groups": [ "${aws_security_group.group-x.id}" ], "to_port": 8990 }, { "cidr_blocks": [ "10.192.22.52/32" ], "from_port": 8989, "protocol": "tcp", "to_port": 8990 }, { "from_port": 8989, "protocol": "tcp", "self": true, "to_port": 8990 } ], "name": "group-y", "vpc_id": "vpc-9bffd1fe" } } } }Debug Output
https://gist.github.com/jbscare/6918964f0500c71e3c25560b731e5a4f
Panic Output
No panic output.
Expected Behavior
After applying this config, a second
terraform apply(or aterraform plan) should show that no changes are needed.Actual Behavior
The second
terraform applyshows that it still wants to make changes. Usingterraform planand a pipe to sed to show only the things that it wants to change:That is, it wants to remove ingress.9858312, which contains a CIDR block and a security group, and replace it with ingress.210426432 containing the security group and ingress.2277696117 containing the CIDR block.
Applying the change doesn't seem to change the state; it continues to think it needs to apply this exact change.
Steps to Reproduce
NOTE that this doesn't consistently reproduce the problem. If it doesn't, try
It often reproduces the problem after only one or two tries, but I've sometimes had to try as many as a dozen times.
Important Factoids
We use IAM roles to apply our configs.
If you get to a point where you can reproduce the problem, you can eliminate the problem by changing any of the parameters in the config. For example, if you:
Then a
terraform applycommand will apply the change, and a secondterraform applycommand will say there are no changes.If you then put the config back to the original parameters,
terraform applywill go back to continuing to think there are changes ever after applying.We suspect that the reason it doesn't always reproduce is that the security group ID is also signficant, and that some SG IDs trigger the problem while some don't. Unfortunately, that's not a parameter you can just change in your config.
In our testing, we found that these pairs of security group IDs did reproduce the problem:
aws_security_group.group-x: Creation complete after 1s (ID: sg-0a92d7e18ae410e22)
aws_security_group.group-y: Creation complete after 1s (ID: sg-0a3cc8872feb026da)
aws_security_group.group-x: Creation complete after 1s (ID: sg-01172b325c5570bda)
aws_security_group.group-y: Creation complete after 1s (ID: sg-0007241960f06f4a0)
aws_security_group.group-x: Creation complete after 1s (ID: sg-05c9cc7e76b83e419)
aws_security_group.group-y: Creation complete after 1s (ID: sg-0b5b97a5d904f9ca2)
While these combinations did not reproduce the problem:
aws_security_group.group-x: Creation complete after 1s (ID: sg-0eaa35cb65915d1f8)
aws_security_group.group-y: Creation complete after 1s (ID: sg-0063137ed8d0db6c8)
aws_security_group.group-x: Creation complete after 1s (ID: sg-0e7b0209bcbc66d83)
aws_security_group.group-y: Creation complete after 1s (ID: sg-07c0f432f1a920bf0)
aws_security_group.group-x: Creation complete after 1s (ID: sg-0f985965ac7712fb8)
aws_security_group.group-y: Creation complete after 1s (ID: sg-0813d8604667c8371)
aws_security_group.group-x: Creation complete after 1s (ID: sg-0d37d705636acc7fb)
aws_security_group.group-y: Creation complete after 1s (ID: sg-0c66b0c05258f92f0)
aws_security_group.group-x: Creation complete after 1s (ID: sg-0ebfed8677b378e81)
aws_security_group.group-y: Creation complete after 1s (ID: sg-0f389437a07360b42)
aws_security_group.group-x: Creation complete after 1s (ID: sg-0d83646a14c0e3f30)
aws_security_group.group-y: Creation complete after 1s (ID: sg-0e28c2ecea3dac365)
Just in case there's some pattern there (we're not seeing one, but who knows).
I really wanted to title this bug "A CIDR, a SG, and self=true walk into a bar", but managed to resist.
References
None.
The text was updated successfully, but these errors were encountered: