panic in aws/resource_aws_instance.go in v3.25.0

[jayalane]

• Provisioner issues

My account is getting panic in line 1665 of version v3.25.0 of aws/resource_aws_instance.go

if len(ibds["ebs"].([]map[string]interface{})) == 0 {

The backtrace is:

panic: interface conversion: interface {} is nil, not []map[string]interface {}
2021-01-25T18:57:14.067-0800 [DEBUG] plugin.terraform-provider-aws_v3.25.0_x5: 
2021-01-25T18:57:14.067-0800 [DEBUG] plugin.terraform-provider-aws_v3.25.0_x5: goroutine 11830 [running]:
2021-01-25T18:57:14.067-0800 [DEBUG] plugin.terraform-provider-aws_v3.25.0_x5: github.com/terraform-providers/terraform-provider-aws/aws.readBlockDevices(0xc0020f4600, 0xc002532000, 0xc000abc598, 0x0, 0x0)
2021-01-25T18:57:14.067-0800 [DEBUG] plugin.terraform-provider-aws_v3.25.0_x5: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-aws/aws/resource_aws_instance.go:1665 +0x78d
2021-01-25T18:57:14.067-0800 [DEBUG] plugin.terraform-provider-aws_v3.25.0_x5: github.com/terraform-providers/terraform-provider-aws/aws.resourceAwsInstanceRead(0xc0020f4600, 0x66874e0, 0xc00138ec00, 0xb1cc610, 0x749d828)
2021-01-25T18:57:14.067-0800 [DEBUG] plugin.terraform-provider-aws_v3.25.0_x5: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-aws/aws/resource_aws_instance.go:930 +0x1845
2021-01-25T18:57:14.067-0800 [DEBUG] plugin.terraform-provider-aws_v3.25.0_x5: github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).read(0xc000a28e40, 0x7db33c0, 0xc001980340, 0xc0020f4600, 0x66874e0, 0xc00138ec00, 0x0, 0x0, 0x0)
2021-01-25T18:57:14.067-0800 [DEBUG] plugin.terraform-provider-aws_v3.25.0_x5: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/[email protected]/helper/schema/resource.go:290 +0x88
2021-01-25T18:57:14.067-0800 [DEBUG] plugin.terraform-provider-aws_v3.25.0_x5: github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).RefreshWithoutUpgrade(0xc000a28e40, 0x7db33c0, 0xc001980340, 0xc00040bb90, 0x66874e0, 0xc00138ec00, 0xc000d7df60, 0x0, 0x0, 0x0)
2021-01-25T18:57:14.067-0800 [DEBUG] plugin.terraform-provider-aws_v3.25.0_x5: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/[email protected]/helper/schema/resource.go:564 +0x1cb
2021-01-25T18:57:14.067-0800 [DEBUG] plugin.terraform-provider-aws_v3.25.0_x5: github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ReadResource(0xc0012d5240, 0x7db33c0, 0xc001980340, 0xc001980380, 0xc001980340, 0x6ecfe80, 0x716afc0)
2021-01-25T18:57:14.067-0800 [DEBUG] plugin.terraform-provider-aws_v3.25.0_x5: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/[email protected]/helper/schema/grpc_provider.go:575 +0x43b
2021-01-25T18:57:14.067-0800 [DEBUG] plugin.terraform-provider-aws_v3.25.0_x5: github.com/hashicorp/terraform-plugin-go/tfprotov5/server.(*server).ReadResource(0xc0014d1f60, 0x7db33c0, 0xc001980340, 0xc00208a060, 0xc0014d1f60, 0xc001bf4750, 0xc0013ceba0)
2021-01-25T18:57:14.067-0800 [DEBUG] plugin.terraform-provider-aws_v3.25.0_x5: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/[email protected]/tfprotov5/server/server.go:298 +0x105
2021-01-25T18:57:14.067-0800 [DEBUG] plugin.terraform-provider-aws_v3.25.0_x5: github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ReadResource_Handler(0x716afc0, 0xc0014d1f60, 0x7db3480, 0xc001bf4750, 0xc00208a000, 0x0, 0x7db3480, 0xc001bf4750, 0xc00198e000, 0x917)
2021-01-25T18:57:14.067-0800 [DEBUG] plugin.terraform-provider-aws_v3.25.0_x5: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/[email protected]/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:344 +0x214
2021-01-25T18:57:14.068-0800 [DEBUG] plugin.terraform-provider-aws_v3.25.0_x5: google.golang.org/grpc.(*Server).processUnaryRPC(0xc0001fee00, 0x7dd6320, 0xc000888600, 0xc001824f00, 0xc000920510, 0xb18b330, 0x0, 0x0, 0x0)
2021-01-25T18:57:14.068-0800 [DEBUG] plugin.terraform-provider-aws_v3.25.0_x5: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/[email protected]/server.go:1194 +0x522
2021-01-25T18:57:14.068-0800 [DEBUG] plugin.terraform-provider-aws_v3.25.0_x5: google.golang.org/grpc.(*Server).handleStream(0xc0001fee00, 0x7dd6320, 0xc000888600, 0xc001824f00, 0x0)
2021-01-25T18:57:14.068-0800 [DEBUG] plugin.terraform-provider-aws_v3.25.0_x5: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/[email protected]/server.go:1517 +0xd05
2021-01-25T18:57:14.068-0800 [DEBUG] plugin.terraform-provider-aws_v3.25.0_x5: google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc001154ce0, 0xc0001fee00, 0x7dd6320, 0xc000888600, 0xc001824f00)
2021-01-25T18:57:14.068-0800 [DEBUG] plugin.terraform-provider-aws_v3.25.0_x5: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/[email protected]/server.go:859 +0xa5
2021-01-25T18:57:14.068-0800 [DEBUG] plugin.terraform-provider-aws_v3.25.0_x5: created by google.golang.org/grpc.(*Server).serveStreams.func1
2021-01-25T18:57:14.068-0800 [DEBUG] plugin.terraform-provider-aws_v3.25.0_x5: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/[email protected]/server.go:857 +0x1fd

I have no idea what's going on, but seems panic is prima facie bug. I'm testing this fix on my local provider:

terraform-providers(3:58823)$ git diff
diff --git a/aws/resource_aws_instance.go b/aws/resource_aws_instance.go
index 7e69c85f6..fdd7b24e0 100644
--- a/aws/resource_aws_instance.go
+++ b/aws/resource_aws_instance.go
@@ -1662,7 +1662,7 @@ func readBlockDevices(d *schema.ResourceData, instance *ec2.Instance, conn *ec2.
    // and #readBlockDevicesFromInstance only returns 1 reference to a block-device
    // stored in ibds["root"]
    if _, ok := d.GetOk("ebs_block_device"); ok {
-       if len(ibds["ebs"].([]map[string]interface{})) == 0 {
+       if ibds["ebs"] != nil && len(ibds["ebs"].([]map[string]interface{})) == 0 {
            ebs := make(map[string]interface{})
            for k, v := range ibds["root"].(map[string]interface{}) {
                ebs[k] = v

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v0.14.5
+ provider registry.terraform.io/hashicorp/aws v3.25.0
+ provider registry.terraform.io/hashicorp/external v2.0.0
+ provider registry.terraform.io/hashicorp/template v2.2.0

Affected Resource(s)

It is crashing during plan before it starts updating what the plan would do, walking the dag I guess. I ran it under TF_LOG=1 but the immediate logging around the panic lines don't seem related to aws_instance resource. I can't just put the whole config out here.

• aws_instance maybe

Terraform Configuration Files

I won't be able to upload all the terraform config. If I can reproduce it in a smaller set, I'll submit that. I will also note here if my proposed patch fixes the issue.

Debug Output

Panic Output

I can't find a crash.log but the panic and backtrace are above.

Expected Behavior

Successful apply.

Actual Behavior

Lot of output like:

Error:  rpc error: code = Canceled desc = context canceled
Error:  rpc error: code = Canceled desc = context canceled
Error:  rpc error: code = Canceled desc = context canceled

Steps to Reproduce

1. terraform apply

Important Factoids

Biggish? 3133 lines of tf, 532 items in 'terraform state list' and 47 ec2. Mostly just vanilla S3 and Ec2. More VPC and subnets than our normal accounts (multi-tenant SaaS implementation).

References

• #0000

[ewbankkit]

terraform-provider-aws/aws/resource_aws_instance.go

Lines 930 to 932 in 431cf30

	if err := readBlockDevices(d, instance, conn); err != nil {
	return err
	}

calling

terraform-provider-aws/aws/resource_aws_instance.go

Lines 1655 to 1673 in 431cf30

	func readBlockDevices(d *schema.ResourceData, instance *ec2.Instance, conn *ec2.EC2) error {
	ibds, err := readBlockDevicesFromInstance(d, instance, conn)
	if err != nil {
	return err
	}
	// This handles cases where the root device block is of type "EBS"
	// and #readBlockDevicesFromInstance only returns 1 reference to a block-device
	// stored in ibds["root"]
	if _, ok := d.GetOk("ebs_block_device"); ok {
	if len(ibds["ebs"].([]map[string]interface{})) == 0 {
	ebs := make(map[string]interface{})
	for k, v := range ibds["root"].(map[string]interface{}) {
	ebs[k] = v
	}
	ebs["snapshot_id"] = ibds["snapshot_id"]
	ibds["ebs"] = append(ibds["ebs"].([]map[string]interface{}), ebs)
	}
	}

ibds will be nil if there are no instance block devices:

terraform-provider-aws/aws/resource_aws_instance.go

Lines 1732 to 1747 in 431cf30

	func readBlockDevicesFromInstance(d *schema.ResourceData, instance *ec2.Instance, conn *ec2.EC2) (map[string]interface{}, error) {
	blockDevices := make(map[string]interface{})
	blockDevices["ebs"] = make([]map[string]interface{}, 0)
	blockDevices["root"] = nil
	// Ephemeral devices don't show up in BlockDeviceMappings or DescribeVolumes so we can't actually set them
	instanceBlockDevices := make(map[string]*ec2.InstanceBlockDeviceMapping)
	for _, bd := range instance.BlockDeviceMappings {
	if bd.Ebs != nil {
	instanceBlockDevices[aws.StringValue(bd.Ebs.VolumeId)] = bd
	}
	}
	if len(instanceBlockDevices) == 0 {
	return nil, nil
	}

idbs["ebs"] will be nil if the only block device is the root device:

terraform-provider-aws/aws/resource_aws_instance.go

Lines 1797 to 1805 in 431cf30

	if blockDeviceIsRoot(instanceBd, instance) {
	blockDevices["root"] = bd
	} else {
	if vol.SnapshotId != nil {
	bd["snapshot_id"] = aws.StringValue(vol.SnapshotId)
	}
	blockDevices["ebs"] = append(blockDevices["ebs"].([]map[string]interface{}), bd)
	}

[ewbankkit]

Very similar:

• terraform-provider-aws_v3.17.0: panic: interface conversion: interface {} is nil, not []map[string]interface {} #16689
• r/instance: prevent panic when instance does not have block devices #16694

[jayalane]

I did build the provider with my proposed fix (just testing for that elb list being nil). make test works and my terraform plan completes without incident.

[bflad]

Hi @jayalane 👋 Thank you for raising this and sorry you ran into trouble here. If you want to submit your proposed code patch, which looks okay, the maintainers can handle additional testing and get the fix out tomorrow. Thanks again.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.