custom_resources: Provider Lambda function is missing lambda:GetFunctionConfiguration

[erwaxler]

Describe the bug

The Landing Zone Accelerator solution leverages the custom_resources module to create service-linked roles via CDK custom resources. When this custom resource Lambda function is invoked several times in succession, users intermittently receive the following error:

Received response status [FAILED] from custom resource. Message returned: AccessDeniedException: Resource is not in the state functionActive

We believe this is the result of queuing incoming requests and the role attached to the cdk.custom_resources.Provider function is missing the permission: lambda:GetFunctionConfiguration

Expected Behavior

Custom resource provider implements appropriate permissions and retries to execute successfully when invoked several times in succession.

Current Behavior

Transient failures:

Received response status [FAILED] from custom resource. Message returned: AccessDeniedException: Resource is not in the state functionActive

Reproduction Steps

Deploy v1.4.3 of the Landing Zone Accelerator on AWS.

For a smaller sample that can be extracted without deploying the entire LZA solution, you may use this custom resource construct that is used by LZA to create the service-linked roles:

https://github.com/awslabs/landing-zone-accelerator-on-aws/blob/1614a01824c5a43f97fadfb8ec0c3627a0f343dd/source/packages/%40aws-accelerator/constructs/lib/aws-iam/service-linked-role.ts#L87

Possible Solution

Add lambda:GetFunctionConfiguration permission to the provider Lambda function's IAM role.

Additional Information/Context

No response

CDK CLI Version

2.79

Framework Version

No response

Node.js Version

16.20.1

OS

Amazon Linux

Language

Typescript

Language Version

No response

Other information

No response

[pahud]

this is probably related to #24358

The custom resource essentially check the functionActive state before each invocation:

aws-cdk/packages/aws-cdk-lib/custom-resources/lib/provider-framework/runtime/outbound.ts

Lines 61 to 80 in 47df704

	/**
	* The status of the Lambda function is checked every second for up to 300 seconds.
	* Exits the loop on 'Active' state and throws an error on 'Inactive' or 'Failed'.
	*
	* And now we wait.
	*
	* Use functionActive instead of functionActiveV2, since functionActiveV2 is only
	* available on SDK 2.1080.0 and up, Lambda installs 2.1055.0 by default,
	* and we use the SDK version that Lambda includes by default.
	*/
	await waitUntilFunctionActive({
	client: lambda,
	maxWaitTime: 60,
	}, {
	FunctionName: req.FunctionName,
	});
	return await lambda.invoke(req);
	}
	}

But it should work as expected.

Which region did you deploy?

Instead of running the LZA, are you able to provide a smallest code snippet that reproduces this issue?

[erwaxler]

@pahud Agreed on it likely being related to #24358, I'll work on a smaller snippet to reproduce the error. The error has been seen in at least us-east-1 and ap-southeast-2, but we've heard reports from 5+ customers so I believe the error to be region-agnostic. I'll work on a smaller snippet to reproduce the behavior more predictably.

[pahud]

@erwaxler

Are you able to see how many custom resources of this will be created in your LZA deployment?

https://github.com/awslabs/landing-zone-accelerator-on-aws/blob/1614a01824c5a43f97fadfb8ec0c3627a0f343dd/source/packages/%40aws-accelerator/constructs/lib/aws-iam/service-linked-role.ts#L87

[erwaxler]

@pahud Still working on a smaller reproducible snippet. LZA creates 6 individual custom resources to create the service-linked roles.