API Gateway v2 integration & route: ConflictException due to concurrent modification #18018

ghost · 2021-03-10T13:50:44Z

Getting error when creating multiple aws_apigatewayv2_integration and aws_apigatewayv2_route resources

error creating API Gateway v2 integration: ConflictException: Unable to complete operation due to concurrent modification. Please try again later.

error creating API Gateway v2 route: ConflictException: Unable to complete operation due to concurrent modification. Please try again later

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v0.13.6
AWS provider 3.24.0

Affected Resource(s)

aws_apigatewayv2_integration
aws_apigatewayv2_route

Terraform Configuration Files

resource "aws_apigatewayv2_integration" "lambda" {
  for_each               = local.lambda_resources
  api_id                 = aws_apigatewayv2_api.this.id
  credentials_arn        = aws_iam_role.api.arn
  integration_type       = "AWS_PROXY"
  connection_type        = "INTERNET"
  integration_method     = "POST"
  integration_uri        = each.value
  payload_format_version = "2.0"
}

resource "aws_apigatewayv2_route" "lambda" {
  for_each  = local.lambda_resources
  api_id    = aws_apigatewayv2_api.this.id
  route_key = "${each.value.method} ${each.value.endpoint}"
  target    = "integrations/${aws_apigatewayv2_integration.lambda[each.key].id}"
}

The text was updated successfully, but these errors were encountered:

ewbankkit · 2021-03-10T16:34:29Z

@dovydas-phrasee Thanks for raising this issue.
How many integrations are being created in parallel?

ghost · 2021-03-10T17:02:35Z

@ewbankkit Around 30

ewbankkit · 2021-03-10T21:17:32Z

I get no errors creating 30 integrations in parallel for a single API:

aws_apigatewayv2_integration.test[1]: Creating...
aws_apigatewayv2_integration.test[24]: Creating...
aws_apigatewayv2_integration.test[8]: Creating...
aws_apigatewayv2_integration.test[10]: Creating...
aws_apigatewayv2_integration.test[6]: Creating...
aws_apigatewayv2_integration.test[27]: Creating...
aws_apigatewayv2_integration.test[21]: Creating...
aws_apigatewayv2_integration.test[12]: Creating...
aws_apigatewayv2_integration.test[25]: Creating...
aws_apigatewayv2_integration.test[3]: Creating...
aws_apigatewayv2_integration.test[5]: Creating...
aws_apigatewayv2_integration.test[29]: Creating...
aws_apigatewayv2_integration.test[22]: Creating...
aws_apigatewayv2_integration.test[7]: Creating...
aws_apigatewayv2_integration.test[18]: Creating...
aws_apigatewayv2_integration.test[15]: Creating...
aws_apigatewayv2_integration.test[20]: Creating...
aws_apigatewayv2_integration.test[17]: Creating...
aws_apigatewayv2_integration.test[13]: Creating...
aws_apigatewayv2_integration.test[11]: Creating...
aws_apigatewayv2_integration.test[14]: Creating...
aws_apigatewayv2_integration.test[28]: Creating...
aws_apigatewayv2_integration.test[26]: Creating...
aws_apigatewayv2_integration.test[4]: Creating...
aws_apigatewayv2_integration.test[16]: Creating...
aws_apigatewayv2_integration.test[19]: Creating...
aws_apigatewayv2_integration.test[9]: Creating...
aws_apigatewayv2_integration.test[23]: Creating...
aws_apigatewayv2_integration.test[2]: Creating...
aws_apigatewayv2_integration.test[0]: Creating...
aws_apigatewayv2_integration.test[8]: Creation complete after 0s [id=8tat3ai]
aws_apigatewayv2_integration.test[24]: Creation complete after 0s [id=piif7ak]
aws_apigatewayv2_integration.test[1]: Creation complete after 2s [id=qs76lrn]
aws_apigatewayv2_integration.test[25]: Creation complete after 1s [id=qqoarni]
aws_apigatewayv2_integration.test[27]: Creation complete after 1s [id=p3x8si4]
aws_apigatewayv2_integration.test[21]: Creation complete after 1s [id=1cp3yx9]
aws_apigatewayv2_integration.test[12]: Creation complete after 1s [id=ze1952f]
aws_apigatewayv2_integration.test[3]: Creation complete after 1s [id=ecfw9rh]
aws_apigatewayv2_integration.test[7]: Creation complete after 1s [id=9l5h8kj]
aws_apigatewayv2_integration.test[10]: Creation complete after 1s [id=6mp1qze]
aws_apigatewayv2_integration.test[6]: Creation complete after 1s [id=k7xydv3]
aws_apigatewayv2_integration.test[11]: Creation complete after 1s [id=yrd1h78]
aws_apigatewayv2_integration.test[22]: Creation complete after 1s [id=h8gkcvr]
aws_apigatewayv2_integration.test[17]: Creation complete after 1s [id=di979ym]
aws_apigatewayv2_integration.test[14]: Creation complete after 1s [id=achqvu1]
aws_apigatewayv2_integration.test[28]: Creation complete after 1s [id=zjv5kck]
aws_apigatewayv2_integration.test[5]: Creation complete after 1s [id=wijbomi]
aws_apigatewayv2_integration.test[13]: Creation complete after 1s [id=0r5tzr2]
aws_apigatewayv2_integration.test[16]: Creation complete after 1s [id=meil19d]
aws_apigatewayv2_integration.test[26]: Creation complete after 1s [id=eqzg6hm]
aws_apigatewayv2_integration.test[29]: Creation complete after 1s [id=9e8an0r]
aws_apigatewayv2_integration.test[19]: Creation complete after 1s [id=pqyq93n]
aws_apigatewayv2_integration.test[20]: Creation complete after 1s [id=fqmu575]
aws_apigatewayv2_integration.test[15]: Creation complete after 1s [id=y3vzc7f]
aws_apigatewayv2_integration.test[4]: Creation complete after 1s [id=vn9vnjm]
aws_apigatewayv2_integration.test[23]: Creation complete after 1s [id=r2amamr]
aws_apigatewayv2_integration.test[2]: Creation complete after 1s [id=zbsslb2]
aws_apigatewayv2_integration.test[0]: Creation complete after 1s [id=wu9h0dv]
aws_apigatewayv2_integration.test[9]: Creation complete after 1s [id=ek7epw2]
aws_apigatewayv2_integration.test[18]: Creation complete after 1s [id=i6gmi8g]

@dovydas-phrasee You could try lowering the parallelism value for terraform apply.

ghost · 2021-03-12T09:42:47Z

@ewbankkit tried to lower parallelism to 5, but it didn't help. And got error for creating aws_apigatewayv2_route

error creating API Gateway v2 route: ConflictException: Unable to complete operation due to concurrent modification. Please try again later

But the error is not consistent

alexjurkiewicz · 2021-03-26T04:50:40Z

There is a similar restriction with AppSync APIs (#16211). I am interested in contributing to a solution here. I believe the provider could implement a per-api-id semaphore to prevent concurrent access.

This solution seems novel in the codebase, so I would like maintainer feedback about the best approach 🙏

Burnett2k · 2021-07-20T15:41:41Z

I am experiencing a similar issue when trying to create aws_apigatewayv2_route. In my case, we have a module that allows a map of routes to be passed in, which are then created using a for_each. We are seeing about a 40% failure rate with the error below. Sometimes it works flawlessly, other times it fails. It is very intermittent which also makes it hard to reproduce consistently.

error creating API Gateway v2 route: ConflictException: Unable to complete operation due to concurrent modification. Please try again later

It appears a similar issue was fixed for apigatewayv1 by adding retry logic here:

terraform-provider-aws/aws/config.go

Lines 679 to 686 in 5e6e4df

    
           client.apigatewayconn.Handlers.Retry.PushBack(func(r *request.Request) { 
        
           	// Many operations can return an error such as: 
        
           	//   ConflictException: Unable to complete operation due to concurrent modification. Please try again later. 
        
           	// Handle them all globally for the service client. 
        
           	if tfawserr.ErrMessageContains(r.Error, apigateway.ErrCodeConflictException, "try again later") { 
        
           		r.Retryable = aws.Bool(true) 
        
           	} 
        
           })

Could we potentially add that same fix for apigatewayv2?

okdana · 2021-09-17T17:38:31Z

I'm also seeing this, with aws 3.57. I have less than a dozen routes+integrations being created from a map by a module, and i get one to three ConflictExceptions like this for the integrations at least half the time i try to apply. Adding -parallelism=1 seems to work around it 100% of the time, though i wouldn't say my testing has been exhaustive yet.

I think the v1 fix mentioned in the previous comment is related to the nearly identical issue #16439.

DanyC97 · 2021-10-29T15:15:54Z

@bflad 👋 i've been trying to get my head around under which PR the work mentioned in #16439 (comment) was done?

Looking in the changelog it looks like there is no PR

BUX FIXES

....
service/apigateway: All operations will now automatically retry on ConflictException: Unable to complete operation due to concurrent modification. Please try again later. errors.

Been hit by this issue last few days and would love to see if it can be fixed even in a simple way of extending the same code against v2

DanyC97 · 2021-11-01T18:46:33Z

okay so it looks like i've managed to find out the PR which wasn't added in the changelog followed by another commit d906813

DanyC97 · 2021-11-09T21:33:19Z

@maryelizbeth 👋 sorry to tag you directly, by a long shot is anything you or anyone in your team can help getting this issue back on the radar ?

Nowadays a lot of folks are using either v1 or v2 of the API GW and i'm very much surprised that up until now this issue hasn't been a priority.

Would appreciate if the priority gets bumped (i'm not there yet to contribute , otherwise i would have done it ... already spent 2 weeks fighting with this issue)

si3mshady · 2021-12-07T01:21:14Z

Seeing the same error when trying to create a simple serverless function with a custom authorizer. This is a bad bug.

Resource handler returned
message: "Unable to complete
operation due to concurrent
modification. Please try
again later. (Service:
ApiGateway, Status Code:
409, Request ID: fa501120-9c
3f-4efa-90c0-355bc14c00cb,
Extended Request ID: null)"
(RequestToken: 93f797df-
fe7d-5e0d-9d16-287af48122e7,
HandlerErrorCode:
AlreadyExists) The following resource(s)
failed to create:
[CustomAuthorizer]. Rollback
requested by user.

dgrushkin-techstack · 2022-01-10T11:16:01Z

Hey guys, any ETA about this fix?

hajdukd · 2022-03-31T12:38:51Z

Also an issue for private service integrations :/ Please give ETA for the fix?

andrey-clearscale · 2022-08-29T07:52:52Z

The same issue with aws_apigatewayv2_api_mapping. To deal with the issue I have to execute terraform apply with -parallelism=1 which drastically increases execution time

javigs82 · 2022-12-15T12:29:49Z

hello! any news about the ETA for fixing this?

Still happens with terraform 1.3.6 and aws provider v4.46.0

zjalicf-old · 2023-02-27T13:11:37Z

Hi guys, I am also getting
creating API Gateway v2 integration: ConflictException: Unable to complete operation due to concurrent modification. Please try again later.

module.api_gateway.aws_apigatewayv2_integration.this["PATCH /{proxy+}"]: Creating...
module.api_gateway.aws_apigatewayv2_integration.this["POST /{proxy+}"]: Creating...
module.api_gateway.aws_apigatewayv2_integration.this["DELETE /{proxy+}"]: Creating...
module.api_gateway.aws_apigatewayv2_integration.this["OPTIONS /xyz"]: Creating...
module.api_gateway.aws_apigatewayv2_integration.this["POST /xyz"]: Creating...
module.api_gateway.aws_apigatewayv2_integration.this["OPTIONS /{proxy+}"]: Creating...
module.api_gateway.aws_apigatewayv2_integration.this["GET /{proxy+}"]: Creating...
module.api_gateway.aws_apigatewayv2_integration.this["DELETE /{proxy+}"]: Creation complete after 1s [id=gyl0mnl]
module.api_gateway.aws_apigatewayv2_integration.this["POST /{proxy+}"]: Creation complete after 1s [id=5d787ga]
module.api_gateway.aws_apigatewayv2_integration.this["GET /{proxy+}"]: Creation complete after 1s [id=nbs78ys]
module.api_gateway.aws_apigatewayv2_integration.this["POST /xyz"]: Creation complete after 1s [id=iwszqfo]
module.api_gateway.aws_apigatewayv2_integration.this["OPTIONS /{proxy+}"]: Creation complete after 1s [id=ko65n7t]
module.api_gateway.aws_apigatewayv2_integration.this["OPTIONS /xyz"]: Creation complete after 1s [id=z62zgam]
╷
│ Error: creating API Gateway v2 integration: ConflictException: Unable to complete operation due to concurrent modification. Please try again later.
│ 
│   with module.api_gateway.aws_apigatewayv2_integration.this["PATCH /{proxy+}"],
│   on .terraform/modules/api_gateway/main.tf line 143, in resource "aws_apigatewayv2_integration" "this":
│  143: resource "aws_apigatewayv2_integration" "this" {
│

This should be fixed...

remil1000 · 2023-02-28T07:47:42Z

I think the related commit that was implemented for API Gateway v1 (and REST API) is this one d906813
it has been shuffled around since and the piece of code is now located here https://github.com/hashicorp/terraform-provider-aws/blob/main/internal/conns/config.go#L261

I'm not familiar with AWS Go SDK internals nor Terraform handlers but would it be possible to use something very similar for API Gateway v2 by importing github.com/aws/aws-sdk-go/service/apigatewayv2 in internal/conns/config.go and then adding the following right next to https://github.com/hashicorp/terraform-provider-aws/blob/main/internal/conns/config.go#L261

client.apigatewayv2.Handlers.Retry.PushBack(func(r *request.Request) {
	if tfawserr.ErrMessageContains(r.Error, apigatewayv2.ErrCodeConflictException, "try again later") {
		r.Retryable = aws.Bool(true)
	}
})

it seems the apigatewayv2 package in AWS Go SDK implements the exact same Exception
https://github.com/aws/aws-sdk-go/blob/main/service/apigatewayv2/errors.go#L28
vs
https://github.com/aws/aws-sdk-go/blob/main/service/apigateway/errors.go#L23

github-actions · 2023-03-16T21:33:28Z

This functionality has been released in v4.59.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

github-actions · 2023-04-16T02:13:37Z

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

ghost added the service/apigatewayv2 Issues and PRs that pertain to the apigatewayv2 service. label Mar 10, 2021

github-actions bot added the needs-triage Waiting for first response or review from a maintainer. label Mar 10, 2021

ewbankkit added waiting-response Maintainers are waiting on response from community or contributor. and removed needs-triage Waiting for first response or review from a maintainer. labels Mar 10, 2021

ghost removed waiting-response Maintainers are waiting on response from community or contributor. labels Mar 10, 2021

ghost changed the title ~~API Gateway v2 integration: ConflictException: Unable to complete operation due to concurrent modification~~ API Gateway v2 integration & route: ConflictException due to concurrent modification Mar 12, 2021

ewbankkit added the thinking label Mar 12, 2021

hc-github-team-terraform-aws removed the thinking label Oct 6, 2021

mxro mentioned this issue Jan 4, 2022

Cannot create API Gateway in Serverless API template: error creating API Gateway v2 route: ConflictException goldstack/goldstack#40

Closed

zjalicf-old mentioned this issue Mar 1, 2023

fix: apigatewayv2 error handling for ConflictException: Unable to complete operation due to concurrent modification. #29735

Merged

ewbankkit added the enhancement Requests to existing resources that expand the functionality or scope. label Mar 10, 2023

ewbankkit closed this as completed in #29735 Mar 10, 2023

github-actions bot added this to the v4.59.0 milestone Mar 10, 2023

github-actions bot locked as resolved and limited conversation to collaborators Apr 16, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

API Gateway v2 integration & route: ConflictException due to concurrent modification #18018

API Gateway v2 integration & route: ConflictException due to concurrent modification #18018

ghost commented Mar 10, 2021 •

edited by ghost

Loading

ewbankkit commented Mar 10, 2021

ghost commented Mar 10, 2021

ewbankkit commented Mar 10, 2021

ghost commented Mar 12, 2021 •

edited by ghost

Loading

alexjurkiewicz commented Mar 26, 2021

Burnett2k commented Jul 20, 2021

okdana commented Sep 17, 2021

DanyC97 commented Oct 29, 2021

DanyC97 commented Nov 1, 2021

DanyC97 commented Nov 9, 2021 •

edited

Loading

si3mshady commented Dec 7, 2021

dgrushkin-techstack commented Jan 10, 2022

hajdukd commented Mar 31, 2022

andrey-clearscale commented Aug 29, 2022 •

edited

Loading

javigs82 commented Dec 15, 2022 •

edited

Loading

zjalicf-old commented Feb 27, 2023

remil1000 commented Feb 28, 2023

github-actions bot commented Mar 16, 2023

github-actions bot commented Apr 16, 2023

API Gateway v2 integration & route: ConflictException due to concurrent modification #18018

API Gateway v2 integration & route: ConflictException due to concurrent modification #18018

Comments

ghost commented Mar 10, 2021 • edited by ghost Loading

Community Note

Terraform CLI and Terraform AWS Provider Version

Affected Resource(s)

Terraform Configuration Files

ewbankkit commented Mar 10, 2021

ghost commented Mar 10, 2021

ewbankkit commented Mar 10, 2021

ghost commented Mar 12, 2021 • edited by ghost Loading

alexjurkiewicz commented Mar 26, 2021

Burnett2k commented Jul 20, 2021

okdana commented Sep 17, 2021

DanyC97 commented Oct 29, 2021

DanyC97 commented Nov 1, 2021

DanyC97 commented Nov 9, 2021 • edited Loading

si3mshady commented Dec 7, 2021

dgrushkin-techstack commented Jan 10, 2022

hajdukd commented Mar 31, 2022

andrey-clearscale commented Aug 29, 2022 • edited Loading

javigs82 commented Dec 15, 2022 • edited Loading

zjalicf-old commented Feb 27, 2023

remil1000 commented Feb 28, 2023

github-actions bot commented Mar 16, 2023

github-actions bot commented Apr 16, 2023

ghost commented Mar 10, 2021 •

edited by ghost

Loading

ghost commented Mar 12, 2021 •

edited by ghost

Loading

DanyC97 commented Nov 9, 2021 •

edited

Loading

andrey-clearscale commented Aug 29, 2022 •

edited

Loading

javigs82 commented Dec 15, 2022 •

edited

Loading