Support adding events to existing functions

[jlhood]

Update: This top-level comment contains the finalized spec based on feedback from the community. Next step is implementation. We're looking for contributors who can help implement this feature. Thanks!

Description:

Per #349 (comment), creating a separate RFC for a SAM syntax that would allow functions provided by nested applications (AWS::Serverless::Application) to be attached to event triggers via the same Events syntax as AWS::Serverless::Function resources.

High-Level Approach: Add a new SAM resource type: AWS::Serverless::FunctionReference. It doesn't create a new AWS::Lambda::Function resource. Instead, it allows you to specify the name and ARN of an existing function and then attach event sources to it using the exact same Events syntax as AWS::Serverless::Function.

The nice thing about this approach is it's not strictly tied to nested apps. It could be used with Fn::ImportValue for example, if you want to add triggers to a function created in a separate stack in your account.

However, we will offer a simplified syntax for the nested application use case specifically.

Example 1: Nested stack with function reference

  # Example native CloudFormation nested stack that creates a function and outputs the function's name as template output variable NestedFunctionName
  NestedStack:
    Type: AWS::CloudFormation::Stack
    Properties:
      TemplateURL: <url>

  NestedFunctionExample:
    Type: AWS::Serverless::FunctionReference
    Properties:
      FunctionName: !GetAtt NestedStack.Outputs.NestedFunctionName
      Events: 
        GetRoot:
          Type: Api
          Properties:
            Path: /
            Method: GET

Example 2: Nested application with function reference

  # Example nested SAR application that creates a function and outputs the function's name as template output variable NestedFunctionName
  NestedApp:
    Type: AWS::Serverless::Application
    Properties:
      Location:
        ApplicationId: <app id>
        SemanticVersion: 1.0.0

  NestedFunctionExample:
    Type: AWS::Serverless::FunctionReference
    Properties:
      FunctionName: !GetAtt NestedApp.Outputs.NestedFunctionName
      Events: 
        GetRoot:
          Type: Api
          Properties:
            Path: /
            Method: GET

Example 3: More compact syntax for nested application use case

  # Example nested SAR application that creates a function and outputs the function's name as template output variable NestedFunctionName
  NestedApp:
    Type: AWS::Serverless::Application
    Properties:
      Location:
        ApplicationId: <app id>
        SemanticVersion: 1.0.0
      Events:
        NestedFunctionName: # This tells SAM to use !GetAtt NestedApp.Outputs.NestedFunctionName whenever we need the function name
          - Type: SQS
            Properties:
              Queue: !GetAtt MyQueue.Arn
              BatchSize: 10
          - Type: Api
            Properties:
              Path: /
              Method: GET

Example 4: Attaching events to function whose name is exported by a different stack

  # Assumes other stack exports function name as ExportedFunctionName
  ImportValueExample:
    Type: AWS::Serverless::FunctionReference
    Properties:
      FunctionName: !ImportValue ExportedFunctionName
      Events: 
        SQSTrigger:
          Type: SQS
          Properties:
            Queue: !GetAtt MyQueue.Arn
            BatchSize: 10

Notes/Questions:

1. Does SAM need both name and ARN specified or is one or the other sufficient? Might be event-specific... I know API events + implicit swagger require the function name and not ARN and AWS::Lambda::Permission can take name, ARN or alias ARN so it might be safer to require both? Alternatively, we could try to parse the ARN for the function name, but that's error-prone and also the ARN could be an intrinsic function dict and not the actual value itself.
   1. Decided to go with just FunctionName and construct the ARN assuming the function is in the same account.
2. Implementation needs to support Conditions.

[sgates]

I believe this would solve the major issue we're currently having, that is, sharing a single API Gateway across multiple AWS::Serverless::Application's. This would allow us to define groups of lambdas and endpoints in a modular way (in a sub stack) and then bring them all together in a unified API front end. This would be a huge win for those experiencing the pain of 200 resource limit, max policy size limits, or juggling multiple gateways for a "single" API.

[mikebroberts]

Definitely useful. I think this is a good first pass. Vague things on my mind:

• Try to allow a simple model if possible where only one of Name and ARN is necessary
• Do versions / aliases play into this more than just ARNs? E.g. would traffic shifting impact "upstream" function references?
• How would permissions work, e.g. if target Lambda in a different account - can SAM do anything to smooth that out?

[jlhood]

Thanks for the feedback, @mikebroberts!

Try to allow a simple model if possible where only one of Name and ARN is necessary

I think we can simplify it to only require FunctionName and then construct the ARN if necessary assuming the function is owned in the same account (see below for why cross-account wouldn't be supported).

Do versions / aliases play into this more than just ARNs? E.g. would traffic shifting impact "upstream" function references?

Good question. In cases where I've seen traffic shifting used, you end up doing something like !Sub ${MyLambda}:live wherever you need to specify a function name and it all "just works." So I think you could do the same for AWS::Serverless::FunctionReference specifying the alias for the function name and SAM should do the right thing, but I might be missing something. Is there a specific use case you can think of where this could be an issue?

How would permissions work, e.g. if target Lambda in a different account - can SAM do anything to smooth that out?

This is an interesting use case. After playing around with this, I don't see a way SAM can help in the cross-account case, because when you use SAM to add an event trigger to a function, it creates an AWS::Lambda::Permission resource in the template to add permissions allowing the relevant service principal to invoke the function. Lambda doesn't allow you to add a permission to a Lambda function owned by a different account so this wouldn't deploy.

[jlhood]

We were discussing this today and while we agreed AWS::Serverless::FunctionReference is a more general solution (works with AWS::Serverless::Application, AWS::CloudFormation::Stack and Fn::ImportValue), we should try to provide an even more concise experience for the specific use case of adding triggers to functions defined within an AWS::Serverless::Application resource. We came up with this syntax:

  NestedApp:
    Type: AWS::Serverless::Application
    Properties:
      Location:
        ApplicationId: <app id>
        SemanticVersion: 1.0.0
      FunctionReferences:
        NestedFunctionName: # the key is the nested app output that should be used as the FunctionName property of the underlying FunctionReference
          MyApiTrigger:
            Type: Api
            Properties:
              Path: /
              Method: GET
          MySQSTrigger:
            Type: SQS
            Properties:
              # ...

Concerns:

1. Syntax is slightly different from function events which could be confusing.
2. FunctionName output as object key could be confusing/hard to read for users.
3. If we find out during implementation (or with future event types) that we need more than just the function name, we won't be able to easily expand this syntax since the function name is an object key instead of a property.

Wanted to put this out to the community for feedback. @sgates @mikebroberts Any strong objections or feedback?

[sgates]

I think this makes sense - In this scenario, the API Gateway still lives in the main template, and then groups of lambdas (the NestedApps) would effectively be given access to it through the FunctionReferences in the main template? If I'm understanding that correctly I think that would be fine.

Would this mean in the nested stack, the lambdas defined there don't need Event properties because those "trickle down" from the main template?

1. This is true, could the syntax not just be the the same - like add an "Events" level that contains MyApiTrigger and MySQSTrigger?
2. Would this be implicit, or would it require the user to explicitly mark these as exports in the nested stack?
3. True... not sure what the alternative would be, this seems pretty simple, anything more general might be more complicated

[jlhood]

Thanks for the quick response, @sgates! In general, this is more a developer experience/syntax discussion, not changing the proposed functionality described in the original AWS::Serverless::FunctionReference RFC. Just adding extra convenience for the nested app use case.

This is true, could the syntax not just be the the same - like add an "Events" level that contains MyApiTrigger and MySQSTrigger?

I'll play around with the syntax. You're right that we might be able to get it even closer to the Events syntax. The only caveat is that we have to map the function name somewhere, although the Events object keys could do that as long as we're willing to accept the risk of my 3rd concern.

Would this be implicit, or would it require the user to explicitly mark these as exports in the nested stack?

The nested stack template would have to include the function name as a template output, but would not have to export the output.

True... not sure what the alternative would be, this seems pretty simple, anything more general might be more complicated

Thanks, that's helpful feedback!

[jlhood]

I think this is the closest I can get to looking like the Events property of AWS::Serverless::Function:

  # Example nested SAR application that creates a function and outputs the function's name and ARN as template output variables: NestedFunctionName
  NestedApp:
    Type: AWS::Serverless::Application
    Properties:
      Location:
        ApplicationId: <app id>
        SemanticVersion: 1.0.0
      Events:
        NestedFunctionName: # This tells SAM to use !GetAtt NestedApp.Outputs.NestedFunctionName whenever we need the function name
          - Type: SQS
            Properties:
              Queue: !GetAtt MyQueue.Arn
              BatchSize: 10
          - Type: Api
            Properties:
              Path: /
              Method: GET

I had to turn it into an array of event triggers, because normally SAM uses different keys for different event triggers, but if we're using the key for the function name, we need a way to support multiple event triggers on the same function.

[sgates]

This makes sense to me, and the syntax isn't as confusing as the previous version because it explicitly calls out "Events" - more intuitive IMO. Thanks for all the effort on this!

[TeddyHandleman]

@jlhood To me this looks like it's going in the right direction. The only thing I see that may be a potential issue is integration with an OpenAPI document via the DefinitionBody parameter in an AWS::Serverless::Api resource.

For example:

ApiResource:
      Type: AWS::Serverless::Api
      Properties:
        Name: MyApi
        StageName: foo
        DefinitionBody:
          Fn::Transform:
            Name: AWS::Include
            Parameters:
              Location: ./openapi.yaml

In the OpenAPI document you might have an APIG integration like so:

 x-amazon-apigateway-integration:
        uri:
          Fn::Sub: "arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${MyFunction.Arn}/invocations"

However, maybe your solution just means a change of MyFunction.Arn -> NestedApp.Outputs.MyFunctionArn

[TeddyHandleman]

@jlhood any further updates?

[jlhood]

@TeddyHandleman You're right that if someone is managing their own swagger definition body, they would need to substitute the nested application output function in the API GW integration as you specified.

[jlhood]

Sorry for the delay on this one. Updated the top-level comment with the finalized spec based on the discussion in this issue. Really appreciate the feedback, everyone!

Removing the RFC prefix from the title. Next step is to implement this. It's not currently prioritized internally so I'll add the looking for contributors label in case someone from the community can take this on.

[jlhood]

Implementation notes:

1. Add a SamFunctionReference resource type similar to the SamFunction resource, but with a simpler definition (shouldn't have all of the properties of a Lambda function).
2. Most of the content of SamFunction's to_cloudformation() function will not be necessary for SamFunctionReference. The key logic to reuse is the _generate_event_resources() function.
3. Add the Events property to the SamApplication resource to support the more compact syntax (example 3).
4. Add lots of tests. Make sure to test for intrinsic functions, especially that Condition works correctly for AWS::Serverless::FunctionReference resources.

[Jacco]

This sounds great. I would like to work on this. I have a question about this proposal though. Since the actual function is outside in all cases outside the current template and a role has probably already been assigned to it how would are the proper permissions given to that role?

For example:
For a being able to become a function that handles events from a DDBStream the role of the external function needs to have the Managed Policy AWSLambdaDynamoDBExecutionRole in its role.

Is there a solution for this?

[keetonian]

I believe that the developer creating the function would need to create the role with the correct permissions attached. I don't see a way to add that policy to an existing role.

In the case of your DDB Stream example, the developer would need to give the lambda execution role the correct permissions, and the new AWS::Serverless::FunctionReference would just create the necessary AWS::Lambda::EventSourceMapping to allow DDB to invoke the function.

We discussed this RFC today, and decided that the AWS::Serverless::FunctionReference design is ready for implementation.

[cshenrik]

If my understanding of the proposal is correct, it would not be possible to have a parent stack that defines a shared api gateway and a nested SAR application that will directly add event triggers to the shared api gateway. The event triggers for the function would need to be added in the parent stack as FunctionReferences.

I wish that would be possible, so it would support the following use case:

A number of SAR applications are included in a parent stack to form a complete product for a client. Imagine that there are lots of clients. If a client needs functionality X, I would like to simply add the corresponding SAR application to the parent stack. This way the parent stack would be very concise. If I need to add FunctionReferences to the parent stack, it becomes less concise.

Are there others in the community that see the benefit of supporting this use case?

@jlhood, sorry for being late with this. I am aware that this is being implemented now. Better late than never, I hope.

[markoradinovic]

I decided to write my own macro function that will allow me to define shared api gateway in parent stack and then use reference to exported api gateway from separated SAM stacks. You can find my project here https://github.com/4-cube/cf-shared-apigwv2

Currently, only ApiGatewayV2 (HttpApi) is supported and there is no support to specify Auth per event (I plan to add support for this).

[pathikrit-repo-personal]

Hey just wanted to know whether this feature is out or not basically our problem is something like this-

We want all our api resource to have the same endpoint (base url if you may) as all of this one web app api routes but the problem is that we hit 200 resource limit on SAM so now we are trying to group and break each feature (lambda, layers, api events related to that feature) in each child template but as far as I see there's no provision to break a api gateway into multiple template resource or make it in parent template and import into any of these child template resources. So if you have any comments in that or have any way outs it would really help me going.

and as always appreciate your efforts.

[jeff1evesque]

Working with AWS Support Engineer who pointed me to this issue. Adding a trivial comment in this issue, so I can be updated on this issue. Ultimately, I have a parent APIG created in CloudFormation, then a CodePipeline that "deploys" a SAM template - which I was hoping to deploy a lambda function using the restApiId from the parent CloudFormation stack. However, these approaches are not compatible (SAM vs CloudFormation api ID's).

[omenking]

I want to be able to create isolate SAM applications and I want to attach these functions to an existing API Gateway.
The reason is that I have a service where I offer different functionality to different customers, but without this, I'll have to compose a unique CFN per customer, and this not very modular nor maintainable.

I'm reading through this long thread to check where this ticket is at and if its solves this case or another.

      Events:
        Post:
          Type: Api
          Properties:
            Path: /
            Method: Post
            RestApiId:
              Ref: !Ref EvalautorApiId

Maybe its better related to #349

I found these tickets via this StackOverflow post:
https://stackoverflow.com/questions/55992260/whats-the-correct-way-to-structure-a-large-api-gateway-using-aws-sam-nested-app

[omenking]

Okay I'm back and I have my own solution which I posted over here (CFN templates included):

#1335 (comment)

My Solutions: I wanted to attach services to an existing API Gateway, So instead of using implicit events where SAM relies on generating out a Swagger template, I just explicitly defined an AWS::ApiGateway::Resource, AWS::ApiGateway::Method and AWS::Lambda::Permission and in my SAM template and it worked.

I can't test via sam local start-api but I can still test lambdas via sam local start-lambda which is good enough.

I don't care about nesting or import values so this solution works well for me.

[Jacco]

I am back :-)

Redid the changes from the old PR on fresh develop checkout (it was a while ago)...

Added two examples in the integration tests. One of the integration tests uses an application from the serverlessrepo that I created for this purpose.

I am pretty sure all scenarios described in top of this issue are supported.

If we go ahead with this PR, I can add the needed documentation changes.

Let me know whats missing and how we can move forward.

[artiomnist]

Hey everyone!

Any progress on this feature? @Jacco @jlhood

We're looking to move to nested applications to avoid the resource limit and can see this feature adding massive value in terms of sharing an API Gateway and having multiple nested applications with functions reference the API Gateway!

Thank you for all your work!

[yohanrw]

@peterkulik this is a good question for the aws/sam slack channel, but since you're here, I'll try to answer with what we do at my company (I don't work for AWS).

We combine multiple lambdas behind a single api gateway, and single template (SAM). We consider this group of lambdas with the gateway to be an "API". Every API we write has its own api gateway and template.

For resources that need to be shared between lambdas within an API, they're in the same template, so they can just be used. For resources that need to be referenced by other APIs, we export the appropriate information from the API template as an output (like the ARN, bucket name, url, whatever). For resources that need to be shared between multiple APIs, these live in their own template/project, for example, we have an infrastructure CF template that builds out things like VPC, Subnets, WAF rules, etc.... and it then exports those things as outputs. Those outputs become input params in our API SAM templates.

The above issue is related to the fact that it is not currently possible to share an API gateway between templates with SAM - if you have say, 5 APIs and you want to combine them into a single gateway, there isn't a real way to do that. I contributed to this issue in that I'm interested in being able to group multiple sets of lambdas into a 'master' template (as sub templates) and share a single API gateway resource from the master template between them. Hopefully this will be added at some point, but until then, we work with multiple API gateways, one for each API (which might consist of 1-20ish lambdas)

Hope this helps - you can ping me on the official AWS slack channel if you want to talk about it!

@sgates Hello, we are hitting the same problem of the stack resource limit. However in your reply you mentioned you are working with multiple APIs. Then,

1. Your app is pointing to different API end points to grab data?
2. Cant we assign a custom domain and make that domain common for all API gateways?
3. Authentication and Authorization is important. We will be using Firebase. Multiple API gateways means multiple configurations of Authentication/Authorization ? So before accessing each API Gasteway, your user needs to login?

[dgokcin]

any update on this issue? we really want to bind functions to an existing api gateway.

[ccaum]

Is your use case to divide up endpoints sub-paths to templates maintained by different teams where a central teams owns the top level API GW definition? Or is there a different challenge you're trying to solve?

[dgokcin]

Is your use case to divide up endpoints sub-paths to templates maintained by different teams where a central teams owns the top level API GW definition? Or is there a different challenge you're trying to solve?

yes. we would like to use api gateway as a single point of enterance to our backend apps. we want that api gw to have the domain api.company.com. each team/domain will have their own templates.

[dgokcin]

Is your use case to divide up endpoints sub-paths to templates maintained by different teams where a central teams owns the top level API GW definition? Or is there a different challenge you're trying to solve?

yes. we would like to use api gateway as a single point of enterance to our backend apps. we want that api gw to have the domain api.company.com. each team/domain will have their own templates.

I do not know what makes more sense. having more api gateways per team and binding them to central api gateway or something else.

[cortexcompiler]

I agree, I think there are a number of use cases for adding integrations to existing API Gateways from separate stacks. I personally like to group my Lambdas and APIs by domain/subdomain in order to isolate changes in repositories and stacks. To keep a reasonable number of (Edge Optimized) API Gateways it makes sense to share Gateways rather than have one per subdomain. This is in addition to the stack limitations folks have previously mentioned.

We are resorting to using explicit AWS::ApiGateway::Resource and AWS::ApiGateway::Method Resources in the separate "Lambda Stack" referencing the exported Gateway and root resource from the "Gateway Stack". We used a AWS::ApiGateway::RestApi with a root level GET method and MOCK integration, along with a Stage and Deployment. You can export the RestApiId and RootResource Id. SAM does some deploy magic for you, so you do need to get comfortable with the relationship between a Deployment and a Stage. (this is not my favorite solution, but it works)

Gateway Stack snippet:

Resources:
  AwsApiGateway:
    Type: AWS::ApiGateway::RestApi
    Properties:
      Name: !Sub ${SomeGatewayName}-api-gateway

Outputs:
  AwsApiGateway:
    Description: 'API Gateway endpoint URL'
    Value: !Sub 'https://${AwsApiGateway}.execute-api.${AWS::Region}.amazonaws.com/${StageName}'
  AwsApiGatewayRestApiId:
    Description: 'ARN for the AWS API Gateway'
    Value: !Ref AwsApiGateway
    Export:
      Name: !Sub ${SomeGatewayName}-RestApiId
  AwsApiGatewayRootResourceId:
    Value: !GetAtt AwsApiGateway.RootResourceId
    Export:
      Name: !Sub ${SomeGatewayName}-RootResourceId

Lambda Stack snippet:

Resources:
  SomeFunction:
    Type: AWS::Serverless::Function

  SomeResource:
    Type: AWS::ApiGateway::Resource
    Properties:
      RestApiId: 
        Fn::ImportValue: !Sub ${SomeGatewayName}-RestApiId
      ParentId: 
        Fn::ImportValue: !Sub ${SomeGatewayName}-RootResourceId
      PathPart: 'something'

  DeleteSomethingMethod:
    Type: AWS::ApiGateway::Method
    Properties:
      RestApiId: 
        Fn::ImportValue: !Sub ${SomeGatewayName}-RestApiId
      ResourceId: !Ref SomeResource
      HttpMethod: DELETE
      AuthorizationType: NONE
      Integration:
        IntegrationHttpMethod: POST
        Type: AWS_PROXY
        Uri: !Sub
          - arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${lambdaArn}/invocations
          - lambdaArn: !GetAtt SomeFunction.Arn

[ccaum]

I'm curious if API Gateway's sub base path mapping feature helps with this scenario. With base path mapping, you can have a central API custom domain with nested stacks that define an AWS::ApiGatewayV2::ApiMapping resource and a AWS::Serverless::Api resource that AWS::Serverless::Function resources can map events to.

Here's an example with demo code: https://aws.amazon.com/blogs/compute/using-multiple-segments-in-amazon-api-gateway-base-path-mapping/

[SimonCMoore]

This is a proposed solution following an RFC process to #2734
Note a revised solution might be to use serverless::connectors with eventsources along similar lines.
Please continue to +1 interest to drive our prioritization process.

[ccaum]

In March 2021, API Gateway added base path mapping support, which may help solve this problem. With base path mapping, you can have a central API custom domain with nested stacks that define an AWS::ApiGatewayV2::ApiMapping resource and a AWS::Serverless::Api resource that AWS::Serverless::Function resources can map events to.

Here's an example with demo code: https://aws.amazon.com/blogs/compute/using-multiple-segments-in-amazon-api-gateway-base-path-mapping/

Would this meet your needs?

[Ragavendhran-N]

This feature would help a lot
As referring the API resource from another template would solve the complex process of mapping via open API

[MateoAV]

Any news on this? It would be much appreciated to have an official workaround using mappings or support importing the RestApiId from another stack.

[Ragavendhran-N]

Try the API gateway swagger import mechanism, as a separate stage as part of jenkin build it's a workaround but helpful

[MateoAV]

Thanks for the advice, I would follow along with the workarounds proposed. Still, it would be nice to have official support for this from AWS.