multiValueHeaders is an unmitigated disaster

[laverdet]

This isn't a bug in this project per se but something I thought I should bring to your attention.

Setting more than one header of the same name in a Lambda response is very difficult to do reliably. Here is the current state of affairs:

• APIGateway: Does not support multiValueHeaders in payload format "2.0", only in "1.0"
• Elastic Load Balancer: Only supports multiValueHeaders if you explicitly opt-in to it with lambda.multi_value_headers.enabled. Furthermore it reverses the header order which is devastating for headers like Set-Cookie
• Lambda Function URL: Does not support multiValueHeaders at all

So without metadata on the deployment and routing of the Lambda function this response is inherently unsafe. I'm honestly bewildered by how wrong the AWS team has consistently botched this 26 year old protocol.

I found this idea on StackOverflow which is honestly a brilliant workaround. You can toggle the casing in the returned headers in order to reliable send multiple headers:
https://stackoverflow.com/questions/66284664/multiple-set-cookie-headers-ignored-by-api-gateway-in-combination-with-lambda-in#answer-66317294

I'm going to further test this solution to see if it maintains the order of the headers and will report back.

Edit: It's bad:

• APIGateway Payload 1.0: Returns only the first header
• APIGateway Payload 2.0: Returns only the last header
• Elastic Load Balancer: Returns headers of each case, order still reversed. lambda.multi_value_headers.enabled must be false or headers is ignored entirely.
• Lambda URL: Returns the last header

There is a cookies response property which is supported by APIGateway payloads 2.0, and Lambda URL. Unsupported by APIGateway payload 1.0 & Elastic Load Balancer.

Based on this reverse engineering I think that Lambda URL is using APIGateway on the backend with payload format 2.0, though I can't find this documented anywhere.

I put together a CloudFormation stack to demonstrate the issue in AWS's services:
execute.sh

#!/bin/bash
set -e

STACK_NAME=lambda-header-bug
VPC_ID=$(aws ec2 describe-vpcs | jq -r '.Vpcs[] | select(.IsDefault).VpcId')
SUBNET_IDS=$(aws ec2 describe-subnets | jq -r '[ .Subnets[] | select(.VpcId == "'"$VPC_ID"'").SubnetId ] | join(",")')
aws cloudformation deploy \
    --template-file Stack.yaml \
    --stack-name "$STACK_NAME" \
    --capabilities CAPABILITY_IAM \
    --parameter-overrides \
        VpcId="$VPC_ID" \
        SubnetIds="$SUBNET_IDS"
echo

OUTPUT=$(aws cloudformation describe-stacks --stack-name "$STACK_NAME" | jq '.Stacks[0].Outputs | map({ key: .OutputKey, value: .OutputValue }) | from_entries')
API_GATEWAY_URL=$(echo "$OUTPUT" | jq -r .ApiGatewayURL)
FUNCTION_URL=$(echo "$OUTPUT" | jq -r .FunctionURL)
LOAD_BALANCER_URL=$(echo "$OUTPUT" | jq -r .LoadBalancerURL)

echo "$API_GATEWAY_URL"
curl -vv "$API_GATEWAY_URL" 2>&1 | grep -i set-cookie || true
echo

echo "$FUNCTION_URL"
curl -vv "$FUNCTION_URL" 2>&1 | grep -i set-cookie || true
echo

echo "$LOAD_BALANCER_URL"
curl -vv "$LOAD_BALANCER_URL" 2>&1 | grep -i set-cookie || true

echo
echo To delete the stack, please run: aws cloudformation delete-stack --stack-name "$STACK_NAME"

Stack.yaml

AWSTemplateFormatVersion: 2010-09-09
Description: Lambda reversed header issue

Parameters:
  VpcId:
    Type: String
  SubnetIds:
    Type: CommaDelimitedList

Outputs:
  ApiGatewayURL:
    Value: !Sub ${Api.ApiEndpoint}/${Stage}
  FunctionURL:
    Value: !GetAtt FunctionURL.FunctionUrl
  LoadBalancerURL:
    Value: !Sub http://${LoadBalancer.DNSName}

Resources:
  # Function definition
  Function:
    Type: AWS::Lambda::Function
    Properties:
      Handler: index.handler
      MemorySize: 512
      Role: !GetAtt Role.Arn
      Runtime: nodejs16.x
      Timeout: 30
      Code:
        ZipFile:
          |
            exports.handler = async (event) => ({
              statusCode: 200,
              body: JSON.stringify(event),
              isBase64Encoded: false,
              cookies: [
                'a=1',
                'a=2',
              ],
              multiValueHeaders: {
               'x-set-cookie': [
                 'a=1',
                 'a=2'
               ],
              },
              'content-type': [ 'text/plain' ],
            });

  Role:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Action: sts:AssumeRole
            Effect: Allow
            Principal:
              Service: lambda.amazonaws.com

  LogGroup:
    Type: AWS::Logs::LogGroup
    DeletionPolicy: Delete
    UpdateReplacePolicy: Delete
    Properties:
      LogGroupName: !Sub /aws/lambda/${Function}
      RetentionInDays: 1

  LogPolicy:
    Type: AWS::IAM::Policy
    Properties:
      PolicyName: AllowCloudWatch
      PolicyDocument:
        Version: 2012-10-17
        Statement:
        - Effect: Allow
          Action:
            - logs:CreateLogStream
            - logs:PutLogEvents
          Resource: !GetAtt LogGroup.Arn
      Roles:
        - !Ref Role

  # Public function URL
  FunctionURL:
    Type: AWS::Lambda::Url
    Properties:
      AuthType: NONE
      TargetFunctionArn: !GetAtt Function.Arn

  PermissionURL:
    Type: AWS::Lambda::Permission
    Properties:
      Action: lambda:InvokeFunctionUrl
      FunctionName: !GetAtt Function.Arn
      FunctionUrlAuthType: NONE
      Principal: "*"

  # Load Balancer
  LoadBalancer:
    Type: AWS::ElasticLoadBalancingV2::LoadBalancer
    Properties:
      SecurityGroups:
        - !Ref SecurityGroup
      Subnets: !Ref SubnetIds

  SecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: !Ref AWS::StackName
      VpcId: !Ref VpcId
      SecurityGroupIngress:
        - CidrIp: 0.0.0.0/0
          FromPort: 80
          IpProtocol: tcp
          ToPort: 80
        - CidrIpv6: ::/0
          FromPort: 80
          IpProtocol: tcp
          ToPort: 80

  Listener:
    Type: AWS::ElasticLoadBalancingV2::Listener
    Properties:
      LoadBalancerArn: !Ref LoadBalancer
      Port: 80
      Protocol: HTTP
      DefaultActions:
        - Type: fixed-response
          FixedResponseConfig:
            StatusCode: 500

  # Load balancer target
  PermissionInvoke:
    Type: AWS::Lambda::Permission
    Properties:
      Action: lambda:InvokeFunction
      FunctionName: !GetAtt Function.Arn
      Principal: elasticloadbalancing.amazonaws.com

  TargetGroup:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
      TargetType: lambda
      Targets:
        - Id: !GetAtt Function.Arn
      TargetGroupAttributes:
        - Key: lambda.multi_value_headers.enabled
          Value: true

  ListenerRule:
    Type: AWS::ElasticLoadBalancingV2::ListenerRule
    Properties:
      ListenerArn: !Ref Listener
      Priority: 1
      Actions:
        - Type: forward
          ForwardConfig:
            TargetGroups:
              - TargetGroupArn: !Ref TargetGroup
      Conditions:
        - Field: http-request-method
          HttpRequestMethodConfig:
            Values:
              - GET
              - HEAD
              - OPTIONS

  # APIGateway
  Api:
    Type: AWS::ApiGatewayV2::Api
    Properties:
      Name: !Ref AWS::StackName
      ProtocolType: HTTP

  Integration:
    Type: AWS::ApiGatewayV2::Integration
    Properties:
      ApiId: !Ref Api
      CredentialsArn: !GetAtt ApiRole.Arn
      IntegrationType: AWS_PROXY
      IntegrationUri: !Sub arn:${AWS::Partition}:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${Function.Arn}/invocations
      # Payload format 1.0 supports `multiValueHeaders`, but not `cookies`
      # Payload format 2.0 does not support `multiValueHeaders`, but supports `cookies`
      PayloadFormatVersion: "2.0"

  Route:
    Type: AWS::ApiGatewayV2::Route
    Properties:
      ApiId: !Ref Api
      RouteKey: $default
      Target: !Sub integrations/${Integration}

  Stage:
    Type: AWS::ApiGatewayV2::Stage
    Properties:
      ApiId: !Ref Api
      AutoDeploy: true
      StageName: latest

  ApiRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Action: sts:AssumeRole
            Effect: Allow
            Principal:
              Service: apigateway.amazonaws.com
      Policies:
        - PolicyName: ApiGateway
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Action: lambda:InvokeFunction
                Effect: Allow
                Resource: !Sub ${Function.Arn}