Undefined header errors with LambdaProxyWebRequest and destination lambdas

[sleighzy]

Description

We use the serverless-newrelic-lambda-layers plugin for our lambdas. This includes both http lambdas and destination lambdas. We have experienced an issue whereby the x-forwarded-port header is attempted to be retrieved from non-existent headers when our destination lambda is invoked. This is preventing the lambda from running.

Expected Behavior

This issue started occurring on May 16th. It appears that the node-newrelic v11.17.0 release introduced a change that is causing this problem, which is being used by the serverless-newrelic-lambda-layers.

"TypeError: Cannot read properties of undefined (reading 'x-forwarded-port')",
       "    at new LambdaProxyWebRequest (/opt/nodejs/node_modules/newrelic/lib/serverless/api-gateway.js:20:29)",
       "    at Runtime.wrappedHandler (/opt/nodejs/node_modules/newrelic/lib/serverless/aws-lambda.js:152:28)",
       "    at runInContextCb (/opt/nodejs/node_modules/newrelic/lib/shim/shim.js:1188:22)",
       "    at AsyncLocalStorage.run (node:async_hooks:338:14)",
       "    at AsyncLocalContextManager.runInContext (/opt/nodejs/node_modules/newrelic/lib/context-manager/async-local-context-manager.js:65:36)",
       "    at TransactionShim.applySegment (/opt/nodejs/node_modules/newrelic/lib/shim/shim.js:1178:25)",
       "    at Runtime.transactionWrapper (/opt/nodejs/node_modules/newrelic/lib/shim/transaction-shim.js:414:17)",
       "    at Runtime.patchHandlerSync [as handler] (/opt/nodejs/node_modules/newrelic-lambda-wrapper/index.js:140:25)",
       "    at Runtime.handleOnceNonStreaming (file:///var/runtime/index.mjs:1173:29)"

Looking at this release it appears there may be an issue in #2191 around the APIGateway check to see if this is a lambda proxy event. This previously used to check for the presence of headers, but was updated to just check the version number, e.g. returns true if the version is 1.0.. If this is true then it attempts to normalise the headers and pull the x-forwarded-port header from it, which fails as the lowerCaseHeaders are undefined.

For the lambda experiencing this issue it is an AWS destination lambda vs. http so does not have those headers. The lambda proxy check in this change is however incorrectly identifying it as needing the web proxy wrapped around it.

Steps to Reproduce

This can be replicated by using two lambdas.

• One lambda deployed as a destination lambda
• The second lambda, performing any given operation, and configured to invoke the destination lambda on success or failure

After successfully executing the normal lambda operation the destination lambda will not be invoked and the logs will contain the information as per the description re: the undefined object when trying to get the headers.

Your Environment

• ex: Browser name and version:
• ex: Node version:
• ex: Operating System and version:

Additional context

As per the server-newrelic-lambda-layers documentation we have added the custom layerArn value to pin to an earlier version of the lambda layers (68 as 69 introduces the problem) that contains the previous version of the node agent that does not contain this breaking change. This is working as expected.

newRelic:
    layerArn: arn:aws:lambda:ap-southeast-2:451483290750:layer:NewRelicNodeJS18X:68

[workato-integration]

https://new-relic.atlassian.net/browse/NR-284368

[mrickard]

@sleighzy Thank you for the thorough bug report and reproduction case. I'm sorry to hear you're having trouble with this change. We'll look into this.

[sleighzy]

Thanks for the quick turnaround on the fix. One thing to note is though the fix will prevent the error from occurring the logic in here will still wrap any lambda with version 1.0 in a LambdaProxyWebRequest, even though these may not be API gateway lambdas, e.g. a destination lambda.