Error with agent version 9.12.1 and aws-sdk version 3.266.1

[atodd]

Description

The latest release of this agent is causing issues with aws-sdk:

[nuxt] [request error] [unhandled] [500] The request signature we calculated does not match the signature you provided. Check your key and signing method.
  at throwDefaultError (./.output/server/node_modules/@aws-sdk/smithy-client/dist-cjs/default-error-handler.js:8:22)
  at deserializeAws_restXmlGetObjectCommandError (./.output/server/node_modules/@aws-sdk/client-s3/dist-cjs/protocols/Aws_restXml.js:4356:51)
  at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
  at async ./.output/server/node_modules/@aws-sdk/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24
  at async wrappedMiddleware (/usr/local/lib/node_modules/newrelic/node_modules/@newrelic/aws-sdk/lib/v3/smithy-client.js:82:22)
  at async ./.output/server/node_modules/@aws-sdk/middleware-signing/dist-cjs/middleware.js:14:20
  at async ./.output/server/node_modules/@aws-sdk/middleware-retry/dist-cjs/retryMiddleware.js:27:46
  at async ./.output/server/node_modules/@aws-sdk/middleware-flexible-checksums/dist-cjs/flexibleChecksumsMiddleware.js:58:20
  at async wrappedHeaderMw (/usr/local/lib/node_modules/newrelic/node_modules/@newrelic/aws-sdk/lib/v3/smithy-client.js:60:12)
  at async ./.output/server/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:5:22

Expected Behavior

Troubleshooting or NR Diag results

Steps to Reproduce

Your Environment

Node version: 18.12.1
New Relic version: 9.12.1
AWS SDK version: 9.266.1

Additional context

[workato-integration]

https://issues.newrelic.com/browse/NEWRELIC-7612

[bizob2828]

@atodd sorry for the troubles. Do you mind posting a reproduction case please?

[ahmednaser94]

Our production code broke with same error and after debugging for 2 days when we removed new relic key from env vars everything worked fine

[ahmednaser94]

@bizob2828 the steps to reproduce in nodejs
when we were uploading files (images or videos) it gave this error and it gone away when we removed newrelic key from env vars after trying dozens of solutions to find the bug but after all it was due to newrelic

[bizob2828]

@ahmednaser94 and @atodd I was able to triage and provide a fix. This will go out next week as 9.13.0. In the meantime you can pin newrelic to 9.11.0. If you're using 3.290.0+ you will need to pin [email protected] but install @newrelic/[email protected] separately.

[atodd]

@bizob2828 Thank you for the quick response and fix!

[bizob2828]

@atodd this has been released in 9.13.0 of the agent. Please upgrade and let me know how it goes

[marcoswca]

I'm using the serverless-newrelic-lambda-layers and I'm facing the same problem when I'm trying to get the pre-signed URL to download some file

And this is happening because of this host;x-new-relic-disable-dt

Is this related to this thread?

[mrickard]

@marcoswca The issue you're having is related to this thread. What's the layer ARN your function's using? If it's less than the latest, you can update that in the AWS Lambda console. (Latest layer versions can be found here: https://layers.newrelic-external.com/)

[marcoswca]

@mrickard This is happening with the latest layer (version 19). I updated to use version 18 and it's working now. Thank you!

[mrickard]

@marcoswca That's interesting--it's the opposite of what I'd expect. What's the full ARN you're using?

Node 18 version 18 contains 9.12.1, which we're aware has this conflict with AWS SDK.
Node 18 layer version 19 contains 9.13.0, which shipped a fix for that conflict.
We've also just published a new Node layer version in all regions, which would be version 20 for Node 18. That contains some new features as well as the fix from the previous version.

[pawanakhilMM]

Hey @bizob2828 @mrickard , currently facing the same issue: SignatureDoesNotMatch
The request signature we calculated does not match the signature you provided. Check your key and signing method.

Currently we are using latest v3 aws-sdk @ 3.299.0 and newrelic @ 9.14.1 . And node version 18.12.1

As you can see in the screenshot it's happening because of this value.

When we disable newrelic, preSignedURL is working fine with value of x-Amz-SignedHeaders as host

Is this related to this issue ?

[jmartin4563]

@pawanakhilMM @marcoswca would either of you be able to provide a repro case or code snippet so we're able to investigate the issues you're having? Particularly interested in how you're generating the presigned urls, as I believe there's a couple different ways to do that.

[pawanakhilMM]

@jmartin4563
We are using With Client and Command approach as per the documentation.

Here is the code snippet.

import { S3, GetObjectCommand, PutObjectCommand } from '@aws-sdk/client-s3';
import { getSignedUrl } from '@aws-sdk/s3-request-presigner';

async generateSignedUrl(
    operation: 'getObject' | 'putObject',
    params: {
      s3Key: string;
      bucketName: string;
      metadata?: { [key: string]: string };
    },
  ): Promise<string> {
       
      const s3Client= new S3({ region: 'us-east-1' });

      let command: GetObjectCommand | PutObjectCommand;

      switch (operation) {
        case 'getObject':
          command = new GetObjectCommand({
            Bucket: params.bucketName,
            Key: params.s3Key,
          });
          break;

        case 'putObject':
          command = new PutObjectCommand({
            Bucket: params.bucketName,
            Key: params.s3Key,
            Metadata: params.metadata,
          });
          break;
      }

      return getSignedUrl(s3Client, command, {
        expiresIn: 18000,
      });
  }

[jmartin4563]

@pawanakhilMM thank you so much! i'll start digging into this

[jmartin4563]

@pawanakhilMM I've confirmed the issue you and @marcoswca are seeing with Presigned URLs, since this is a separate issue from this current thread, I've opened up newrelic/node-newrelic-aws-sdk#176 to keep things clean/separate, and will do my investigation/fixes off that issue. Thank you again for the report, closing this issue in lieu of newrelic/node-newrelic-aws-sdk#176