AuthFailure with IAM role and signature version 4 #59

apptaro · 2017-12-28T02:33:56Z

Net::Amazon::EC2 0.33 have an auth issue with IAM role and signature version 4.

$ cat test.pl
#!/usr/bin/perl
use strict;
use Net::Amazon::EC2;
my ($signature_version) = @ARGV;
if (not defined $signature_version) {
  die "No signature version specified\n";
}
my $ec2 = Net::Amazon::EC2->new(
  region            => 'ap-northeast-1',
  signature_version => $signature_version
);
my $reservations = $ec2->describe_instances;
foreach my $reservation (@$reservations) {
  foreach my $instance ($reservation->instances_set) {
    print $instance->instance_id . "\n";
  }
}

$ chmod u+x test.pl

$ ./test.pl 2
i-0066cd49489e4faa9
i-05eaa8b03dd55d22b

$ ./test.pl 4
Amazon EC2 Errors [Request 08ecf962-c346-4f75-afc2-837eaec2f78f]:
[AuthFailure] AWS was not able to validate the provided access credentials
$

I'm not sure if this issue is region dependent.
This issue has been reported here about two years ago:
https://rt.cpan.org/Public/Bug/Display.html?id=107491

The text was updated successfully, but these errors were encountered:

jadeallenx · 2017-12-28T17:56:34Z

OK, I have a PR up which - I think - addresses this issue. You can apply it on your test system by:

Grabbing the diff curl -LO https://patch-diff.githubusercontent.com/raw/mrallen1/net-amazon-ec2/pull/60.patch
applying it using patch -p1 < 60.patch
perl Makefile.PL
make install

And re-running your test script with signature v4.

jadeallenx · 2017-12-28T17:58:45Z

Or you could git clone --branch fix_iam_v4_sigs https://github.com/mrallen1/net-amazon-ec2 and do the usual Makefile nonsense.

jadeallenx · 2017-12-28T18:05:09Z

It'd be super helpful if you could enable debugging in net-amazon-ec2 by setting debug => 1 when you initialize the ec2 object. Thanks.

Obviously, you'll want to leave out your secrets if you post the debug output. :)

apptaro · 2017-12-29T08:52:16Z

Hi, I applied the patch and the issue is gone!

jadeallenx · 2017-12-29T16:32:43Z

Great. I'll merge this PR and get a new release up on CPAN

jadeallenx · 2017-12-30T01:01:54Z

OK, I released 0.35 to CPAN today. Open a new ticket if there's more trouble.

nikolai-derzhak-distillery · 2018-05-23T20:08:24Z

Sorry for commenting on closed issue but I have the same issue with 0.36.

It is just like 25% of tries actually succeed. Any ideas how to troubleshoot it ?

It happens also with version 2 signature too.

Failure output:

/ec2-consistent-snapshot.bin --use-iam-role --no-freeze-filesystem /etc/host
name --region=us-west-1 --debug --signature-version 4 
ec2-consistent-snapshot.bin: Authenticating with IAM role
ec2-consistent-snapshot.bin: Wed May 23 20:08:12 2018: No volume ids specified; discovering volume ids
ec2-consistent-snapshot.bin: Wed May 23 20:08:12 2018: Discovering volume ids for: /etc/hostname
ec2-consistent-snapshot.bin: Wed May 23 20:08:12 2018: Determining instance id
ec2-consistent-snapshot.bin: Wed May 23 20:08:12 2018: create EC2 object
ec2-consistent-snapshot.bin: Endpoint: https://ec2.us-west-1.amazonaws.com
ec2-consistent-snapshot.bin: Wed May 23 20:08:12 2018: Fetching instance description for i-0c89d93e5ad619faf
Amazon EC2 Errors [Request c027f6ea-b2b6-47f1-be3d-31d1e4027578]:
[AuthFailure] AWS was not able to validate the provided access credentials
ec2-consistent-snapshot.bin: Wed May 23 20:08:12 2018: done

Success output:

# /ec2-consistent-snapshot.bin --use-iam-role --no-freeze-filesystem /etc/host
name --region=us-west-1 --debug --signature-version 4 
ec2-consistent-snapshot.bin: Authenticating with IAM role
ec2-consistent-snapshot.bin: Wed May 23 20:07:11 2018: No volume ids specified; discovering volume ids
ec2-consistent-snapshot.bin: Wed May 23 20:07:11 2018: Discovering volume ids for: /etc/hostname
ec2-consistent-snapshot.bin: Wed May 23 20:07:11 2018: Determining instance id
ec2-consistent-snapshot.bin: Wed May 23 20:07:11 2018: create EC2 object
ec2-consistent-snapshot.bin: Endpoint: https://ec2.us-west-1.amazonaws.com
ec2-consistent-snapshot.bin: Wed May 23 20:07:11 2018: Fetching instance description for i-0c89d93e5ad619faf
ec2-consistent-snapshot.bin: Wed May 23 20:07:11 2018: Found EBS block devices for i-0c89d93e5ad619faf: 
ec2-consistent-snapshot.bin: Wed May 23 20:07:11 2018:     vol-0cd3b94c4fbfc6da0 /dev/sda1
ec2-consistent-snapshot.bin: Wed May 23 20:07:11 2018:     vol-073e02d9776bc83e2 /dev/sdm
ec2-consistent-snapshot.bin: Wed May 23 20:07:11 2018:     vol-0b5879ae4bd7672c2 /dev/sdn

--> Working on Net::Amazon::EC2
Fetching http://www.cpan.org/authors/id/M/MA/MALLEN/Net-Amazon-EC2-0.36.tar.gz ... OK
Configuring Net-Amazon-EC2-0.36 ... OK

nikolai-derzhak-distillery · 2018-05-23T22:37:00Z

My solution is to keep trying util it managed to generate signature properly for both AWS calls :)

until trigger_snapshot; do
  sleep 3
done

jadeallenx · 2018-05-24T18:57:42Z

Huh. That it "sometimes works" indicates to me there's some kind of race condition going on in the code somewhere (either yours or mine) which messes up the v4 signing code. Could you set the debug flag on your Net::Amazon::EC2 object? It will be very spammy, but it would help isolate the failure. (Also, if you paste in the output, please make sure you remove your AWS credentials 😄)

nikolai-derzhak-distillery · 2018-05-24T21:13:17Z

It is iam role attached to instance to all creds are temporary :) but thanks .
That is actually could be part of problem it seems.

Here is failure of describeInstance API Call:

ec2-consistent-snapshot.bin: Thu May 24 21:07:52 2018: create EC2 object
ec2-consistent-snapshot.bin: Endpoint: https://ec2.us-west-1.amazonaws.com
ec2-consistent-snapshot.bin: Thu May 24 21:07:52 2018: Fetching instance description for i-0887f8a21c3ce1ccd
$VAR1 = {
          'Errors' => [
                        {
                          'Error' => {
                                       'Message' => 'AWS was not able to validate the provided access credentials',
                                       'Code' => 'AuthFailure'
                                     }
                        }
                      ],
          'RequestID' => 'a6968303-9ac8-4737-9faf-72537f8b018a'
        };


Last error was: AWS was not able to validate the provided access credentials at Net/Amazon/EC2.pm line 484.
    Net::Amazon::EC2::_parse_errors(Net::Amazon::EC2=HASH(0x559ca3d4c120), HASH(0x559ca4e13c90)) called at Net/Amazon/EC2.pm line 2017
    Net::Amazon::EC2::describe_instances(undef, "InstanceId", "i-0887f8a21c3ce1ccd") called at script/ec2-consistent-snapshot line 414
    main::ec2_instance_description("https://ec2.us-west-1.amazonaws.com") called at script/ec2-consistent-snapshot line 440
    main::discover_volume_ids("https://ec2.us-west-1.amazonaws.com") called at script/ec2-consistent-snapshot line 147
    require main called at /usr/local/share/perl5/site_perl/PAR.pm line 645
    PAR::_run_member(Archive::Zip::ZipFileMember=HASH(0x559ca17f4340), 1) called at script/main.pl line 26
    require main called at /usr/local/share/perl5/site_perl/PAR.pm line 645
    PAR::_run_member(Archive::Zip::ZipFileMember=HASH(0x559ca17f4628)) called at /usr/local/share/perl5/site_perl/PAR.pm line 432
    PAR::import("PAR") called at -e line 606
    eval {...} called at -e line 41
    __par_pl::BEGIN() called at script/ec2-consistent-snapshot line 0
    eval {...} called at script/ec2-consistent-snapshot line 0
ec2-consistent-snapshot.bin: Thu May 24 21:07:52 2018: done
CONTENT: Action=DescribeInstances&InstanceId=i-0887f8a21c3ce1ccd&Version=2014-06-15



CANONICAL REQUEST: POST
/

content-type:application/x-www-form-urlencoded
host:ec2.us-west-1.amazonaws.com
x-amz-date:20180524T210752Z

content-type;host;x-amz-date
519707a9e49b186fae6ed30297ecfd4359032677634a29dd9a01b5a899a457f5



STRING TO SIGN: AWS4-HMAC-SHA256
20180524T210752Z
20180524/us-west-1/ec2/aws4_request
4cc5c0e691e58b816b2abf58014ef1fbbab8beb636a7c955e9d433b7792a63b1



HTTP REQUEST: POST https://ec2.us-west-1.amazonaws.com
Authorization: AWS4-HMAC-SHA256 Credential=ASIAxxxxxx/20180524/us-west-1/ec2/aws4_request, SignedHeaders=content-type;host;x-amz-date, Signature=db84ca08fbf1a51eb616ab755f31313aababc3c727fc8ef9dfa6575090bc15a5
Content-Type: application/x-www-form-urlencoded
X-Amz-Date: 20180524T210752Z

Action=DescribeInstances&InstanceId=i-0887f8a21c3ce1ccd&Version=2014-06-15





ERROR CODE: AuthFailure MESSAGE: AWS was not able to validate the provided access credentials FOR REQUEST: a6968303-9ac8-4737-9faf-72537f8b018a

nikolai-derzhak-distillery · 2018-05-24T21:14:44Z

It is also can fail on CreateSnapshot action too.

nikolai-derzhak-distillery · 2018-05-24T21:15:36Z

Oh. And I use PAR/pp to build binary (well in fact it extracts all that stuff in /tmp)

markstos · 2018-08-24T19:33:34Z

I also experience this issue when using Net::Amazon::EC2 via ec2-consistent-snapshot with IAM Role authentication. Typically, the first authorization call falls and a follow-up call succeeds. I'm using v4 signatures.

I may switch to using Paws or static credentials as a workaround.

jadeallenx · 2018-08-24T19:59:15Z

That sort of sounds like the code isn't getting the token at first and then when you try a second time it works.

markdwhite · 2018-08-30T01:32:44Z

I'm also having this problem and am currently working on the idea that it relates to time sync differences between the instance and the AWS IAM auth server based on these:

aws/aws-cli#2205
aws/aws-cli#1948

It may the the wrong track and have had no success after trying a few things, but I'll offer this here in case it's helpful or if someone else can offer more informed advice.

Edit: Though as I can repeatedly run aws cli commands with no issues, and the ec2-consistent-snapshot fails about 60% of the time, this is quite possibly the wrong track.

markstos · 2018-08-30T13:18:10Z

I was in contact with AWS Premium Support about this issue and they also suspected it was a time sync issue. But I compared the server in question to others and found it was accurate within a second. Also, AWS Premium Support mentioned the difference needed to be 5 minutes for this to matter, but in the second linked issue above, someone reported the problem with only about a one minute difference.

So, time skew is certainly one cause of this issue, but I'm not convinced it's the only cause.

markdwhite · 2018-08-31T04:07:20Z

Certainly nothing I've done with time sync has made any difference to IAM auth problems (called from ec2-consistent-snapshot as I think you've noticed). I see why you've switched to static credentials and I'm thinking that need to be my interim solution too.

markdwhite · 2018-09-03T07:24:53Z

And static credentials are also failing, just based on a simple test of:

perl -de1
DB<1> use Net::Amazon::EC2; my $ec2 = Net::Amazon::EC2->new(AWSAccessKeyId => 'XXXXXXXXX', SecretAccessKey => 'XXXXXXXX', signature_version => 4, debug => 1); $ec2->describe_instances;     
(removed output)                                  
Last error was: AWS was not able to validate the provided access credentials at /usr/share/perl5/Net/Amazon/EC2.pm line 484.          
Net::Amazon::EC2::_parse_errors(Net::Amazon::EC2=HASH(0x55ebbbc2f340), HASH(0x55ebbbc32618)) called at /usr/share/perl5/Net/Amazon/EC2.pm line 2017

This suggests that it isn't solely an IAM issue, and the end result is that it will fail 2/3 or 3/4 times but work sometimes. However this test is using 0.35-1 which is all I have available on a production server.

And can this ticket be reopened?

tavisma · 2018-10-16T21:39:12Z

We are also seeing this issue across several different AWS regions, can we reopen this issue?

tavisma · 2018-10-16T23:02:45Z

We have observed this issue occurring in Net::Amazon::EC2 0.36 on ActivePerl-5.26 and ActivePerl-5.24
Oddly enough the issue doesn't manifest when using Perl v5.16.3 installed by default in Centos 7 (with Net::Amazon::EC2 force installed via CPAN)

markstos · 2018-10-17T00:10:58Z

@tavisma I gave up on getting this solved in the Perl stack, which is not supported by AWS. I believe you happen to experiencing this via ec2-consistent-snapshot. I rewrote that tool in Bash, which in turn uses the officially supported aws CLI tool.

https://github.com/RideAmigosCorp/ec2-consistent-snapshot.sh

tavisma · 2018-10-17T00:13:43Z

Thanks, i'll check it out

apptaro mentioned this issue Dec 28, 2017

role-based authentication fails in new AWS regions, including eu-central-1, ca-central-1 and others. alestic/ec2-consistent-snapshot#76

Closed

jadeallenx mentioned this issue Dec 28, 2017

Add X-Amz-Security-Token to v4 sigs if we have one #60

Merged

jadeallenx closed this as completed Dec 30, 2017

markstos mentioned this issue Aug 30, 2018

Ubuntu PPA does not support 18.04 alestic/ec2-consistent-snapshot#104

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

AuthFailure with IAM role and signature version 4 #59

AuthFailure with IAM role and signature version 4 #59

apptaro commented Dec 28, 2017

jadeallenx commented Dec 28, 2017

jadeallenx commented Dec 28, 2017

jadeallenx commented Dec 28, 2017 •

edited

Loading

apptaro commented Dec 29, 2017

jadeallenx commented Dec 29, 2017

jadeallenx commented Dec 30, 2017

nikolai-derzhak-distillery commented May 23, 2018 •

edited

Loading

nikolai-derzhak-distillery commented May 23, 2018 •

edited

Loading

jadeallenx commented May 24, 2018 •

edited

Loading

nikolai-derzhak-distillery commented May 24, 2018 •

edited

Loading

nikolai-derzhak-distillery commented May 24, 2018

nikolai-derzhak-distillery commented May 24, 2018 •

edited

Loading

markstos commented Aug 24, 2018

jadeallenx commented Aug 24, 2018

markdwhite commented Aug 30, 2018 •

edited

Loading

markstos commented Aug 30, 2018

markdwhite commented Aug 31, 2018

markdwhite commented Sep 3, 2018 •

edited

Loading

tavisma commented Oct 16, 2018

tavisma commented Oct 16, 2018

markstos commented Oct 17, 2018

tavisma commented Oct 17, 2018

AuthFailure with IAM role and signature version 4 #59

AuthFailure with IAM role and signature version 4 #59

Comments

apptaro commented Dec 28, 2017

jadeallenx commented Dec 28, 2017

jadeallenx commented Dec 28, 2017

jadeallenx commented Dec 28, 2017 • edited Loading

apptaro commented Dec 29, 2017

jadeallenx commented Dec 29, 2017

jadeallenx commented Dec 30, 2017

nikolai-derzhak-distillery commented May 23, 2018 • edited Loading

nikolai-derzhak-distillery commented May 23, 2018 • edited Loading

jadeallenx commented May 24, 2018 • edited Loading

nikolai-derzhak-distillery commented May 24, 2018 • edited Loading

nikolai-derzhak-distillery commented May 24, 2018

nikolai-derzhak-distillery commented May 24, 2018 • edited Loading

markstos commented Aug 24, 2018

jadeallenx commented Aug 24, 2018

markdwhite commented Aug 30, 2018 • edited Loading

markstos commented Aug 30, 2018

markdwhite commented Aug 31, 2018

markdwhite commented Sep 3, 2018 • edited Loading

tavisma commented Oct 16, 2018

tavisma commented Oct 16, 2018

markstos commented Oct 17, 2018

tavisma commented Oct 17, 2018

jadeallenx commented Dec 28, 2017 •

edited

Loading

nikolai-derzhak-distillery commented May 23, 2018 •

edited

Loading

nikolai-derzhak-distillery commented May 23, 2018 •

edited

Loading

jadeallenx commented May 24, 2018 •

edited

Loading

nikolai-derzhak-distillery commented May 24, 2018 •

edited

Loading

nikolai-derzhak-distillery commented May 24, 2018 •

edited

Loading

markdwhite commented Aug 30, 2018 •

edited

Loading

markdwhite commented Sep 3, 2018 •

edited

Loading