"Failed to get existing workspaces" #791

tra0x · 2019-09-26T18:57:44Z

Running Atlantis on Kubernetes. TF version 0.12.9. We're using explicit AWS credentials in ~/.aws/credentials instead of an AWS instance role. Using the Kubernetes statefulset from here.

When we run atlantis plan on a PR we get the following error

running "/usr/local/bin/terraform init -input=false -no-color -upgrade" in "/atlantis/repos/AdaSupport/infrastructure/25/default/terraform/pre-production/_global": exit status 1

Initializing the backend...

Error: Failed to get existing workspaces: AccessDenied: Access Denied
	status code: 403, request id: 8810C4AC95ED2473, host id: 7hcdfIaos79TlMEYqsGjcJEBrmkAu6xhzu1P2JJmAHOHaS8pgJ2o30haf9364UgzQwVFCdqWy1k=

When we cd to /atlantis/repos/AdaSupport/infrastructure/25/default/terraform/pre-production/_global and run /usr/local/bin/terraform init -input=false -no-color -upgrade on the pod itself we see no errors.

bash-5.0# /usr/local/bin/terraform init -input=false -no-color -upgrade

Initializing the backend...

Initializing provider plugins...
- Checking for available provider plugins...
- Downloading plugin for provider "aws" (hashicorp/aws) 2.29.0...

Terraform has been successfully initialized!

Any idea what could be causing this?

The text was updated successfully, but these errors were encountered:

tra0x · 2019-09-26T19:22:29Z

cc: @lkysow

lkysow · 2019-09-26T20:36:06Z

Hmm, can you check that when you exec in you're running as the atlantis user?

noqcks · 2019-09-26T21:04:27Z

@tomesco's coworker here.

Note that we also have EC2 roles applied to the AWS nodes, but we are attempting to use ~/.aws/credentials instead since we're running Kubernetes on AWS and have no method to maintain EC2 roles at the pod level just yet.

We found another issue we think could be related. hashicorp/aws-sdk-go-base#7

noqcks · 2019-09-26T21:13:19Z

TF_LOG trace

running "/usr/local/bin/terraform init -input=false -no-color -upgrade" in "/atlantis/repos/AdaSupport/infrastructure/25/default/terraform/pre-production/_global": exit status 1
2019/09/26 19:53:49 [INFO] Terraform version: 0.12.7  
2019/09/26 19:53:49 [INFO] Go runtime version: go1.12.9
2019/09/26 19:53:49 [INFO] CLI args: []string{"/usr/local/bin/tf/versions/0.12.7/terraform", "init", "-input=false", "-no-color", "-upgrade"}
2019/09/26 19:53:49 [DEBUG] Attempting to open CLI config file: /home/atlantis/.terraformrc
2019/09/26 19:53:49 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2019/09/26 19:53:49 [INFO] CLI command args: []string{"init", "-input=false", "-no-color", "-upgrade"}

Initializing the backend...
2019/09/26 19:53:49 [TRACE] Meta.Backend: built configuration for "s3" backend with hash value 2328449992
2019/09/26 19:53:49 [TRACE] Meta.Backend: backend has not previously been initialized in this working directory
2019/09/26 19:53:49 [DEBUG] New state was assigned lineage "REDACTED"
2019/09/26 19:53:49 [TRACE] Meta.Backend: moving from default local state only to "s3" backend
2019/09/26 19:53:49 [INFO] Setting AWS metadata API timeout to 100ms
2019/09/26 19:53:49 [INFO] AWS Auth using Profile: "REDACTED"
2019/09/26 19:53:49 [DEBUG] Trying to get account information via sts:GetCallerIdentity
2019/09/26 19:53:49 [DEBUG] [aws-sdk-go] DEBUG: Request ec2metadata/GetMetadata Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET /latest/meta-data/iam/security-credentials/ HTTP/1.1
Host: 169.254.169.254
User-Agent: aws-sdk-go/1.22.0 (go1.12.9; linux; amd64)
Accept-Encoding: gzip


-----------------------------------------------------
2019/09/26 19:53:49 [DEBUG] [aws-sdk-go] DEBUG: Response ec2metadata/GetMetadata Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.0 200 OK
Content-Length: 44
Accept-Ranges: bytes
Connection: close
Content-Type: text/plain
Date: Thu, 26 Sep 2019 19:53:49 GMT
Etag: "2844914381"
Last-Modified: Thu, 26 Sep 2019 19:52:46 GMT
Server: EC2ws


-----------------------------------------------------
2019/09/26 19:53:49 [DEBUG] [aws-sdk-go] NODE_ROLE_NAME
2019/09/26 19:53:49 [DEBUG] [aws-sdk-go] DEBUG: Request ec2metadata/GetMetadata Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET /latest/meta-data/iam/security-credentials/NODE_ROLE_NAME
HTTP/1.1
Host: 169.254.169.254
User-Agent: aws-sdk-go/1.22.0 (go1.12.9; linux; amd64)
Accept-Encoding: gzip


-----------------------------------------------------
2019/09/26 19:53:49 [DEBUG] [aws-sdk-go] DEBUG: Response ec2metadata/GetMetadata Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.0 200 OK
Content-Length: 1286
Accept-Ranges: bytes
Connection: close
Content-Type: text/plain
Date: Thu, 26 Sep 2019 19:53:49 GMT
Etag: "3499774215"
Last-Modified: Thu, 26 Sep 2019 19:52:46 GMT
Server: EC2ws


-----------------------------------------------------
2019/09/26 19:53:49 [DEBUG] [aws-sdk-go] {
  "Code" : "Success",
  "LastUpdated" : "2019-09-26T19:52:46Z",
  "Type" : "AWS-HMAC",
  "AccessKeyId" : "REDACTED",
  "SecretAccessKey" : "REDACTED",
  "Token" : "REDACTED",
  "Expiration" : "2019-09-27T02:27:46Z"
}
2019/09/26 19:53:49 [DEBUG] [aws-sdk-go] DEBUG: Request sts/GetCallerIdentity Details:
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: sts.amazonaws.com
User-Agent: aws-sdk-go/1.22.0 (go1.12.9; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.12.7
Content-Length: 43
Authorization: AWS4-HMAC-SHA256 Credential=REDACTED/REDACTED/us-east-1/sts/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date;x-amz-security-token, Signature=REDACTED
Content-Type: application/x-www-form-urlencoded; charset=utf-8
X-Amz-Date: REDACTED
X-Amz-Security-Token: REDACTED
Accept-Encoding: gzip

Action=GetCallerIdentity&Version=2011-06-15
-----------------------------------------------------
2019/09/26 19:53:50 [DEBUG] [aws-sdk-go] DEBUG: Response sts/GetCallerIdentity Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Connection: close
Content-Length: 489
Content-Type: text/xml
Date: Thu, 26 Sep 2019 19:53:50 GMT
X-Amzn-Requestid: REDACTED


-----------------------------------------------------
2019/09/26 19:53:50 [DEBUG] [aws-sdk-go] <GetCallerIdentityResponse xmlns="https://sts.amazonaws.com/doc/2011-06-15/">
  <GetCallerIdentityResult>
    <Arn>arn:aws:sts::REDACTED:assumed-role/INSTANCE_ROLE_NAME</Arn>
    <UserId>REDACTED</UserId>
    <Account>REDACTED</Account>
  </GetCallerIdentityResult>
  <ResponseMetadata>
    <RequestId>REDACTED</RequestId>
  </ResponseMetadata>
</GetCallerIdentityResponse>
2019/09/26 19:53:50 [DEBUG] checking for provider in "."
2019/09/26 19:53:50 [DEBUG] checking for provider in "/usr/local/bin/tf/versions/0.12.7"
2019/09/26 19:53:50 [DEBUG] checking for provisioner in "."
2019/09/26 19:53:50 [DEBUG] checking for provisioner in "/usr/local/bin/tf/versions/0.12.7"
2019/09/26 19:53:50 [INFO] Failed to read plugin lock file .terraform/plugins/linux_amd64/lock.json: open .terraform/plugins/linux_amd64/lock.json: no such file or directory
2019/09/26 19:53:50 [TRACE] backend/local: state manager for workspace "default" will:
 - read initial snapshot from terraform.tfstate
 - write new snapshots to terraform.tfstate
 - create any backup at terraform.tfstate.backup
2019/09/26 19:53:50 [TRACE] statemgr.Filesystem: reading initial snapshot from terraform.tfstate
2019/09/26 19:53:50 [TRACE] statemgr.Filesystem: snapshot file has nil snapshot, but that's okay
2019/09/26 19:53:50 [TRACE] statemgr.Filesystem: read nil snapshot
2019/09/26 19:53:50 [TRACE] Meta.Backend: ignoring local "default" workspace because its state is empty
2019/09/26 19:53:50 [DEBUG] New state was assigned lineage "REDACTED"
2019/09/26 19:53:50 [TRACE] Preserving existing state lineage "REDACTED"

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.
2019/09/26 19:53:50 [DEBUG] [aws-sdk-go] DEBUG: Request s3/ListObjects Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET /?prefix=env%3A%2F HTTP/1.1
Host: REDACTED
User-Agent: aws-sdk-go/1.22.0 (go1.12.9; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.12.7
Authorization: AWS4-HMAC-SHA256 Credential=REDACTED/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=REDACTED
X-Amz-Content-Sha256: REDACTED
X-Amz-Date: REDACTED
X-Amz-Security-Token: REDACTED
Accept-Encoding: gzip


-----------------------------------------------------
2019/09/26 19:53:51 [DEBUG] [aws-sdk-go] DEBUG: Response s3/ListObjects Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 403 Forbidden
Connection: close
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Thu, 26 Sep 2019 19:53:50 GMT
Server: AmazonS3
X-Amz-Bucket-Region: us-east-1
X-Amz-Id-2: REDACTED
X-Amz-Request-Id: REDACTED


-----------------------------------------------------
2019/09/26 19:53:51 [DEBUG] [aws-sdk-go] <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>REDACTED</RequestId><HostId>REDACTED</HostId></Error>
2019/09/26 19:53:51 [DEBUG] [aws-sdk-go] DEBUG: Validate Response s3/ListObjects failed, attempt 0/5, error AccessDenied: Access Denied
	status code: 403, request id: REDACTED, host id: REDACTED

Error: Failed to get existing workspaces: AccessDenied: Access Denied
	status code: 403, request id: REDACTED, host id: REDACTED

tra0x · 2019-09-27T14:17:24Z

Hmm, can you check that when you exec in you're running as the atlantis user?

Resolved, thank you @lkysow ! Seems when we mounted credentials to {$HOME}/.aws it was as root , which was a different location than atlantis' {$HOME}.

lkysow · 2019-09-27T14:35:41Z

Thanks for the follow-up Thomas. Is this a bug with the helm chart?

tra0x · 2019-09-27T15:26:47Z

Can't comment on the helm chart as we manually configured and deployed manifest.

lkysow · 2019-09-27T20:19:25Z

Ahh I see I missed your comment you were using the raw statefulset.

tra0x closed this as completed Sep 27, 2019

lkysow added the question Further information is requested label Sep 27, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

"Failed to get existing workspaces" #791

"Failed to get existing workspaces" #791

tra0x commented Sep 26, 2019 •

edited

Loading

tra0x commented Sep 26, 2019

lkysow commented Sep 26, 2019

noqcks commented Sep 26, 2019

noqcks commented Sep 26, 2019

tra0x commented Sep 27, 2019

lkysow commented Sep 27, 2019

tra0x commented Sep 27, 2019

lkysow commented Sep 27, 2019

"Failed to get existing workspaces" #791

"Failed to get existing workspaces" #791

Comments

tra0x commented Sep 26, 2019 • edited Loading

tra0x commented Sep 26, 2019

lkysow commented Sep 26, 2019

noqcks commented Sep 26, 2019

noqcks commented Sep 26, 2019

tra0x commented Sep 27, 2019

lkysow commented Sep 27, 2019

tra0x commented Sep 27, 2019

lkysow commented Sep 27, 2019

tra0x commented Sep 26, 2019 •

edited

Loading