adding staging bucket stack, arm builds, some directions (aws#3205)

* separated the merge-build and pr-build buildspecs

* aim the codebuild stack at the right buildspec

* updated the buildspecs to release and sign arm variants as well

* updated the build infrastructure to build for arm as well as amd archs

* added a note about adding more artifacts to be signed and copied

* fixing a couple of other things found in review

* added a bucket stack with a bucket policy

* missed a letter in the arm build naming

* added a missed file in the build directory structure

* updated the descriptions for all of the codebuild projects

build-infrastructure/README.md

@@ -42,13 +42,27 @@ The directory structure that is expected is as follows,
 
 ```
 buildspecs/
-|- build.yml
+|- merge-build.yml
+|- pr-build.yml
 |- signing.yml
 |- copy.yml
 ```
 
 Everything else is already set up for you.
 
+## Adding a new artifact get signed and copied
+
+There are a few changes that need to be made to add another artifact that needs to be signed and copied to the CodePipeline. They are as follows,
+
+1. The CodeBuild project that feeds into the signer has to export a variable that carries the name of the artifact to sign. Check out the `exported-variables` key within `merge-build.yml` for an example of how to set this up. You create an environment variable and any defined environment variable can be exported from a codebuild project.
+1. In `release-pipeline-stack.yml`, add the project that will feed into the signer, and add an IAM role to go with it. See the `AmdBuildCodeBuildProjectServiceRole` and `AmdBuildCodeBuildProject` keys in the file for examples. Follow established naming conventions.
+1. In `release-pipeline-stack.yml`, add an action to the appropriate stage under the `BuildAndSignCodePipeline` key. Likely, you're going to want to define a new item in the `Actions` list under one of the `Stages`. Pay close attention to define both `OutputArtifacts` and `Namespace` for your new action. See `Action` with the `Name` called `MakeAmd` for an example.
+1. In `release-pipeline-stack.yml`, add a new entry in the `InputArtifacts` list for the `Action` with `Name` called `GPG`. This name should correspond to the `OutputArtifacts` name that you defined above.
+1. In `release-pipeline-stack.yml`, the `Action` with `Name` called `GPG` has a key called `EnvironmentVariables` with a JSON string as value. Be careful not to unstringify the JSON, CodePipeline through CloudFormation expects this key to have stringified JSON, and nothing else. Add an entry to this declaring the new artifact that you want to sign. Your exported variable from the CodeBuild buildspec and the `Namespace` that you declared comes into play here. You can find the name of the artifact in the value specified by `#{<namespace_name>.<exported_variable_name>}`. Create a new JSON object with the `name` key containing the environment variable name you'd like to be set, the `value` key containing something akin to `#{something.something}`, and the `type` key set to `PLAINTEXT`.
+1. Within the signing CodeBuild environment, the primary source is the directory that the build starts out in. All the other input sources are what CodeBuild considers secondary sources and they get their own directories within the build environment. Those directories are stored in environment variables in the format `$CODEBUILD_SRC_DIR_<output_artifact_name>`. This allows you to locate the new artifact that needs to be signed and call the `sign_file` function with the new artifact.
+1. The signing is going to generate a new file that ends in `.asc`. You have to export that out of the signing environment for it to be picked up by the Copy to S3 CodeBuild project.
+1. The Copy to S3 CodeBuild project is already written to handle multiple files so no changes are required there.
+
 ## Secrets Manager access logs
 
 There is a separate template called `audit-logs-stack.yml` that contains audit logging for the key stored in AWS Secrets Manager. You can use CloudTrail to find the `GetSecretValue` events using the Event Name filter or using `secretsmanager.amazonaws.com` as the Event Source. This applies for the last 90 days.

build-infrastructure/audit-logs-stack.yml

@@ -1,5 +1,5 @@
 ---
-AWSTemplateFormatVersion: "2010-09-09"
+AWSTemplateFormatVersion: '2010-09-09'
 Description: A template that creates an audit log and associated logs and buckets
 
 Parameters:
@@ -44,13 +44,13 @@ Resources:
   AuditLogsBucket:
     Type: AWS::S3::Bucket
     Properties:
-      BucketName: !Sub "audit-logs-bucket-${AWS::AccountId}"
+      BucketName: !Sub 'audit-logs-bucket-${AWS::AccountId}'
 
   AuditLogsBucketPolicy:
     DependsOn: AuditLogsBucket
     Type: AWS::S3::BucketPolicy
     Properties:
-      Bucket: !Sub "audit-logs-bucket-${AWS::AccountId}"
+      Bucket: !Sub 'audit-logs-bucket-${AWS::AccountId}'
       PolicyDocument:
         Version: 2012-10-17
         Statement:
@@ -59,25 +59,25 @@ Resources:
             Principal:
               Service: cloudtrail.amazonaws.com
             Action: s3:GetBucketAcl
-            Resource: !Sub "arn:aws:s3:::audit-logs-bucket-${AWS::AccountId}"
+            Resource: !Sub 'arn:aws:s3:::audit-logs-bucket-${AWS::AccountId}'
             Condition:
               StringEquals:
-                AWS:SourceArn: !Sub "arn:aws:cloudtrail:${AWS::Region}:${AWS::AccountId}:trail/${KeyAccessTrailName}"
+                AWS:SourceArn: !Sub 'arn:aws:cloudtrail:${AWS::Region}:${AWS::AccountId}:trail/${KeyAccessTrailName}'
           - Sid: AWSCloudTrailWrite20150319
             Effect: Allow
             Principal:
               Service: cloudtrail.amazonaws.com
             Action: s3:PutObject
-            Resource: !Sub "arn:aws:s3:::audit-logs-bucket-${AWS::AccountId}/AWSLogs/${AWS::AccountId}/*"
+            Resource: !Sub 'arn:aws:s3:::audit-logs-bucket-${AWS::AccountId}/AWSLogs/${AWS::AccountId}/*'
             Condition:
               StringEquals:
                 s3:x-amz-acl: bucket-owner-full-control
-                AWS:SourceArn: !Sub "arn:aws:cloudtrail:${AWS::Region}:${AWS::AccountId}:trail/${KeyAccessTrailName}"
+                AWS:SourceArn: !Sub 'arn:aws:cloudtrail:${AWS::Region}:${AWS::AccountId}:trail/${KeyAccessTrailName}'
 
   KeyAccessCloudTrailCloudWatchLogsRole:
     Type: AWS::IAM::Role
     Properties:
-      RoleName: !Sub "key-access-cloudtrail-service-role-${AWS::Region}"
+      RoleName: !Sub 'key-access-cloudtrail-service-role-${AWS::Region}'
       AssumeRolePolicyDocument:
         Version: 2012-10-17
         Statement:
@@ -96,14 +96,14 @@ Resources:
                   - logs:CreateLogStream
                 Resource:
                   - !GetAtt KeyAccessAuditLogGroup.Arn
-                  - !Sub "${KeyAccessAuditLogGroup.Arn}:*"
+                  - !Sub '${KeyAccessAuditLogGroup.Arn}:*'
               - Sid: CloudTrailLogStreamEventAccess
                 Effect: Allow
                 Action:
                   - logs:PutLogEvents
                 Resource:
                   - !GetAtt KeyAccessAuditLogGroup.Arn
-                  - !Sub "${KeyAccessAuditLogGroup.Arn}:*"
+                  - !Sub '${KeyAccessAuditLogGroup.Arn}:*'
 
   KeyAccessCloudTrailAuditLog:
     DependsOn:
@@ -118,4 +118,4 @@ Resources:
       IsLogging: true
       IsMultiRegionTrail: true
       TrailName: !Ref KeyAccessTrailName
-      S3BucketName: !Sub "audit-logs-bucket-${AWS::AccountId}"
+      S3BucketName: !Sub 'audit-logs-bucket-${AWS::AccountId}'

build-infrastructure/codebuild-devbuild-stack.yml

@@ -42,7 +42,7 @@ Resources:
       QueuedTimeoutInMinutes: 60
       ServiceRole: !Ref ServiceRoleArm
       Source:
-        BuildSpec: buildspecs/build.yml
+        BuildSpec: buildspecs/pr-build.yml
         Location: !Ref GithubFullRepoName
         Type: GITHUB
       TimeoutInMinutes: 60
@@ -96,7 +96,7 @@ Resources:
       QueuedTimeoutInMinutes: 60
       ServiceRole: !Ref ServiceRoleAmd
       Source:
-        BuildSpec: buildspecs/build.yml
+        BuildSpec: buildspecs/pr-build.yml
         Location: !Ref GithubFullRepoName
         Type: GITHUB
       TimeoutInMinutes: 60
@@ -145,13 +145,13 @@ Resources:
             Statement:
               - Effect: Allow
                 Resource:
-                  - !Join 
+                  - !Join
                     - ':'
                     - - 'arn:aws:logs'
                       - !Ref 'AWS::Region'
                       - !Ref 'AWS::AccountId'
                       - !Sub 'log-group:/aws/codebuild/${BuildProjectName}-amd'
-                  - !Join 
+                  - !Join
                     - ':'
                     - - 'arn:aws:logs'
                       - !Ref 'AWS::Region'
@@ -172,7 +172,7 @@ Resources:
                   - 's3:GetBucketLocation'
               - Effect: Allow
                 Resource:
-                  - !Sub '${BuildBucketArn}/*' 
+                  - !Sub '${BuildBucketArn}/*'
                 Action:
                   - 's3:GetObject'
                   - 's3:PutObject'
@@ -198,13 +198,13 @@ Resources:
             Statement:
               - Effect: Allow
                 Resource:
-                  - !Join 
+                  - !Join
                     - ':'
                     - - 'arn:aws:logs'
                       - !Ref 'AWS::Region'
                       - !Ref 'AWS::AccountId'
                       - !Sub 'log-group:/aws/codebuild/${BuildProjectName}-arm'
-                  - !Join 
+                  - !Join
                     - ':'
                     - - 'arn:aws:logs'
                       - !Ref 'AWS::Region'