Merge #229

229: verify and publish binaries generated by CI r=saem a=alaviss

## Summary

This is the final portion of continous delivery pipeline which creates a
release based on artifacts created during a CI run done as part of a
merge via bors.

The release created will contain a JSON manifest and release binaries.
The JSON manifest follows this schema:

```json
{
  "manifestVersion": 0,
  "version": "version identifier (ie. 0.1.0-dev.20000)",
  "source": {
    "name": "source-archive-name",
    "sha256": "checksum"
  },
  "binaries": [
    {
      "target": "the target triplet (eg. x86_64-linux-gnu)",
      "name": "binary-archive-name",
      "sha256": "checksum"
    }
  ]
}
```

This manifest allow tools to easily pick out the artifact that they need
depending on the target operating system, and it also prevent the need
to hard-code a specific file name format in such tools.

## Details

This PR comes with:

- A release manifest manipulation tool: This tool is used to create a
  release manifest from archive manifests generated by niminst. At the
  moment it does not have many features other than the ones required by
  the pipeline. Refer to the tool built-in help for details on how to
  use it.

- An additional test in main CI to verify that the release manifest tool
  can create a release manifest from binaries generated there.

- Extra steps in publisher that download and create a release out of
  artifacts created by main CI.



Co-authored-by: Leorize <[email protected]>

.github/workflows/ci.yml

@@ -448,6 +448,12 @@ jobs:
           name: release binaries
           path: release-binary
 
+      - name: Download release source archive
+        uses: actions/[email protected]
+        with:
+          name: source archive
+          path: source-archive
+
       - name: Install tools
         run: |
           sudo apt-get update
@@ -479,6 +485,26 @@ jobs:
           name: differences between binary from source archive and git
           path: ${{ steps.diff.outputs.result }}
 
+      - name: Verify archive manifests
+        run: |
+          # Verify them by using the manifest builder tool to create a manifest
+          # for the full bundle.
+          bin/nim c tools/release_manifest.nim
+          cd release-binary
+          ../tools/release_manifest add ../source-archive/*.json *.json
+
+          # Print the resulting manifest
+          echo "Success! Generated manifest:"
+          jq . manifest.json
+
+      # This allow the publisher to run the tool directly without having to
+      # clone the compiler.
+      - name: Upload release manifest tool
+        uses: actions/[email protected]
+        with:
+          name: release manifest tool
+          path: tools/release_manifest
+
   passed:
     name: All check passed
     needs: [binaries, test, tooling, source, source_binaries, package, test_package, orc]

.github/workflows/publisher.yml

@@ -20,20 +20,73 @@ jobs:
       # Publish action needs a checkout
       - uses: actions/[email protected]
 
-      # Download the latest instance of generated documentation from the build
-      # during bors staging.
+      # Download the latest instance of artifacts from a build done previously
       - name: Download generated docs
         uses: dawidd6/[email protected]
         with:
           workflow: ci.yml
-          workflow_conclusion: completed
+          workflow_conclusion: success
           commit: ${{ github.event.after }}
           # Keep up-to-date with ci.yml
           name: Generated docs
           path: doc/html
 
+      - name: Download generated source archive
+        uses: dawidd6/[email protected]
+        with:
+          workflow: ci.yml
+          workflow_conclusion: success
+          commit: ${{ github.event.after }}
+          # Keep up-to-date with ci.yml
+          name: source archive
+          path: release-staging
+
+      - name: Download generated release binaries
+        uses: dawidd6/[email protected]
+        with:
+          workflow: ci.yml
+          workflow_conclusion: success
+          commit: ${{ github.event.after }}
+          # Keep up-to-date with ci.yml
+          name: release binaries
+          path: release-staging
+
+      - name: Download release manifest tool
+        uses: dawidd6/[email protected]
+        with:
+          workflow: ci.yml
+          workflow_conclusion: success
+          commit: ${{ github.event.after }}
+          # Keep up-to-date with ci.yml
+          name: release manifest tool
+          path: release-staging
+
       - name: Publish docs
         uses: JamesIves/[email protected]
         with:
           branch: gh-pages
           folder: doc/html
+
+      - id: release-files
+        name: Create release manifest
+        run: |
+          # Github Artifacts strip executable permissions so it has to be set again
+          chmod 755 release_manifest
+          # Create a new release manifest
+          ./release_manifest add *.json
+
+          toUpload=$(./release_manifest files-to-upload --format:github-actions)
+          echo "::set-output name=result::$toUpload"
+          echo "::set-output name=version::$(./release_manifest version)"
+        working-directory: release-staging
+
+      - name: Create pre-release
+        uses: softprops/[email protected]
+        with:
+          prerelease: true
+          files: ${{ steps.release-files.outputs.result }}
+          tag_name: ${{ steps.release-files.outputs.version }}
+          fail-on-unmatched-files: true
+          target_commitish: ${{ github.event.after }}
+          body: |
+            Continuous delivery for commit ${{ github.event.after }}