Automate the creation of release tags after every release

[bbarani]

Is your feature request related to a problem? Please describe.
Currently the plugin teams are creating release tags manually after every release. This can lead to discrepancies when they cut the release tag using a different commit id compared to the one shipped with the artifact

Describe the solution you'd like
The build system knows the the commit id associated with each (generated) component artifacts shipped as part of a distribution. We should come up with an automated process (workflow) to cut release tags for all the components participating in a release based on the commit id (in the build manifest file) after every release.

• Create proof of concept
• Create a design document
• Implementation

Describe alternatives you've considered
Manually creating release tags after release

[dblock]

This can help, given a manifest, e.g. https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.2.1/53/linux/x64/dist/opensearch/manifest.yml

version=1.2.1.0
for im in $(yq eval -o=j manifest.yml | jq -cr '.components[]'); do
      name=$(echo $im | jq -r '.name' -)
      repo=$(echo $im | jq -r '.repository' | sed s/.git$//g)
      commit_id=$(echo $im | jq -r '.commit_id' -)     
      echo $name $repo $commit_id
done

[dblock]

For 1.2.3 this is what I am doing:

doit.sh

#!/bin/bash
set -e

version=1.2.3
build_number=295

wget https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/$version/$build_number/linux/x64/dist/opensearch/manifest.yml

for im in $(yq eval -o=j manifest.yml | jq -cr '.components[]'); do
      name=$(echo $im | jq -r '.name' -)
      repo=$(echo $im | jq -r '.repository' -)
      commit_id=$(echo $im | jq -r '.commit_id' -)
      echo "Tagging $name at $commit_id ..."
      mkdir $name
      cd $name
      git init
      git remote add origin $repo
      git fetch --depth 1 origin $commit_id
      git checkout FETCH_HEAD
      if [ "$name" == "OpenSearch" ]; then
            git tag $version
      else
            git tag $version.0
      fi
      git push origin --tags
      cd ..
done

common-utils and performance-analyzer-rca

The common-utils and performance-analyzer-rca projects don't redistribute anything, so I had to tag that one manually.

[peterzhuamazon]

We need to add another workflow to change input manifest with tags and PR changes.

[zelinh]

Why we don't use the build manifest to fetch the component name, repo, commit_id and etc.
e.g. from https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/$version/$build_number/linux/x64/build/opensearch/manifest.yml
This will also include plugin like common-util which didn't redistribute.

[zelinh]

The current approach I am thinking is:
With version and build number as input parameters, we could acquire the build manifest from S3 using downloadFromS3 groovy method with manifest path like following (distribution-build-opensearch/$version/$build_number/linux/x64/build/opensearch/manifest.yml). We could get the information (e.g. component’s name, commit id, repo link) from the build manifest with implemented BuildManifest.groovy and some new functions to retrieve data from components. With all these, we could then create GitHub release tag for all components with a Github bot account.

[zelinh]

Closing this as this has been completed.