dra: use a google bucket that contains the elastic qualifier

[v1v]

Motivation/summary

Let's simplify the process ot create a DRA staging artifact with a given qualifier and use a Google Bucket that contains all the branches and their qualifiers.

This can help to support other Unified RElease projects without the need to care about the caveats when using push commits events.

In addition, I refactored some functions and simplify the logic:

1. Staging runs for all the branches except 8.x
2. Snapshot runs for all the branches but no Qualifier! In other words, the qualifer is not set if snapshot
3. package.sh and dra.sh uses util.sh to gather the Qualifier.
4. utils.sh contains functions to avoid getting crazy with a complex dra.sh or package.sh scripts.

Checklist

• Update CHANGELOG.asciidoc
• Documentation has been updated

For functional changes, consider:

• Is it observable through the addition of either logging or metrics?
• Is its use being published in telemetry to enable product improvement?
• Have system tests been added to avoid regression?

How to test these changes

#!/usr/bin/env bash

elastic_qualifier() {
  local branch=$1
  qualifier=""
  if curl -sf -o /dev/null "https://storage.googleapis.com/artifacts-api/test/$branch" ; then
    qualifier=$(curl -s "https://storage.googleapis.com/artifacts-api/test/$branch")
  fi
  echo "$qualifier"
}

echo ">>>> NO FILES"

qualifier=$(elastic_qualifier "8.x")
echo "-${qualifier}-"
qualifier=$(elastic_qualifier "main")
echo "-${qualifier}-"
qualifier=$(elastic_qualifier "8.17")
echo "-${qualifier}-"



echo ">>>> WITH FILES"
destination_bucket="gs://artifacts-api/test/"
out="8.x"
echo "$out-qualifier" > "$out"
gsutil cp "$out" "$destination_bucket"
out="main"
echo "$out-qualifier" > "$out"
gsutil cp "$out" "$destination_bucket"
out="8.17"
echo "$out-qualifier" > "$out"
gsutil cp "$out" "$destination_bucket"


qualifier=$(elastic_qualifier "8.x")
echo "-${qualifier}-"
qualifier=$(elastic_qualifier "main")
echo "-${qualifier}-"
qualifier=$(elastic_qualifier "8.17")
echo "-${qualifier}-"

>>>> NO FILES
--
--
--
>>>> WITH FILES                             
-8.x-qualifier-
-main-qualifier-
-8.17-qualifier-

See the dRA buildkite pipeline build -> https://buildkite.com/elastic/apm-server-package/builds/3810

Related issues

[mergify]

This pull request does not have a backport label. Could you fix it @v1v? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-7.17 is the label to automatically backport to the 7.17 branch.
• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit.
• backport-8.x is the label to automatically backport to the 8.x branch.

[mergify]

backport-8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label.

[kuisathaverat]

Could a project have different qualifier for the same branch?

[v1v]

Yes!

That's only for the release process, aka Staging. The release team will ask teams to:

1. Create an alpha1 qualifier release.
2. Then a new release with a different qualifier -> rc1, ... rc2 ... (let'\s say rc2 is the final one... in this example)
3. Finally, the release is approved and they will take rc2 as the valid release, hence the Release Team will ask product teams to run the DRA build without a qualifier for the same git commit.

We initially supported the env variable called ELASTIC_QUALIFIER, but since this is gonna happen to a bunch of other GitHub repositories, @dliappis and I came up with centralising the qualifier and branch in a Google Bucket. So we update the Google bucket and then the DRA Buildkite pipeline will use it.

[kuisathaverat]

so the answer is no, two Elastic Stack projects cannot have a different qualifier for the same branch. When the qualifier for branch X changes to Q, all projects must use the qualifier Q for branch X. Am I right?

[v1v]

Yes, temporarily. Let me clarify.

Currently, each product team triggers a new DRA with a new qualifier. Eventually, they will converge with the same qualifier per branch, either within a few hours or within one or two days at the most.

So far, this is trying to solve what you just described if all the Projects enrolled on the Unified Release use the same approach as here. Otherwise, it's a manual process and what you just said it can happen

[v1v]

@Mergifyio backport 7.17

[mergify]

backport 7.17

✅ Backports have been created

• #19369 [7.17] (backport #15350) dra: use a google bucket that contains the elastic qualifier has been created for branch 7.17 but encountered conflicts