Add script for tag porting when migrating libraries into the core mon…

…orepo (#1802)

Co-authored-by: Elliot Winkler <[email protected]>

docs/migrate-tags.md

@@ -0,0 +1,169 @@
+# `migrate-tags`
+
+When migrating libraries into the core monorepo, the original git history is transferred using the `git-filter-repo` tool (instructions [here](https://github.com/MetaMask/core/issues/1079#issuecomment-1700126302)), but tags attached to release commits are excluded from the process. This is because the tag names (`v[major].[minor].[patch]`) first need to be adjusted to conform to the scheme used by the core repo (`@metamask/<package-name>@[major].[minor].[patch]`).
+
+The `./scripts/migrate-tags.sh` script automates the process of enumerating the tags and associated release commit messages in the original repo, searching the migrated git history in the core repo's `merged-packages/<package-name>` directory for each commit message, creating tags with correctly-formatted names and attaching them to the found release commits, and pushing those tags to the core repo.
+
+## A. Preparations
+
+- The migration target package must be inside of the `merged-packages/` directory with its git history fully migrated.
+- The script must be run from the root directory of the core repo.
+- The `/tmp/<package-name>` directory used during the git history migration process should still be accessible. If not, perform steps 1-5 of [these instructions](https://github.com/MetaMask/core/issues/1079#issuecomment-1700126302) before proceeding.
+- If the script isn't executable, run `chmod +x ./scripts/migrate-tags.sh`.
+- By default, this script will run in "dry mode", printing out all pairs of release commit hashes and prefixed tag names, but not modifying the local or remote repo in any way. To override this and actually create/push tags, run the script with a `--no-dry-run` flag appended at the end.
+
+## B. Options
+
+- `<package-name>` (required).
+  - Only supply the package directory name. Exclude the `@metamask/` namespace.
+- `-r`, `--remote` (optional): the git remote repo where the tags will be pushed.
+  - Default if omitted: "test".
+- `-v`, `--version-before-package-rename` (optional)
+  - Default if omitted: `0.0.0`.
+  - **If `-v` is not passed, all tag names will be prepended with the `@metamask/` namespace.**
+- `-t`, `--tag-prefix-before-package-rename` (optional)
+  - Default if omitted: `<package-name>` supplied in the first argument.
+- `-d`, `--tmp-dir` (optional)
+  - Default if ommited: `/tmp`
+  - Specifies the temporary directory where `git-filter-repo` was applied to a clone of the original repo.
+- `-p`, `--sed-pattern` (optional): sed pattern for extracting verson numbers from the original repo's tag names.
+  - Default if omitted: `'s/^v//'`
+  - If the original tag names follow a different naming scheme than `v[major].[minor].[patch]`, adjust this setting.
+- `--no-dry-run` (optional):
+  - Default if omitted: `false`.
+  - If not specified, the script will run in "dry run" mode. The script will print out all pairs of release commit hashes and prefixed tag names, but without modifying the local or remote repo in any way.
+  - **This flag MUST be enabled for tags to be created and pushed.**
+  - Make sure to specify the correct remote repo where the tags will be pushed by using the `-r` flag.
+
+## C. Usage
+
+### 1. General Case (package never renamed)
+
+- For most cases, you will only need to specify the `<package-name>` as the first argument.
+
+```shell
+> ./scripts/migrate-tags.sh eth-json-rpc-provider
+```
+
+```output
+328a43ed @metamask/[email protected]
+06c41f6a @metamask/[email protected]
+de124c41 @metamask/[email protected]
+0aa45a9a @metamask/[email protected]
+d3a9f01c @metamask/[email protected]
+```
+
+### 2. Renamed Package
+
+- If the migration target package has been renamed, specify the `-v`, `--version-before-package-rename` option.
+
+```shell
+> ./scripts/migrate-tags.sh json-rpc-engine -v 6.1.0
+```
+
+```output
+67c7fee5 @metamask/[email protected]
+23aa8d9e @metamask/[email protected]
+76394323 @metamask/[email protected]
+22ff65e0 @metamask/[email protected]
+c753c16c @metamask/[email protected]
+670d8dd7 [email protected]
+9646dc26 [email protected]
+...
+```
+
+- The above output shows two `7.0.0` entries. If any duplicate release commits are found, the script will create and push tags only on the most recent commit.
+- The user has the option to supply a custom regex pattern under `-p` to narrow down the search results for the release commits.
+
+### 3. Package will be Renamed on the first Post-Migration Release
+
+- If the migration target package will be renamed after the migration, **specify the latest release version** in `-v`.
+
+```shell
+> ./scripts/migrate-tags.sh json-rpc-middleware-stream -v 5.0.1
+```
+
+```output
+38c007a3 [email protected]
+c34b1704 [email protected]
+8c6b70e5 [email protected]
+f7290013 [email protected]
+e08455ca [email protected]
+d90fe43d [email protected]
+...
+```
+
+### 4. Non-Dry Mode
+
+- To override dry run mode and actually create/push tags, run the script with a `--no-dry-run` flag at the end.
+- Make sure to specify the correct remote repo where the tags will be pushed by using the `-r` flag.
+
+```shell
+> ./scripts/migrate-tags.sh json-rpc-middleware-stream -v 5.0.1 -r origin --no-dry-run
+```
+
+```output
+Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
+To https://github.com/[USERNAME]/[FORKNAME]
+ * [new tag]           [email protected] -> [email protected]
+Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
+To https://github.com/[USERNAME]/[FORKNAME]
+ * [new tag]           [email protected] -> [email protected]
+Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
+
+...
+
+To https://github.com/[USERNAME]/[FORKNAME]
+ * [new tag]           [email protected] -> [email protected]
+Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
+To https://github.com/[USERNAME]/[FORKNAME]
+ * [new tag]           [email protected] -> [email protected]
+```
+
+## D. Verify
+
+- Check whether the tags have correctly been pushed to the remote repo.
+
+```shell
+> git ls-remote --tags origin | grep 'json-rpc-engine'
+```
+
+```output
+22ff65e0f76710188b527bd5d3f81dd2103c5514        refs/tags/@metamask/[email protected]
+7639432339e60767a8239d681911375833bc3839        refs/tags/@metamask/[email protected]
+23aa8d9e59d9275c0725cb0264057e082034dae9        refs/tags/@metamask/[email protected]
+67c7fee5141f6c0bb2f459c1cb3062c02bbf6a15        refs/tags/@metamask/[email protected]
+304f6efa4d1be2460c9d0bec48224cefcf7fd208        refs/tags/[email protected]
+4909d7fd95a555a7ae18cb1f9840db4fe1f3c85d        refs/tags/[email protected]
+93e2b7224f7370468466e2e5e29a2c10da016b11        refs/tags/[email protected]
+286c2716a7b856b95f74d64edd9e653728dd031c        refs/tags/[email protected]
+...
+```
+
+## E. Troubleshooting
+
+> [!WARNING]
+> DO NOT run this script on the core repo until the results have been tested on a fork.
+
+The following commands should NOT be run on the core repo unless something has gone very wrong.
+
+### 1. Delete remote tags
+
+**WARNING**: Proceed with EXTREME CAUTION
+
+```shell
+> git ls-remote --tags <remote-repo> | grep '<package-name>' | cut -f2 | sed 's|refs/tags/||g' | xargs git push --delete <remote-repo>
+```
+
+- ALWAYS create a backup clone repo in advance and delete local tags AFTER remote tags.
+- If something goes wrong, try `git push <remote-repo>` to push the local tags to remote.
+- If the local tags have been deleted, push the unaltered tags in the backup clone repo to remote.
+- If this fails, ask a teammate who has the correct tags on local to push them to remote.
+
+### 2. Delete local tags
+
+```shell
+> git tag | grep '<package-name>' | xargs git tag --delete
+```
+
+- If anything goes wrong, run `git pull --all` and the tags in the remote repo will be restored to local.

scripts/migrate-tags.sh

@@ -0,0 +1,165 @@
+#!/usr/bin/env bash
+
+source "$PWD/scripts/semver.sh"
+
+remote='test'
+version_before_package_rename='0.0.0'
+tag_prefix_before_package_rename="$1"
+tmp_dir='/tmp'
+sed_pattern='s/^v//'
+dry_run=true
+
+print-usage() {
+  cat <<EOF
+
+Migrates tags.
+
+$0 [OPTIONS] PACKAGE_NAME
+
+OPTIONS:
+
+-r REMOTE
+--remote REMOTE
+  Specifies the remote git repo where the tags will be pushed.
+
+-v VERSION_BEFORE_PACKAGE_RENAME
+--version-before-package-rename VERSION_BEFORE_PACKAGE_RENAME
+  The version before the package rename. If package was never renamed, omit this and all tag names will be prepended with the '@metamask/' namespace.
+
+-t TAG_PREFIX_BEFORE_PACKAGE_RENAME
+--tag-prefix-before-package-rename TAG_PREFIX_BEFORE_PACKAGE_RENAME
+  Specifies the tag prefix before the package rename. Defaults to the package name.
+
+-d TMP_DIR
+--tmp-dir TMP_DIR
+  Specifies the temporary directory where the $(git-filter-repo)-applied clone of the original repo is located. Defaults to '/tmp'.
+
+-p SED_PATTERN
+--sed-pattern SED_PATTERN
+  sed pattern for extracting verson numbers from the original repo's tag names. Adjust if the original tag names follow a different naming scheme than 'v0.0.0'.
+
+--no-dry-run
+  If specified, the tags will be created and pushed to the remote repo. Otherwise, the tags and associated release commit hashes will only be printed to stdout.
+EOF
+}
+
+while [[ $# -gt 0 ]]; do
+  key="$1"
+
+  case $key in
+  -h | --help)
+    print-usage
+    exit 0
+    ;;
+  -r | --remote)
+    remote="$2"
+    shift # past argument
+    shift # past value
+    ;;
+  -v | --version-before-package-rename)
+    version_before_package_rename="$2"
+    shift # past argument
+    shift # past value
+    ;;
+  -t | --tag-prefix-before-package-rename)
+    tag_prefix_before_package_rename="$2"
+    shift # past argument
+    shift # past value
+    ;;
+  -d | --tmp-dir)
+    tmp_dir="$2"
+    shift # past argument
+    shift # past value
+    ;;
+  -p | --sed-pattern)
+    sed_pattern="$2"
+    shift # past argument
+    shift # past value
+    ;;
+  --no-dry-run)
+    dry_run=false
+    shift # past argument
+    shift # past value
+    ;;
+  *) # package name
+    package_name="$1"
+    shift # past argument
+    ;;
+  esac
+done
+
+get-tag-commit-pairs() {
+  echo "$(cd $tmp_dir/$package_name && git tag --format="%(refname:short)"$'\t'"%(objectname)")"
+}
+
+get-version-message-pairs() {
+  local version
+  local message
+  while IFS=$'\t' read -r tag commit; do
+    version="$(echo "$tag" | sed "$sed_pattern")"
+    message="$(cd $tmp_dir/$package_name && git log $commit -n 1 --oneline --format='%s')"
+    echo "$version"$'\t'"$message"
+  done <<<"$(get-tag-commit-pairs)"
+}
+
+find-commits-matching-message() {
+  local expected_message="$1"
+  while IFS=$'\t' read -r commit actual_message; do
+    if [[ $actual_message == $expected_message ]]; then
+      echo "$commit"
+    fi
+  done <<<"$(git log --oneline --format='%H%x09%s' --grep="$expected_message" --fixed-strings)"
+}
+
+get-version-commit-pairs() {
+  local commits
+  local num_commits
+  local commits_as_string
+  local error
+  while IFS=$'\t' read -r version message; do
+    commits="$(find-commits-matching-message "$message")"
+    num_commits="$(echo $commits | wc -l | sed -E 's/^[ ]+//')"
+    commits_as_string="$(echo $commits | awk '{ if(FNR == 1) { printf "%s", $0 } else { printf ", %s", $0 } }')"
+    if [[ $num_commits -eq 0 ]]; then
+      error="Could not find commit for version '$version' and message '$message'."
+    elif [[ $num_commits -gt 1 ]]; then
+      error="More than one commit found for '$version' and message '$message': $commits_as_string"
+    else
+      error=""
+    fi
+    echo "$version"$'\t'"$commits"$'\t'"$message"$'\t'"$error"
+  done <<<"$(get-version-message-pairs)"
+}
+
+get-commit-tagname-pairs() {
+  local tag_name
+  while IFS=$'\t' read -r version commit message error; do
+    if semverLT "$version" "$version_before_package_rename" || semverEQ "$version" "$version_before_package_rename"; then
+      tag_name="$tag_prefix_before_package_rename@$version"
+    else
+      tag_name="@metamask/$package_name@$version"
+    fi
+    echo "$tag_name"$'\t'"$commit"$'\t'"$message"$'\t'"$error"
+  done <<<"$(get-version-commit-pairs)"
+}
+
+main() {
+  if [[ -z $package_name ]]; then
+    echo "Missing package name."
+    print-usage
+    exit 1
+  fi
+  while IFS=$'\t' read -r tag_name commit message error; do
+    if [[ -n $error ]]; then
+      echo "ERROR: $error" >&2
+    elif [[ $dry_run == true ]]; then
+      echo "$commit"$'\t'"$tag_name"$'\t'"$message"
+    else
+      echo "Creating tag '$tag_name'..."
+      git tag "$tag_name" "$commit"
+      git push "$remote" "$tag_name"
+    fi
+  done <<<"$(get-commit-tagname-pairs)"
+}
+
+main