Skip to content

Commit 4b59b09

Browse files
maru-avaalarso16
maru-ava
and
alarso16
committed
[migrate-coreth][1] Add documentation and tooling for repo migration
Adds README documenting proposed procedure and associated tooling. Co-authored-by: Austin Larson <[email protected]>
1 parent 4d07d90 commit 4b59b09

File tree

6 files changed

+286
-0
lines changed

6 files changed

+286
-0
lines changed

‎flake.nix‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,9 @@
6363
protoc-gen-go-grpc
6464
protoc-gen-connect-go
6565

66+
# Line-oriented search tool
67+
ripgrep
68+
6669
# Solidity compiler
6770
solc
6871

‎graft/README.md‎

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
# Monorepo Grafting
2+
3+
This directory is intended to be the initial point of migration for
4+
other git repos being added to this repo. Such 'grafts' do not have to
5+
initially adhere to repo standards but will be expected to be
6+
refactored where necessary to meet such standards as part of a
7+
PR-based process of migrating to more permanent locations.
8+
9+
## Migrating a Repo
10+
11+
### Stacked Branches
12+
13+
As with with any changes to a codebase, minimizing review friction is
14+
essential. While it would be possible to require review of a huge PR
15+
commit-by-commit, any changes to those commits would need to be
16+
correlated with their originals and the friction would be
17+
considerable. Instead, a [stacked
18+
branch](https://andrewlock.net/working-with-stacked-branches-in-git-part-1/)
19+
approach is suggested to enable effective review of a large migration
20+
in a piecemeal fashion:
21+
22+
- Create a branch per reviewable task
23+
- Subtree merge would be one task, import rewrite another, etc
24+
- The branch for a task subsequent to the initial task would be
25+
based on the previous task branch
26+
- Create a PR per branch
27+
- The initial task would use the master branch as its base
28+
- A subsequent task would use the previous task's branch as its base
29+
- Mark each PR as draft to avoid premature merge
30+
- Request review in order from the initial PR but do not merge yet
31+
- Once all PRs in the series have been approved, merge from the top down
32+
- Avoids cascading rebases and merge conflicts
33+
34+
Tooling such as
35+
[git-machete](https://github.com/VirtusLab/git-machete) or
36+
[jujutsu](https://github.com/jj-vcs/jj) is suggested to simplify
37+
maintaining the series of stacked branches.
38+
39+
### Suggested Procedure
40+
41+
The following do not represent an exhaustive list of tasks and are
42+
used for example purposes only. Regardless of the steps involved, the
43+
creation of an initial branch from master is assumed before the first
44+
step, and the commit of all changes and creation of a new branch from
45+
the current branch before beginning a subsequent step.
46+
47+
- [ ] Add tasks for subtree merge and import rewrite to graft/Taskfile.yml (as per the example of existing tasks)
48+
- These tasks are intended to simplify the repeated invocation that
49+
will be required when a repo is being developed in parallel with
50+
migration.
51+
- [ ] Execute the subtree merge task (it will commit the result automatically)
52+
- [ ] Remove files made redundant by the migration
53+
- Prioritizing file removal before modification minimizes the changes requiring review
54+
- [ ] Execute the rewrite imports task (it will commit the result automatically)
55+
- [ ] Perform required go module changes
56+
- [ ] Migrate CI jobs (unit test, e2e, linting, etc)
57+
- [ ] Get CI jobs passing
58+
59+
### Rebasing Inflight PRs
60+
61+
Provided a subtree merge was used to perform the graft, a common merge
62+
base will exist with which to rebase PRs targeted at the original git
63+
repo. This allows for repo migration and ongoing development to
64+
proceed in parallel. Once a graft has been finalized, outstanding PR
65+
branches from the original repo can be migrated as follows:
66+
67+
```bash
68+
# Fetch the PR branch from the standalone repo's remote
69+
git fetch REMOTE PR_BRANCH_NAME
70+
71+
# Create a local branch from the PR branch.
72+
# Don't track since the new branch is intended to target the new repo.
73+
git checkout -b PR_BRANCH_NAME REMOTE/PR_BRANCH_NAME --no-track
74+
75+
# Rebase onto the target branch in this repo, treating graft/REPO as the subtree root.
76+
# Conflict resolution may be required.
77+
git rebase -Xsubtree=graft/REPO --onto origin/master REMOTE/master
78+
79+
# Push the new PR branch
80+
git push -u origin PR_BRANCH_NAME
81+
```

‎graft/Taskfile.yml‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
# https://taskfile.dev
2+
# To run on a system without task installed, `../scripts/run_task.sh` will execute it with `go run`.
3+
# If in the nix dev shell, `task` is available.
4+
5+
version: '3'
6+
7+
vars:
8+
CORETH_MODULE: github.com/ava-labs/coreth
9+
10+
tasks:
11+
default: ../scripts/run_task.sh --list
12+
13+
coreth-rewrite-imports:
14+
desc: Rewrite imports of out-of-tree coreth to in-tree coreth
15+
cmd: bash ./scripts/rewrite-imports.sh {{.CORETH_MODULE}}
16+
17+
coreth-subtree-merge:
18+
desc: Perform git subtree merge of coreth into graft/coreth (commits)
19+
cmd: bash ./scripts/subtree-merge.sh {{.CORETH_MODULE}} master

‎graft/scripts/get-module-version.sh‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
#!/usr/bin/env bash
2+
3+
set -euo pipefail
4+
5+
# Extracts the version for a Go module from go.mod
6+
# This script outputs ONLY the version string to stdout for use in shell command substitution.
7+
#
8+
# Usage: get-module-version.sh <module-path>
9+
# Example: get-module-version.sh github.com/ava-labs/coreth
10+
#
11+
# Output format:
12+
# - If module uses a pseudo-version (vX.Y.Z-YYYYMMDDHHMMSS-abcdef123456):
13+
# Outputs the first 8 characters of the 12-char commit hash
14+
# - If module uses a regular tag/version:
15+
# Outputs the version as-is
16+
17+
if [ $# -ne 1 ]; then
18+
echo "Error: exactly one argument required" >&2
19+
echo "Usage: $0 <module-path>" >&2
20+
echo "Example: $0 github.com/ava-labs/coreth" >&2
21+
exit 1
22+
fi
23+
24+
MODULE_PATH="$1"
25+
26+
# Get module details from go.mod
27+
MODULE_DETAILS="$(go list -m "${MODULE_PATH}" 2>/dev/null || true)"
28+
if [ -z "${MODULE_DETAILS}" ]; then
29+
echo "Error: module ${MODULE_PATH} not found in go.mod" >&2
30+
exit 1
31+
fi
32+
33+
# Extract version from module details (second field)
34+
VERSION="$(echo "${MODULE_DETAILS}" | awk '{print $2}')"
35+
36+
if [ -z "${VERSION}" ]; then
37+
echo "Error: could not extract version from module details: ${MODULE_DETAILS}" >&2
38+
exit 1
39+
fi
40+
41+
# Check if version is a module hash (pseudo-version format: v*YYYYMMDDHHMMSS-abcdef123456)
42+
if [[ "${VERSION}" =~ ^v.*[0-9]{14}-[0-9a-f]{12}$ ]]; then
43+
# Extract the 12-character commit hash from the end
44+
MODULE_HASH="$(echo "${VERSION}" | grep -Eo '[0-9a-f]{12}$')"
45+
# Use the first 8 characters of the hash
46+
# This is the convention used for avalanchego image tags
47+
echo "${MODULE_HASH::8}"
48+
else
49+
# Regular tag/version - output as-is
50+
echo "${VERSION}"
51+
fi

‎graft/scripts/rewrite-imports.sh‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
#!/usr/bin/env bash
2+
3+
set -euo pipefail
4+
5+
# Rewrites Go module imports from an external package to an internal graft subdirectory
6+
# This script modifies the working tree and commits the changes.
7+
#
8+
# Usage: rewrite-imports.sh <original-module-import-path>
9+
# Example: rewrite-imports.sh github.com/ava-labs/coreth
10+
# Will rewrite: github.com/ava-labs/coreth -> github.com/ava-labs/avalanchego/graft/coreth
11+
12+
if [ $# -ne 1 ]; then
13+
echo "Error: exactly one argument required"
14+
echo "Usage: $0 <original-module-import-path>"
15+
echo "Example: $0 github.com/ava-labs/coreth"
16+
exit 1
17+
fi
18+
19+
ORIGINAL_IMPORT="$1"
20+
21+
# Extract the last component of the import path
22+
# e.g., github.com/ava-labs/coreth -> coreth
23+
PACKAGE_NAME="${ORIGINAL_IMPORT##*/}"
24+
25+
TARGET_IMPORT="github.com/ava-labs/avalanchego/graft/${PACKAGE_NAME}"
26+
27+
REPO_ROOT=$( cd "$( dirname "${BASH_SOURCE[0]}" )"; cd ../.. && pwd )
28+
cd "${REPO_ROOT}"
29+
30+
echo "rewriting ${ORIGINAL_IMPORT} imports to ${TARGET_IMPORT} in all golang files"
31+
32+
# Escape dots in the original import path for regex use
33+
ESCAPED_ORIGINAL="${ORIGINAL_IMPORT//./\\.}"
34+
35+
rg "${ESCAPED_ORIGINAL}" -t go --files-with-matches --null | \
36+
xargs -0 perl -i -pe "s|\\Q${ORIGINAL_IMPORT}\\E|${TARGET_IMPORT}|g"
37+
38+
echo "import rewrite completed successfully"
39+
40+
echo "staging changes..."
41+
git add -u '*.go'
42+
43+
echo "committing changes..."
44+
git commit -m "Rewrite ${ORIGINAL_IMPORT} imports to ${TARGET_IMPORT}
45+
46+
Rewrites all Go import statements from external package ${ORIGINAL_IMPORT}
47+
to internal graft subdirectory ${TARGET_IMPORT}."
48+
49+
echo "changes committed successfully"

‎graft/scripts/subtree-merge.sh‎

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
#!/usr/bin/env bash
2+
3+
set -euo pipefail
4+
5+
# Performs a git subtree merge of an external repository into a graft subdirectory and then commits the result.
6+
#
7+
# Usage: subtree-merge.sh <module-path> [version]
8+
# Example: subtree-merge.sh github.com/ava-labs/coreth
9+
# Example: subtree-merge.sh github.com/ava-labs/coreth 1a498175
10+
#
11+
# Arguments:
12+
# module-path: The Go module path (e.g., github.com/ava-labs/coreth)
13+
# version: (Optional) The version/tag/SHA to merge (can be a tag, branch, or commit SHA)
14+
# If not provided, the version will be discovered from go.mod
15+
#
16+
# The repository URL is constructed by prepending https:// to the module path.
17+
# The target path is automatically derived as graft/[repo-name] where repo-name
18+
# is the last component of the module path.
19+
#
20+
# Prerequisites:
21+
# - Must be run from a git repository
22+
# - Target path must not already exist
23+
#
24+
# What this script does:
25+
# 1. Constructs repository URL from module path
26+
# 2. Discovers version from go.mod if not provided
27+
# 3. Derives target path from module path (graft/[last-component])
28+
# 4. Validates that target path doesn't already exist
29+
# 5. Adds the external repo as a temporary git remote
30+
# 6. Performs a merge with 'ours' strategy (keeps our history, adds theirs)
31+
# 7. Reads the external repo's tree into the target subdirectory
32+
# 8. Commits the merge with a descriptive message
33+
# 9. Removes the temporary remote
34+
35+
if [ $# -lt 1 ] || [ $# -gt 2 ]; then
36+
echo "Error: one or two arguments required" >&2
37+
echo "Usage: $0 <module-path> [version]" >&2
38+
echo "Example: $0 github.com/ava-labs/coreth" >&2
39+
echo "Example: $0 github.com/ava-labs/coreth 1a498175" >&2
40+
exit 1
41+
fi
42+
43+
MODULE_PATH="$1"
44+
45+
REPO_ROOT=$( cd "$( dirname "${BASH_SOURCE[0]}" )"; cd ../.. && pwd )
46+
cd "${REPO_ROOT}"
47+
48+
# Discover version from go.mod if not provided
49+
if [ $# -eq 2 ]; then
50+
VERSION="$2"
51+
echo "using provided version: ${VERSION}"
52+
else
53+
echo "discovering version from go.mod"
54+
VERSION="$(bash "${REPO_ROOT}/graft/scripts/get-module-version.sh" "${MODULE_PATH}")"
55+
echo "discovered version: ${VERSION}"
56+
fi
57+
58+
# Construct repository URL from module path
59+
REPO_URL="https://${MODULE_PATH}"
60+
61+
# Extract repository name from module path
62+
# Example: github.com/ava-labs/coreth -> coreth
63+
REPO_BASENAME="$(basename "${MODULE_PATH}")"
64+
TARGET_PATH="graft/${REPO_BASENAME}"
65+
66+
# Check if target path already exists
67+
if [ -d "${REPO_ROOT}/${TARGET_PATH}" ]; then
68+
echo "Target path ${TARGET_PATH} already exists, skipping subtree merge"
69+
exit 0
70+
fi
71+
72+
REMOTE_NAME="${REPO_BASENAME}"
73+
74+
echo "adding remote ${REMOTE_NAME} from ${REPO_URL}"
75+
git remote add -f "${REMOTE_NAME}" "${REPO_URL}"
76+
77+
echo "performing subtree merge of ${VERSION} into ${TARGET_PATH}"
78+
git subtree add --prefix="${TARGET_PATH}" "${REMOTE_NAME}" "${VERSION}"
79+
80+
echo "removing ${REMOTE_NAME} remote"
81+
git remote remove "${REMOTE_NAME}"
82+
83+
echo "subtree merge of ${REPO_BASENAME} completed successfully"

0 commit comments

Comments
 (0)