Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
126 changes: 126 additions & 0 deletions .claude/skills/merge-train-infra/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
---
name: merge-train-infra
description: Reference for merge-train automation internals -- workflows, scripts, CI integration, and configuration. Use when modifying or debugging merge-train infrastructure.
---

# Merge-Train Infrastructure

This skill covers the automation internals of the merge-train system. For contributor-facing guidance (creating PRs, labels, handling failures), see the `merge-trains` skill.

## Automation Lifecycle

The merge-train system is fully automated via GitHub Actions in `.github/workflows/merge-train-*.yml`:

1. **PR Creation** (`merge-train-create-pr.yml`): Triggered on push to `merge-train/*` branches. Creates a PR targeting `next` with the `ci-no-squash` label (and `ci-full-no-test-cache` for spartan). Skips merge commits and commits already in `next`.

2. **Body Updates** (`merge-train-update-pr-body.yml`): Triggered on push to `merge-train/**`. Updates the PR body with meaningful commits (those containing PR references like `(#1234)`). The body uses `BEGIN_COMMIT_OVERRIDE` / `END_COMMIT_OVERRIDE` markers for release-please.

3. **Next Integration** (`merge-train-next-to-branches.yml`): Triggered on push to `next`. Merges `next` into each active merge-train branch via `scripts/merge-train/merge-next.sh`. Uses `continue-on-error: true` so a conflict in one branch does not block others. Skips branches whose PR already has auto-merge enabled.

4. **Auto-Merge** (`merge-train-auto-merge.yml`): Runs hourly via cron (`0 * * * *`). Calls `scripts/merge-train/auto-merge.sh` for both merge-train (4-hour inactivity) and backport-train (8-hour inactivity) branches. Uses separate GitHub tokens: `AZTEC_BOT_GITHUB_TOKEN` for API calls and `MERGE_TRAIN_GITHUB_TOKEN` for approvals. Will not auto-merge if the last merge-queue CI run failed or was cancelled.

5. **Recreation & Wakeup** (`merge-train-recreate.yml`): Triggered when a PR is closed (merged). If the merged PR's head branch starts with `merge-train/`, recreates the branch from the base branch (usually `next`). Then runs `scripts/merge-train/wakeup-prs.sh` to add the `ci-wakeup-pr-after-merge` label to all open PRs targeting the branch that have passed CI and have automerge enabled. This triggers a CI re-run (typically a no-op via tree-hash cache) so those PRs can proceed through the merge queue. The label is immediately removed by a step in `ci3.yml` so it can be re-applied on subsequent merges.

6. **Failure Notification** (`merge-queue-dequeue-notify.yml`): Triggered when a PR is dequeued from the merge queue. If the PR's head branch starts with `merge-train/` and the PR was NOT merged, sends a Slack notification via `ci3/merge_train_failure_slack_notify`.

## CI Integration Details

### CI Mode Selection (`.github/ci3_labels_to_env.sh`)

Merge-train branches influence CI mode:
- `merge_group` events or `ci-merge-queue` label → `merge-queue` mode
- If the merge-group event is for `merge-train/spartan` → upgraded to `merge-queue-heavy` mode (10 parallel grind runs instead of 4)
- Target branch `merge-train/docs` → `ci-docs` mode
- Target branch `merge-train/barretenberg` → `ci-barretenberg` mode

### CI Concurrency (`.github/workflows/ci3.yml`)

```yaml
group: ci3-${{ (startsWith(github.event.pull_request.head.ref, 'merge-train/') && github.run_id) || ... }}
```

Merge-train PRs get **full concurrency** (each run has its own unique group via `github.run_id`), while non-merge-train PRs share a group by branch name with cancel-in-progress.

### Instance Postfix (`.github/ci3.sh`)

```bash
if [[ "${PR_HEAD_REF:-}" == merge-train/* ]]; then
export INSTANCE_POSTFIX=${PR_COMMITS:-}
fi
```

Merge-train PRs get a unique instance postfix (commit count) to allow parallel EC2 instances.

### CI Modes in bootstrap.sh

- `ci-docs`: Only builds and tests documentation
- `ci-barretenberg`: Only builds and tests barretenberg (AVM disabled)
- `ci-barretenberg-full`: Full barretenberg CI including acir_tests
- `merge-queue`: 4x AMD64 full + 1x ARM64 fast in parallel
- `merge-queue-heavy`: 10x AMD64 full + 1x ARM64 fast in parallel (used for `merge-train/spartan`)

### Test History Tracking (`ci3/run_test_cmd`)

```bash
if [[ "$is_merge_queue" -eq 1 || ("${TARGET_BRANCH:-}" =~ ^v[0-9]) || ("${TARGET_BRANCH:-}" == merge-train/*) ]]; then
track_test_history=1
fi
```

### Failure Notification (`ci3/bootstrap_ec2`)

When a CI run fails on an EC2 instance, it calls `merge_train_failure_slack_notify` to send failure notifications to the appropriate Slack channel based on the branch name.

## Creating a New Merge Train

1. Create a branch from `next` with naming pattern `merge-train/{team}`
2. Add the branch to the matrix in `.github/workflows/merge-train-next-to-branches.yml`
3. Add the branch-to-Slack-channel mapping in `ci3/merge_train_failure_slack_notify`
4. Optionally add CI mode overrides in `.github/ci3_labels_to_env.sh` and `bootstrap.sh`
5. Push code to the branch -- automation handles PR creation from there

## Key Files Reference

### Workflows

| File | Purpose |
|---|---|
| `.github/workflows/merge-train-readme.md` | User-facing documentation |
| `.github/workflows/merge-train-create-pr.yml` | Auto-creates PRs for train branches |
| `.github/workflows/merge-train-auto-merge.yml` | Hourly cron to auto-merge inactive trains |
| `.github/workflows/merge-train-next-to-branches.yml` | Syncs `next` into all train branches; defines active branches |
| `.github/workflows/merge-train-recreate.yml` | Recreates branch after merge |
| `.github/workflows/merge-train-update-pr-body.yml` | Updates PR body with commit list |
| `.github/workflows/merge-queue-dequeue-notify.yml` | Slack notification on merge-queue dequeue |
| `.github/workflows/squashed-pr-check.yml` | Squash enforcement (skipped for `ci-no-squash`) |

### Scripts

| File | Purpose |
|---|---|
| `scripts/merge-train/auto-merge.sh` | Auto-merge logic -- checks inactivity, last CI status, approves and merges |
| `scripts/merge-train/merge-next.sh` | Merges `next` into a train branch, handles conflicts, cancels stale CI runs |
| `scripts/merge-train/update-pr-body.sh` | Updates PR body with meaningful commits |
| `scripts/merge-train/squash-pr.sh` | Squashes PR commits (used by `ci-squash-and-merge` label) |
| `scripts/merge-train/wakeup-prs.sh` | Adds `ci-wakeup-pr-after-merge` label to qualifying PRs after branch recreation |

### CI Configuration

| File | Purpose |
|---|---|
| `.github/ci3_labels_to_env.sh` | CI mode selection based on labels and target branches |
| `.github/ci3.sh` | Instance postfix for merge-train parallelism |
| `ci3/merge_train_failure_slack_notify` | Slack failure notification with branch-to-channel mapping |
| `ci3/run_test_cmd` | Test history tracking for merge-train branches |
| `ci3/bootstrap_ec2` | EC2 failure notification trigger |
| `bootstrap.sh` | CI mode definitions (`ci-docs`, `ci-barretenberg`, etc.) |

### Other Scripts

| File | Purpose |
|---|---|
| `scripts/auto_close_issues.py` | Auto-closes issues referenced in merged merge-train PRs (GitHub's native auto-close doesn't work for intermediate branches) |
| `scripts/find_orphaned_issues_in_prs.py` | Finds PRs in merge-train commits that reference still-open issues |
| `scripts/dedupe_release_notes.py` | Deduplicates release notes from merge-train merges |
| `scripts/commits` | Pretty git log that groups merge-train children by subsystem |
| `scripts/filter_history` | Filters git history, identifying merge-train merge commits as "containers" |
82 changes: 82 additions & 0 deletions .claude/skills/merge-trains/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
---
name: merge-trains
description: Guide for working with merge-train branches -- creating PRs, choosing the right base branch, understanding labels, handling failures, and bypassing checks.
---

# Working with Merge Trains

## What Is a Merge Train?

A merge train is an automated batching system (inspired by [Rust rollups](https://forge.rust-lang.org/release/rollups.html)) that groups multiple PRs together for coordinated integration into the `next` branch. Instead of each PR going through the merge queue individually, teams push their PRs into a shared `merge-train/*` branch. Periodically, that branch is merged as a single unit into `next`.

## Active Merge-Train Branches

| Branch | Team / Domain | Slack Channel |
|---|---|---|
| `merge-train/avm` | AVM, barretenberg vm2 folder | `#team-bonobos` |
| `merge-train/barretenberg` | Barretenberg folder, but not vm2 folder | `#honk-team` |
| `merge-train/ci` | CI infrastructure / ci3 | `#help-ci` |
| `merge-train/docs` | Documentation | `#dev-rels` |
| `merge-train/fairies` | aztec-nr | `#team-fairies` |
| `merge-train/spartan` | Spartan / infra / yarn-project sequencer and prover orchestration | `#team-alpha` |

## How to Use a Merge Train

### Targeting a Merge Train with Your PR

1. Create your feature branch **off the appropriate merge-train branch** (not our default branch `next`).
2. Open your PR targeting that merge-train branch (e.g., base: `merge-train/barretenberg`).
3. When your PR is approved and merged, it gets squashed into the merge-train branch.
4. The merge-train PR (which targets `next`) automatically accumulates your commit.

### Key Rules for Contributors

- **Base branch matters**: Always branch from the branch specified in the CI_BASE_BRANCH environment variable. If it is not set, then ask the user their intent and offer to set CI_BASE_BRANCH in their shell's RC file.
- **Your PR is squashed into the train**: Individual PRs targeting a merge-train branch are squash-merged as usual. You should not use the merge commit merge method, but the squash method.
- **The train itself is NOT squashed**: The merge-train PR (e.g., `merge-train/barretenberg` -> `next`) is merged with a **merge commit**, preserving the individual squashed commits. This is why the `ci-no-squash` label is automatically applied.
- **You generally don't need to worry about the train PR itself** -- it is fully automated (creation, body updates, approval, merge, and recreation). You only need to pay attention to it if an alert is sent to your team channel.

## CI Behavior for Merge Trains

- **Specialized CI modes**: PRs targeting `merge-train/docs` run docs-only CI. PRs targeting `merge-train/barretenberg` run barretenberg-only CI. This avoids running the full test suite for domain-specific changes.
- **Merge-queue mode**: When the merge-train PR enters GitHub's merge queue, it runs the full `merge-queue` CI mode (4 parallel grind runs on AMD64 + 1 ARM64). `merge-train/spartan` uses the heavier `merge-queue-heavy` mode (10 grind runs).
- **Full concurrency**: Merge-train PRs get unique CI concurrency groups (using `github.run_id`), so multiple CI runs can proceed in parallel without cancelling each other.
- **Test history tracking**: Test results are tracked for merge-train PRs, same as merge-queue runs.

## Handling Merge-Train Failures

### When CI Fails on the Merge-Train PR

Two options from the [merge-train-readme.md](https://github.com/AztecProtocol/aztec-packages/blob/next/.github/workflows/merge-train-readme.md):

**Option 1: Direct Fix** -- Push a fix directly to the merge-train branch. Use bypass merge to expedite (all users have this permission). You can use the ci-skip label to no-op CI if really necessary.

**Option 2: Fix in Next** -- Merge a revert or workaround into `next`. The fix will auto-propagate to the merge-train via the `merge-train-next-to-branches` workflow. Best when the root cause is in `next` or multiple trains are affected.

### When Auto-Merge Is Blocked

The auto-merge script will **not** enable auto-merge if the last merge-queue CI run for the PR concluded with `failure` or `cancelled`. Someone needs to either fix the issue and push, or force-merge.

### Merge Conflicts from Next

When merging `next` into a train branch causes conflicts, the `merge-next.sh` script:
- Aborts the merge
- Posts a comment on the latest `next` commit listing the conflicted files
- The team must manually resolve conflicts on their train branch

## Bypassing Checks / Force-Merging

If the user needs to bypass CI checks for their merge-train PR (e.g., a known flaky failure, an urgent merge, or CI infrastructure issues):

1. **Confirm intent**: Always confirm with the user that they want to skip CI, since this merges untested code into `next`.
2. **Add the `ci-skip` label**: Apply the `ci-skip` label to the merge-train PR. This causes CI to skip entirely. Use: `gh pr edit <PR_NUMBER> --add-label ci-skip`
3. **Force merge in the UI**: The user can then use GitHub's "Merge without waiting for requirements to be met" button (bypass merge) in the PR UI. All users have this permission.

**Important**: Only do this when the user explicitly asks to bypass checks. Always confirm first since it skips all CI validation.

## Backport Trains

A related system exists for backport branches (`backport-to-*`). These use the same auto-merge mechanism but with different settings:
- Branch pattern: `backport-to-`
- Inactivity threshold: 8 hours (instead of 4)
- Merge strategy: merge commit
6 changes: 6 additions & 0 deletions .github/workflows/ci3.yml
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,12 @@ jobs:
if: github.event.pull_request.head.repo.fork != true && (github.event.pull_request.draft == false || contains(github.event.pull_request.labels.*.name, 'ci-draft'))
environment: ${{ startsWith(github.ref, 'refs/tags/v') && 'master' || '' }}
steps:
- name: Remove wakeup label
if: contains(github.event.pull_request.labels.*.name, 'ci-wakeup-pr-after-merge')
env:
GH_TOKEN: ${{ secrets.AZTEC_BOT_GITHUB_TOKEN }}
run: gh pr edit ${{ github.event.pull_request.number }} --remove-label ci-wakeup-pr-after-merge --repo ${{ github.repository }} || true

- name: Checkout
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
with:
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/merge-train-next-to-branches.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ jobs:
- merge-train/barretenberg
- merge-train/ci
- merge-train/docs
- merge-train/fairies
- merge-train/spartan
steps:
- name: Checkout
Expand Down
55 changes: 13 additions & 42 deletions .github/workflows/merge-train-readme.md
Original file line number Diff line number Diff line change
@@ -1,49 +1,20 @@
# Merge Train Documentation

## Overview
## Skills

The merge train is an automated system for managing pull requests in a coordinated queue. It helps prevent merge conflicts and ensures smooth integration of multiple PRs targeting the same base branch.
Inspired by [rust rollups](https://forge.rust-lang.org/release/rollups.html), but that's a non-ideal name for our domain.
The merge-train system is documented via Claude Code skills in `.claude/skills/`. These are structured markdown files designed to be consumed by LLMs (such as Claude) to answer interactive questions about the system.

## How It Works
| Skill | Description |
|---|---|
| [`merge-trains`](../../.claude/skills/merge-trains/SKILL.md) | Contributor-facing guide -- creating PRs, choosing the right base branch, understanding labels, CI behavior, handling failures, and bypassing checks. |
| [`merge-train-infra`](../../.claude/skills/merge-train-infra/SKILL.md) | Infrastructure reference -- workflows, scripts, CI integration, configuration, and how to create a new merge train. |

1. **Merge Train Branches**: Special branches prefixed with `merge-train/` (e.g., `merge-train/barretenberg`, `merge-train/docs`)
2. **Automatic Recreation**: When a merge train PR is merged, the branch is automatically recreated with an empty commit
3. **PR Body Updates**: The merge train PR body is automatically updated with a list of commits whenever changes are pushed
4. **Auto-merge**: PRs that have been inactive for 4+ hours are automatically approved and merged (if they contain meaningful commits)
5. **Next Branch Integration**: Changes from the `next` branch are automatically merged into active merge train branches
### Using the Skills

## Using the Merge Train
Ask Claude Code questions like:
- "How do I add my PR to a merge train?"
- "What happens when a merge-train CI run fails?"
- "How do I create a new merge-train branch?"
- "How does the auto-merge system work?"

### Creating a New Merge-Train

1. **Create Branch**: Fork from `next` with naming pattern `merge-train/{team}`
2. **Enable Auto-Sync**: Add your branch to `.github/workflows/merge-train-next-to-branches.yml`
3. **Open PR**: Create a PR from your merge-train branch to `next` - automation handles the rest

### Adding Your PR to the Merge Train

1. Create your feature branch and make your changes
2. Instead of targeting `master` or `next` directly, target the appropriate merge train branch (e.g., `merge-train/barretenberg`)
3. When your PR merges the changes will be included in the next merge train cycle

### Merge Train Lifecycle

1. **Creation**: A merge train PR is created automatically when changes are pushed to the branch and labeled with `ci-no-squash`.
2. **Accumulation**: Feature PRs are merged into the merge train branch, squashed
3. **Auto-merge**: After 4 hours of inactivity (with meaningful commits), the train is automatically merged with a merge-commit (not a squash)
4. **Recreation**: The cycle starts again with a new merge train

## Handling Merge Failures

When a merge-train fails due to issues from `next`:

### Option 1: Direct Fix
- Push a fix directly to the merge-train branch
- Use bypass merge to expedite (all users have this permission)
- Best for small, quick fixes

### Option 2: Fix in Next
- Merge a revert or workaround into `next`
- The fix will auto-propagate to merge-train via automation
- Best when key assumptions are broken or multiple trains affected
Claude will load the relevant skill and provide contextual answers based on the actual automation code.
7 changes: 7 additions & 0 deletions .github/workflows/merge-train-recreate.yml
Original file line number Diff line number Diff line change
Expand Up @@ -36,3 +36,10 @@ jobs:
# This will generally only succeed if the branch does not exist, as a fallback if the branch is unprotected and gets deleted.
git push origin "$MERGE_TRAIN_BRANCH"

- name: Wake up PRs targeting this branch
env:
GH_TOKEN: ${{ secrets.AZTEC_BOT_GITHUB_TOKEN }}
run: |
MERGE_TRAIN_BRANCH="${{ github.event.pull_request.head.ref }}"
./scripts/merge-train/wakeup-prs.sh "$MERGE_TRAIN_BRANCH"

2 changes: 2 additions & 0 deletions ci3/merge_train_failure_slack_notify
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,8 @@ elif [[ "$REF_NAME" == "merge-train/ci" ]]; then
channel="#help-ci"
elif [[ "$REF_NAME" == "merge-train/docs" ]]; then
channel="#dev-rels"
elif [[ "$REF_NAME" == "merge-train/fairies" ]]; then
channel="#team-fairies"
elif [[ "$REF_NAME" == "merge-train/spartan" ]]; then
channel="#team-alpha"
else
Expand Down
Loading
Loading