ci(gb200): add 1-node mr-github functional test variants

skills/build-and-dependency/SKILL.md

@@ -0,0 +1,211 @@
+---
+name: build-and-dependency
+description: Container-based dev environment setup and dependency management for Megatron-LM. Covers acquiring and launching the CI container, uv package management, updating uv.lock, and linting.
+TRIGGER when: user asks to add, remove, or update a dependency; user edits or asks about pyproject.toml or uv.lock; uv.lock has a merge conflict; user asks to set up a dev environment or pull/build the CI container; user hits a container build error or uv error; user asks to run linting or autoformat.
+DO NOT TRIGGER when: user is only running tests, investigating CI failures, or opening a PR (use testsystem instead).
+---
+
+# Build & Dependency Guide
+
+The core principle: **build and develop inside containers** — the CI container
+ships the correct CUDA toolkit, PyTorch build, and pre-compiled native extensions
+(TransformerEngine, DeepEP, …) that cannot be reproduced on a bare host.
+
+---
+
+## Why Containers
+
+Megatron-LM depends on CUDA, NCCL, PyTorch with GPU support, TransformerEngine,
+and optional components like ModelOpt and DeepEP. Installing these on a bare host
+is fragile and hard to reproduce. The project ships Dockerfiles that pin every
+dependency.
+
+**Use the container as your development environment.** This guarantees:
+
+- Identical CUDA / NCCL / cuDNN versions across all developers and CI.
+- `uv.lock` resolves the same way locally and in CI.
+- GPU-dependent operations (training, testing) work out of the box.
+
+---
+
+## Step 1 — Acquire an Image
+
+**Option A — NVIDIA-internal: pull a CI-built image**
+
+> ⚠️ Requires access to the internal GitLab instance.
+> See `tools/trigger_internal_ci.md` for setup (adding the git remote, obtaining a token).
+
+The internal GitLab CI publishes images to its container registry.
+Derive the registry host from your configured `gitlab` remote — the same
+host you use for `trigger_internal_ci.py`:
+
+```bash
+# Derive host from your 'gitlab' remote:
+GITLAB_HOST=$(git remote get-url gitlab | sed 's/.*@\(.*\):.*/\1/')
+
+docker pull ${GITLAB_HOST}/adlr/megatron-lm/mcore_ci_dev:main
+```
+
+**Option B — Build from scratch (works for everyone)**
+
+> ⚠️ `Dockerfile.ci.dev` has two stages: `main` and `jet`. The `jet` stage
+> requires an internal build secret and will fail without it. Always pass
+> `--target main` to stop at the public stage.
+
+```bash
+# dev image (default)
+docker build \
+  --target main \
+  --build-arg FROM_IMAGE_NAME=$(cat docker/.ngc_version.dev) \
+  --build-arg IMAGE_TYPE=dev \
+  -f docker/Dockerfile.ci.dev \
+  -t megatron-lm:local .
+
+# lts image
+docker build \
+  --target main \
+  --build-arg FROM_IMAGE_NAME=$(cat docker/.ngc_version.lts) \
+  --build-arg IMAGE_TYPE=lts \
+  -f docker/Dockerfile.ci.dev \
+  -t megatron-lm:local-lts .
+```
+
+Which image variant is used is controlled by the PR label `container::lts`;
+absent that label, `dev` is used.
+
+---
+
+## Step 2 — Launch the Container
+
+**Option A — Local Docker runtime**
+
+```bash
+docker run --rm --gpus all \
+  -v $(pwd):/workspace \
+  -w /workspace \
+  megatron-lm:local \
+  bash -c "<your command>"
+```
+
+**Option B — Slurm cluster (for those without a local Docker runtime)**
+
+NVIDIA clusters typically use [Pyxis](https://github.com/NVIDIA/pyxis) +
+[enroot](https://github.com/NVIDIA/enroot). Request an interactive session:
+
+```bash
+srun \
+  --nodes=1 --gpus-per-node=8 \
+  --container-image megatron-lm:local \
+  --container-mounts $(pwd):/workspace \
+  --container-workdir /workspace \
+  --pty bash
+```
+
+For clusters that require a `.sqsh` archive first:
+
+```bash
+enroot import -o megatron-lm.sqsh dockerd://megatron-lm:local
+srun \
+  --nodes=1 --gpus-per-node=8 \
+  --container-image $(pwd)/megatron-lm.sqsh \
+  --container-mounts $(pwd):/workspace \
+  --container-workdir /workspace \
+  --pty bash
+```
+
+---
+
+## Dependency Management
+
+Dependencies are declared in `pyproject.toml`. The venv lives at `/opt/venv`
+inside the container (already on `PATH`).
+
+> **All `uv` operations must be run inside the container.**
+> Never run `uv sync` / `uv pip install` on the host.
+
+### uv Dependency Groups
+
+| Group | Purpose |
+|-------|---------|
+| `training` | Runtime training extras |
+| `dev` | Full dev environment (TransformerEngine, ModelOpt, …) |
+| `lts` | LTS-safe subset (no ModelOpt) |
+| `test` | pytest, coverage, nemo-run |
+| `linting` | ruff, black, isort, pylint |
+| `build` | Cython, pybind11, nvidia-mathdx |
+
+Install commands (inside the container):
+
+```bash
+# Full dev + test environment
+uv sync --locked --group dev --group test
+
+# Linting only
+uv sync --locked --only-group linting
+
+# LTS environment
+uv sync --locked --group lts --group test
+```
+
+Several dependencies are sourced directly from git (TransformerEngine, nemo-run,
+FlashMLA, Emerging-Optimizers, nvidia-resiliency-ext). The locked `uv.lock` file
+pins exact revisions; update it with `uv lock` when changing `pyproject.toml`.
+
+### Adding a New Dependency
+
+Follow this three-step workflow:
+
+1. **Acquire a container image** — see [Step 1](#step-1--acquire-an-image) above.
+2. **Launch the container interactively** — see [Step 2](#step-2--launch-the-container) above.
+3. **Update the lock file inside the container**, then commit it:
+
+   ```bash
+   # Inside the container:
+   uv add <package>          # adds to pyproject.toml and resolves
+   uv lock                   # regenerates uv.lock
+   # Exit the container, then on the host:
+   git add pyproject.toml uv.lock
+   git commit -S -s -m "build: add <package> dependency"
+   ```
+
+### Resolving a merge conflict in uv.lock
+
+`uv.lock` is machine-generated; never resolve conflicts manually. Instead:
+
+```bash
+git checkout origin/main -- uv.lock   # take main's version as the base
+# then inside the container:
+uv lock                               # re-resolve on top of your pyproject.toml changes
+```
+
+---
+
+## Linting
+
+Run before opening a PR:
+
+```bash
+# Check mode (no changes applied)
+BASE_REF=main CHECK_ONLY=true SKIP_DOCS=false bash tools/autoformat.sh
+
+# Fix mode
+BASE_REF=main CHECK_ONLY=false bash tools/autoformat.sh
+```
+
+Tools invoked: `black`, `isort`, `pylint`, `ruff`, `mypy`.
+
+After editing imports in any Python files, always run `uv run isort` on those
+files before committing (repo CLAUDE.md requirement).
+
+---
+
+## Common Pitfalls
+
+| Problem | Cause | Fix |
+|---------|-------|-----|
+| `uv sync --locked` fails | Dependency conflict or stale `uv.lock` | Re-run `uv lock` inside the container and commit updated lock |
+| `ModuleNotFoundError` after pip install | pip installed outside the uv-managed venv | Use `uv add` and `uv sync`, never bare `pip install` |
+| `uv: command not found` inside container | Wrong container image | Use the `megatron-lm` image built from `Dockerfile.ci.dev` |
+| `No space left on device` during uv ops | Cache fills container's `/root/.cache/` | Mount a host cache dir via `-v $HOME/.cache/uv:/root/.cache/uv` |
+| Pre-commit fails with linting errors | Code style violations | Run `BASE_REF=main CHECK_ONLY=false bash tools/autoformat.sh` |
+| `docker build` fails with secret-related error | `Dockerfile.ci.dev` has a `jet` stage that requires an internal secret | Add `--target main` to stop before the `jet` stage |

skills/onboard-gb200-1node-tests/SKILL.md

@@ -0,0 +1,150 @@
+---
+name: onboard-gb200-1node-tests
+description: Onboard 1-node GitHub MR functional tests for GB200 from existing mr-scoped 2-node tests. Use when the user asks to add GB200 github-mr tests, create single-node variants of existing tests, or expand CI coverage for GB200.
+user_invocable: true
+argument: "[model-yaml]  # optional: gpt, moe, or both (default: both)"
+TRIGGER when: user asks to add GB200 mr-github tests, create 1-node variants of functional tests, or onboard tests for GitHub CI on GB200.
+DO NOT TRIGGER when: user is asking about H100 tests or making unrelated changes to existing test configs.
+---
+
+# Onboard GB200 1-Node GitHub MR Tests
+
+Create 1-node (`mr-github`) variants of existing 2-node (`mr`-scoped) GB200 functional tests.
+Each GB200 node has **4 GPUs**. A 2-node test uses 8 GPUs total; the 1-node variant uses 4.
+
+---
+
+## Background
+
+GB200 functional tests live in `tests/test_utils/recipes/gb200/`:
+
+| Recipe file | Notes |
+|-------------|-------|
+| `gpt.yaml` | GPT dense tests, `nodes: 2, gpus: 4` (8 total) |
+| `moe.yaml` | MoE tests, `nodes: 2, gpus: 4` (8 total) |
+| `moe-1node.yaml` | Existing 1-node MoE tests, `nodes: 1, gpus: 4` (4 total) |
+| `gpt-1node.yaml` | 1-node GPT tests (create if not present) |
+
+Model configs live at:
+`tests/functional_tests/test_cases/{model}/{test_case}/model_config.yaml`
+
+1-node test cases use the `_1node` suffix:
+`tests/functional_tests/test_cases/{model}/{test_case}_1node/model_config.yaml`
+
+---
+
+## Workflow
+
+### Step 1 — Find candidate tests
+
+Scan the `products:` block in `gpt.yaml` and `moe.yaml` for entries with `scope: [mr, ...]` or `scope: [mr-slim, ...]`. These are the 2-node tests that need 1-node `mr-github` counterparts.
+
+Ignore tests already covered in `*-1node.yaml` files, and ignore `nightly`, `weekly`, `mr-broken` scopes.
+
+### Step 2 — Read each model config
+
+For each candidate, read its `model_config.yaml` and extract the key parallelism arguments:
+
+```
+--tensor-model-parallel-size   (TP)
+--pipeline-model-parallel-size (PP)
+--expert-model-parallel-size   (EP)
+--expert-tensor-parallel-size  (ETP)
+--context-parallel-size        (CP)
+--global-batch-size
+--micro-batch-size
+```
+
+### Step 3 — Classify: trivial copy vs. needs adaptation
+
+The world size formula is: `world_size = TP × PP × DP` where `DP ≥ EP`.
+
+Going from 8 GPUs → 4 GPUs:
+
+| Condition | Action |
+|-----------|--------|
+| `TP × PP ≤ 4` | **Trivial copy.** Config unchanged; DP is halved automatically. |
+| `TP × PP = 8` (e.g. tp4 pp2) | **Reduce PP.** Set `PP = PP / 2` (e.g. pp2→1). Verify `TP × PP_new ≤ 4`. |
+| `EP > 4` (e.g. ep8 with tp1 pp1) | **Reduce EP.** Set `EP = 4`. Experts stay at `num-experts` (each EP rank holds more experts). |
+| `EP > 4` **and** `TP × PP > 4` | Reduce both PP and EP as above. |
+| ETP test (ep × etp ≤ TP × DP) | Check `EP × ETP ≤ TP × DP_new` after PP reduction. Usually satisfied when pp→1. |
+
+**Do not change GBS** — let gradient accumulation absorb the reduced DP.
+
+### Step 4 — Create `_1node` model config directories
+
+```bash
+# Trivial copy
+mkdir -p tests/functional_tests/test_cases/{model}/{test_case}_1node
+cp tests/functional_tests/test_cases/{model}/{test_case}/model_config.yaml \
+   tests/functional_tests/test_cases/{model}/{test_case}_1node/model_config.yaml
+
+# Then apply any parallelism changes (EP or PP) with Edit tool
+```
+
+### Step 5 — Create or update recipe files
+
+**For GPT tests** — create `tests/test_utils/recipes/gb200/gpt-1node.yaml` (if absent) by cloning `gpt.yaml`'s spec block with `nodes: 1`. Use this template for the spec:
+
+```yaml
+type: basic
+format_version: 1
+maintainers: [mcore]
+loggers: [stdout]
+spec:
+  name: "{test_case}_{environment}_{platforms}"
+  model: gpt          # or moe
+  build: mcore-pyt-{environment}
+  nodes: 1
+  gpus: 4
+  n_repeat: 5
+  platforms: dgx_gb200
+  script_setup: |    # copy verbatim from gpt.yaml / moe.yaml
+    ...
+  script: |-         # copy verbatim from gpt.yaml / moe.yaml
+    ...
+```
+
+**For MoE tests** — append entries to the existing `moe-1node.yaml`.
+
+### Step 6 — Add products entries
+
+Scope convention:
+- **1–2 most representative tests** per recipe: `scope: [mr-github, mr-github-slim]`
+- **All other tests**: `scope: [mr-github]`
+
+```yaml
+products:
+  - test_case: [<test_case>_1node]
+    products:
+      - environment: [dev]
+        scope: [mr-github, mr-github-slim]   # or [mr-github]
+        platforms: [dgx_gb200]
+```
+
+---
+
+## Quick parallelism reference
+
+| Original (8 GPUs) | 1-node config (4 GPUs) | Notes |
+|-------------------|----------------------|-------|
+| tp1 pp1 ep1 → dp8 | tp1 pp1 ep1 → dp4 | trivial |
+| tp2 pp1 ep1 → dp4 | tp2 pp1 ep1 → dp2 | trivial |
+| tp1 pp2 ep1 → dp4 | tp1 pp2 ep1 → dp2 | trivial |
+| tp4 pp1 ep1 → dp2 | tp4 pp1 ep1 → dp1 | trivial |
+| tp1 pp4 ep1 → dp2 | tp1 pp4 ep1 → dp1 | trivial |
+| tp1 pp1 ep8 → dp8 | tp1 pp1 ep4 → dp4 | ep 8→4 |
+| tp4 pp2 ep2 etp2 → dp1 | tp4 pp1 ep2 etp2 → dp1 | pp 2→1 |
+
+---
+
+## Checklist
+
+- [ ] Identified all `mr`-scoped tests in `gpt.yaml` and `moe.yaml` not yet in `*-1node.yaml`
+- [ ] Read model config for each candidate
+- [ ] Classified trivial vs. adaptation needed
+- [ ] Created `_1node/model_config.yaml` for each test
+- [ ] Applied EP or PP reductions where needed
+- [ ] Created/updated recipe YAML with `nodes: 1, gpus: 4`
+- [ ] Assigned `mr-github` scope (+ `mr-github-slim` for 1–2 representative tests per recipe)
+- [ ] Verified no `mr-github-slim` overload (slim suite should stay small)