diff --git a/skills/build-and-dependency/SKILL.md b/skills/build-and-dependency/SKILL.md new file mode 100644 index 00000000000..c7dc89ec1ad --- /dev/null +++ b/skills/build-and-dependency/SKILL.md @@ -0,0 +1,211 @@ +--- +name: build-and-dependency +description: Container-based dev environment setup and dependency management for Megatron-LM. Covers acquiring and launching the CI container, uv package management, updating uv.lock, and linting. +TRIGGER when: user asks to add, remove, or update a dependency; user edits or asks about pyproject.toml or uv.lock; uv.lock has a merge conflict; user asks to set up a dev environment or pull/build the CI container; user hits a container build error or uv error; user asks to run linting or autoformat. +DO NOT TRIGGER when: user is only running tests, investigating CI failures, or opening a PR (use testsystem instead). +--- + +# Build & Dependency Guide + +The core principle: **build and develop inside containers** — the CI container +ships the correct CUDA toolkit, PyTorch build, and pre-compiled native extensions +(TransformerEngine, DeepEP, …) that cannot be reproduced on a bare host. + +--- + +## Why Containers + +Megatron-LM depends on CUDA, NCCL, PyTorch with GPU support, TransformerEngine, +and optional components like ModelOpt and DeepEP. Installing these on a bare host +is fragile and hard to reproduce. The project ships Dockerfiles that pin every +dependency. + +**Use the container as your development environment.** This guarantees: + +- Identical CUDA / NCCL / cuDNN versions across all developers and CI. +- `uv.lock` resolves the same way locally and in CI. +- GPU-dependent operations (training, testing) work out of the box. + +--- + +## Step 1 — Acquire an Image + +**Option A — NVIDIA-internal: pull a CI-built image** + +> ⚠️ Requires access to the internal GitLab instance. +> See `tools/trigger_internal_ci.md` for setup (adding the git remote, obtaining a token). + +The internal GitLab CI publishes images to its container registry. +Derive the registry host from your configured `gitlab` remote — the same +host you use for `trigger_internal_ci.py`: + +```bash +# Derive host from your 'gitlab' remote: +GITLAB_HOST=$(git remote get-url gitlab | sed 's/.*@\(.*\):.*/\1/') + +docker pull ${GITLAB_HOST}/adlr/megatron-lm/mcore_ci_dev:main +``` + +**Option B — Build from scratch (works for everyone)** + +> ⚠️ `Dockerfile.ci.dev` has two stages: `main` and `jet`. The `jet` stage +> requires an internal build secret and will fail without it. Always pass +> `--target main` to stop at the public stage. + +```bash +# dev image (default) +docker build \ + --target main \ + --build-arg FROM_IMAGE_NAME=$(cat docker/.ngc_version.dev) \ + --build-arg IMAGE_TYPE=dev \ + -f docker/Dockerfile.ci.dev \ + -t megatron-lm:local . + +# lts image +docker build \ + --target main \ + --build-arg FROM_IMAGE_NAME=$(cat docker/.ngc_version.lts) \ + --build-arg IMAGE_TYPE=lts \ + -f docker/Dockerfile.ci.dev \ + -t megatron-lm:local-lts . +``` + +Which image variant is used is controlled by the PR label `container::lts`; +absent that label, `dev` is used. + +--- + +## Step 2 — Launch the Container + +**Option A — Local Docker runtime** + +```bash +docker run --rm --gpus all \ + -v $(pwd):/workspace \ + -w /workspace \ + megatron-lm:local \ + bash -c "" +``` + +**Option B — Slurm cluster (for those without a local Docker runtime)** + +NVIDIA clusters typically use [Pyxis](https://github.com/NVIDIA/pyxis) + +[enroot](https://github.com/NVIDIA/enroot). Request an interactive session: + +```bash +srun \ + --nodes=1 --gpus-per-node=8 \ + --container-image megatron-lm:local \ + --container-mounts $(pwd):/workspace \ + --container-workdir /workspace \ + --pty bash +``` + +For clusters that require a `.sqsh` archive first: + +```bash +enroot import -o megatron-lm.sqsh dockerd://megatron-lm:local +srun \ + --nodes=1 --gpus-per-node=8 \ + --container-image $(pwd)/megatron-lm.sqsh \ + --container-mounts $(pwd):/workspace \ + --container-workdir /workspace \ + --pty bash +``` + +--- + +## Dependency Management + +Dependencies are declared in `pyproject.toml`. The venv lives at `/opt/venv` +inside the container (already on `PATH`). + +> **All `uv` operations must be run inside the container.** +> Never run `uv sync` / `uv pip install` on the host. + +### uv Dependency Groups + +| Group | Purpose | +|-------|---------| +| `training` | Runtime training extras | +| `dev` | Full dev environment (TransformerEngine, ModelOpt, …) | +| `lts` | LTS-safe subset (no ModelOpt) | +| `test` | pytest, coverage, nemo-run | +| `linting` | ruff, black, isort, pylint | +| `build` | Cython, pybind11, nvidia-mathdx | + +Install commands (inside the container): + +```bash +# Full dev + test environment +uv sync --locked --group dev --group test + +# Linting only +uv sync --locked --only-group linting + +# LTS environment +uv sync --locked --group lts --group test +``` + +Several dependencies are sourced directly from git (TransformerEngine, nemo-run, +FlashMLA, Emerging-Optimizers, nvidia-resiliency-ext). The locked `uv.lock` file +pins exact revisions; update it with `uv lock` when changing `pyproject.toml`. + +### Adding a New Dependency + +Follow this three-step workflow: + +1. **Acquire a container image** — see [Step 1](#step-1--acquire-an-image) above. +2. **Launch the container interactively** — see [Step 2](#step-2--launch-the-container) above. +3. **Update the lock file inside the container**, then commit it: + + ```bash + # Inside the container: + uv add # adds to pyproject.toml and resolves + uv lock # regenerates uv.lock + # Exit the container, then on the host: + git add pyproject.toml uv.lock + git commit -S -s -m "build: add dependency" + ``` + +### Resolving a merge conflict in uv.lock + +`uv.lock` is machine-generated; never resolve conflicts manually. Instead: + +```bash +git checkout origin/main -- uv.lock # take main's version as the base +# then inside the container: +uv lock # re-resolve on top of your pyproject.toml changes +``` + +--- + +## Linting + +Run before opening a PR: + +```bash +# Check mode (no changes applied) +BASE_REF=main CHECK_ONLY=true SKIP_DOCS=false bash tools/autoformat.sh + +# Fix mode +BASE_REF=main CHECK_ONLY=false bash tools/autoformat.sh +``` + +Tools invoked: `black`, `isort`, `pylint`, `ruff`, `mypy`. + +After editing imports in any Python files, always run `uv run isort` on those +files before committing (repo CLAUDE.md requirement). + +--- + +## Common Pitfalls + +| Problem | Cause | Fix | +|---------|-------|-----| +| `uv sync --locked` fails | Dependency conflict or stale `uv.lock` | Re-run `uv lock` inside the container and commit updated lock | +| `ModuleNotFoundError` after pip install | pip installed outside the uv-managed venv | Use `uv add` and `uv sync`, never bare `pip install` | +| `uv: command not found` inside container | Wrong container image | Use the `megatron-lm` image built from `Dockerfile.ci.dev` | +| `No space left on device` during uv ops | Cache fills container's `/root/.cache/` | Mount a host cache dir via `-v $HOME/.cache/uv:/root/.cache/uv` | +| Pre-commit fails with linting errors | Code style violations | Run `BASE_REF=main CHECK_ONLY=false bash tools/autoformat.sh` | +| `docker build` fails with secret-related error | `Dockerfile.ci.dev` has a `jet` stage that requires an internal secret | Add `--target main` to stop before the `jet` stage | diff --git a/skills/onboard-gb200-1node-tests/SKILL.md b/skills/onboard-gb200-1node-tests/SKILL.md new file mode 100644 index 00000000000..ef5642f73fb --- /dev/null +++ b/skills/onboard-gb200-1node-tests/SKILL.md @@ -0,0 +1,150 @@ +--- +name: onboard-gb200-1node-tests +description: Onboard 1-node GitHub MR functional tests for GB200 from existing mr-scoped 2-node tests. Use when the user asks to add GB200 github-mr tests, create single-node variants of existing tests, or expand CI coverage for GB200. +user_invocable: true +argument: "[model-yaml] # optional: gpt, moe, or both (default: both)" +TRIGGER when: user asks to add GB200 mr-github tests, create 1-node variants of functional tests, or onboard tests for GitHub CI on GB200. +DO NOT TRIGGER when: user is asking about H100 tests or making unrelated changes to existing test configs. +--- + +# Onboard GB200 1-Node GitHub MR Tests + +Create 1-node (`mr-github`) variants of existing 2-node (`mr`-scoped) GB200 functional tests. +Each GB200 node has **4 GPUs**. A 2-node test uses 8 GPUs total; the 1-node variant uses 4. + +--- + +## Background + +GB200 functional tests live in `tests/test_utils/recipes/gb200/`: + +| Recipe file | Notes | +|-------------|-------| +| `gpt.yaml` | GPT dense tests, `nodes: 2, gpus: 4` (8 total) | +| `moe.yaml` | MoE tests, `nodes: 2, gpus: 4` (8 total) | +| `moe-1node.yaml` | Existing 1-node MoE tests, `nodes: 1, gpus: 4` (4 total) | +| `gpt-1node.yaml` | 1-node GPT tests (create if not present) | + +Model configs live at: +`tests/functional_tests/test_cases/{model}/{test_case}/model_config.yaml` + +1-node test cases use the `_1node` suffix: +`tests/functional_tests/test_cases/{model}/{test_case}_1node/model_config.yaml` + +--- + +## Workflow + +### Step 1 — Find candidate tests + +Scan the `products:` block in `gpt.yaml` and `moe.yaml` for entries with `scope: [mr, ...]` or `scope: [mr-slim, ...]`. These are the 2-node tests that need 1-node `mr-github` counterparts. + +Ignore tests already covered in `*-1node.yaml` files, and ignore `nightly`, `weekly`, `mr-broken` scopes. + +### Step 2 — Read each model config + +For each candidate, read its `model_config.yaml` and extract the key parallelism arguments: + +``` +--tensor-model-parallel-size (TP) +--pipeline-model-parallel-size (PP) +--expert-model-parallel-size (EP) +--expert-tensor-parallel-size (ETP) +--context-parallel-size (CP) +--global-batch-size +--micro-batch-size +``` + +### Step 3 — Classify: trivial copy vs. needs adaptation + +The world size formula is: `world_size = TP × PP × DP` where `DP ≥ EP`. + +Going from 8 GPUs → 4 GPUs: + +| Condition | Action | +|-----------|--------| +| `TP × PP ≤ 4` | **Trivial copy.** Config unchanged; DP is halved automatically. | +| `TP × PP = 8` (e.g. tp4 pp2) | **Reduce PP.** Set `PP = PP / 2` (e.g. pp2→1). Verify `TP × PP_new ≤ 4`. | +| `EP > 4` (e.g. ep8 with tp1 pp1) | **Reduce EP.** Set `EP = 4`. Experts stay at `num-experts` (each EP rank holds more experts). | +| `EP > 4` **and** `TP × PP > 4` | Reduce both PP and EP as above. | +| ETP test (ep × etp ≤ TP × DP) | Check `EP × ETP ≤ TP × DP_new` after PP reduction. Usually satisfied when pp→1. | + +**Do not change GBS** — let gradient accumulation absorb the reduced DP. + +### Step 4 — Create `_1node` model config directories + +```bash +# Trivial copy +mkdir -p tests/functional_tests/test_cases/{model}/{test_case}_1node +cp tests/functional_tests/test_cases/{model}/{test_case}/model_config.yaml \ + tests/functional_tests/test_cases/{model}/{test_case}_1node/model_config.yaml + +# Then apply any parallelism changes (EP or PP) with Edit tool +``` + +### Step 5 — Create or update recipe files + +**For GPT tests** — create `tests/test_utils/recipes/gb200/gpt-1node.yaml` (if absent) by cloning `gpt.yaml`'s spec block with `nodes: 1`. Use this template for the spec: + +```yaml +type: basic +format_version: 1 +maintainers: [mcore] +loggers: [stdout] +spec: + name: "{test_case}_{environment}_{platforms}" + model: gpt # or moe + build: mcore-pyt-{environment} + nodes: 1 + gpus: 4 + n_repeat: 5 + platforms: dgx_gb200 + script_setup: | # copy verbatim from gpt.yaml / moe.yaml + ... + script: |- # copy verbatim from gpt.yaml / moe.yaml + ... +``` + +**For MoE tests** — append entries to the existing `moe-1node.yaml`. + +### Step 6 — Add products entries + +Scope convention: +- **1–2 most representative tests** per recipe: `scope: [mr-github, mr-github-slim]` +- **All other tests**: `scope: [mr-github]` + +```yaml +products: + - test_case: [_1node] + products: + - environment: [dev] + scope: [mr-github, mr-github-slim] # or [mr-github] + platforms: [dgx_gb200] +``` + +--- + +## Quick parallelism reference + +| Original (8 GPUs) | 1-node config (4 GPUs) | Notes | +|-------------------|----------------------|-------| +| tp1 pp1 ep1 → dp8 | tp1 pp1 ep1 → dp4 | trivial | +| tp2 pp1 ep1 → dp4 | tp2 pp1 ep1 → dp2 | trivial | +| tp1 pp2 ep1 → dp4 | tp1 pp2 ep1 → dp2 | trivial | +| tp4 pp1 ep1 → dp2 | tp4 pp1 ep1 → dp1 | trivial | +| tp1 pp4 ep1 → dp2 | tp1 pp4 ep1 → dp1 | trivial | +| tp1 pp1 ep8 → dp8 | tp1 pp1 ep4 → dp4 | ep 8→4 | +| tp4 pp2 ep2 etp2 → dp1 | tp4 pp1 ep2 etp2 → dp1 | pp 2→1 | + +--- + +## Checklist + +- [ ] Identified all `mr`-scoped tests in `gpt.yaml` and `moe.yaml` not yet in `*-1node.yaml` +- [ ] Read model config for each candidate +- [ ] Classified trivial vs. adaptation needed +- [ ] Created `_1node/model_config.yaml` for each test +- [ ] Applied EP or PP reductions where needed +- [ ] Created/updated recipe YAML with `nodes: 1, gpus: 4` +- [ ] Assigned `mr-github` scope (+ `mr-github-slim` for 1–2 representative tests per recipe) +- [ ] Verified no `mr-github-slim` overload (slim suite should stay small) diff --git a/skills/build-and-test/SKILL.md b/skills/testsystem/SKILL.md similarity index 51% rename from skills/build-and-test/SKILL.md rename to skills/testsystem/SKILL.md index 6bd823a42e1..3b3d6175915 100644 --- a/skills/build-and-test/SKILL.md +++ b/skills/testsystem/SKILL.md @@ -1,193 +1,24 @@ --- -name: build-and-test -description: Developer environment setup, CI/CD workflows, and CI failure debugging for Megatron-LM. Covers container-based development, uv package management, linting, running tests, CI failure investigation, and common pitfalls. -TRIGGER when: user asks to add, remove, or update a dependency; user edits or asks about pyproject.toml or uv.lock; user asks to set up a dev environment or run tests; user asks about a CI failure or build error. -DO NOT TRIGGER when: user is only reading or discussing code unrelated to dependencies, build, or CI. +name: testsystem +description: Test system, CI pipeline, and CI failure investigation for Megatron-LM. Covers test layout, recipe YAML structure, adding unit and functional tests, CI scope labels, triggering internal GitLab CI, pipeline structure, and debugging CI failures. +TRIGGER when: user asks to run tests, add a test, investigate a CI failure, understand the CI pipeline, or work with test recipes; user opens or pushes to a PR and needs to know which CI label to attach; user wants to trigger the internal GitLab CI pipeline; user asks to download golden values or references a pipeline/run ID in the context of golden values. +DO NOT TRIGGER when: user is only setting up the dev environment or managing dependencies (use build-and-dependency instead). --- -# Developer Guide - -This guide covers the recommended development workflow for Megatron-LM. -The core principle: **build and develop inside containers** — the CI container -ships the correct CUDA toolkit, PyTorch build, and pre-compiled native extensions -(TransformerEngine, DeepEP, …) that cannot be reproduced on a bare host. - ---- - -## Why Containers - -Megatron-LM depends on CUDA, NCCL, PyTorch with GPU support, TransformerEngine, -and optional components like ModelOpt and DeepEP. Installing these on a bare host -is fragile and hard to reproduce. The project ships Dockerfiles that pin every -dependency. - -**Use the container as your development environment.** This guarantees: - -- Identical CUDA / NCCL / cuDNN versions across all developers and CI. -- `uv.lock` resolves the same way locally and in CI. -- GPU-dependent operations (training, testing) work out of the box. - -### Step 1 — Acquire an Image - -**Option A — NVIDIA-internal: pull a CI-built image** - -> ⚠️ Requires access to the internal GitLab instance. -> See `tools/trigger_internal_ci.md` for setup (adding the git remote, obtaining a token). - -The internal GitLab CI publishes images to its container registry. -Derive the registry host from your configured `gitlab` remote — the same -host you use for `trigger_internal_ci.py`: - -```bash -# Derive host from your 'gitlab' remote: -GITLAB_HOST=$(git remote get-url gitlab | sed 's/.*@\(.*\):.*/\1/') - -docker pull ${GITLAB_HOST}/adlr/megatron-lm/mcore_ci_dev:main -``` - -**Option B — Build from scratch (works for everyone)** - -> ⚠️ `Dockerfile.ci.dev` has two stages: `main` and `jet`. The `jet` stage -> requires an internal build secret and will fail without it. Always pass -> `--target main` to stop at the public stage. - -```bash -# dev image (default) -docker build \ - --target main \ - --build-arg FROM_IMAGE_NAME=$(cat docker/.ngc_version.dev) \ - --build-arg IMAGE_TYPE=dev \ - -f docker/Dockerfile.ci.dev \ - -t megatron-lm:local . - -# lts image -docker build \ - --target main \ - --build-arg FROM_IMAGE_NAME=$(cat docker/.ngc_version.lts) \ - --build-arg IMAGE_TYPE=lts \ - -f docker/Dockerfile.ci.dev \ - -t megatron-lm:local-lts . -``` - -Which image variant is used is controlled by the PR label `container::lts`; -absent that label, `dev` is used. - -### Step 2 — Launch the Container - -**Option A — Local Docker runtime** - -```bash -docker run --rm --gpus all \ - -v $(pwd):/workspace \ - -w /workspace \ - megatron-lm:local \ - bash -c "" -``` - -**Option B — Slurm cluster (for those without a local Docker runtime)** - -NVIDIA clusters typically use [Pyxis](https://github.com/NVIDIA/pyxis) + -[enroot](https://github.com/NVIDIA/enroot). Request an interactive session: - -```bash -srun \ - --nodes=1 --gpus-per-node=8 \ - --container-image megatron-lm:local \ - --container-mounts $(pwd):/workspace \ - --container-workdir /workspace \ - --pty bash -``` - -For clusters that require a `.sqsh` archive first: - -```bash -enroot import -o megatron-lm.sqsh dockerd://megatron-lm:local -srun \ - --nodes=1 --gpus-per-node=8 \ - --container-image $(pwd)/megatron-lm.sqsh \ - --container-mounts $(pwd):/workspace \ - --container-workdir /workspace \ - --pty bash -``` +# Test System & CI Guide --- -## Dependency Management - -Dependencies are declared in `pyproject.toml`. The venv lives at `/opt/venv` -inside the container (already on `PATH`). - -> **All `uv` operations must be run inside the container.** -> Never run `uv sync` / `uv pip install` on the host. - -### uv Dependency Groups - -| Group | Purpose | -|-------|---------| -| `training` | Runtime training extras | -| `dev` | Full dev environment (TransformerEngine, ModelOpt, …) | -| `lts` | LTS-safe subset (no ModelOpt) | -| `test` | pytest, coverage, nemo-run | -| `linting` | ruff, black, isort, pylint | -| `build` | Cython, pybind11, nvidia-mathdx | - -Install commands (inside the container): - -```bash -# Full dev + test environment -uv sync --locked --group dev --group test - -# Linting only -uv sync --locked --only-group linting - -# LTS environment -uv sync --locked --group lts --group test -``` - -Several dependencies are sourced directly from git (TransformerEngine, nemo-run, -FlashMLA, Emerging-Optimizers, nvidia-resiliency-ext). The locked `uv.lock` file -pins exact revisions; update it with `uv lock` when changing `pyproject.toml`. - -### Adding a New Dependency - -Follow this three-step workflow: - -1. **Acquire a container image** — see [Step 1](#step-1--acquire-an-image) above. -2. **Launch the container interactively** — see [Step 2](#step-2--launch-the-container) above. -3. **Update the lock file inside the container**, then commit it: - - ```bash - # Inside the container: - uv add # adds to pyproject.toml and resolves - uv lock # regenerates uv.lock - # Exit the container, then on the host: - git add pyproject.toml uv.lock - git commit -S -s -m "build: add dependency" - ``` - ---- - -## Linting - -Run before opening a PR: - -```bash -# Check mode (no changes applied) -BASE_REF=main CHECK_ONLY=true SKIP_DOCS=false bash tools/autoformat.sh -``` - -Tools invoked: `black`, `isort`, `pylint`, `ruff`, `mypy`. - ---- - -## Running Tests - -### Test Layout +## Test Layout ```text tests/ ├── unit_tests/ # pytest, 1 node × 8 GPUs, torch.distributed runner ├── functional_tests/ # end-to-end shell + training scripts +│ └── test_cases/ +│ └── {model}/{test_case}/ +│ ├── model_config.yaml # training args +│ └── golden_values_{env}_{platform}.json └── test_utils/ ├── recipes/ │ ├── h100/ # YAML recipes for H100 jobs @@ -195,12 +26,13 @@ tests/ └── python_scripts/ # helpers (recipe_parser, golden-value download, …) ``` -### How Tests Execute +--- + +## How Tests Execute -All tests run on a **single DGX H100 node (8 GPUs)**. The GitHub Actions runner -invokes `launch_nemo_run_workload.py`, which uses **nemo-run** to launch a -`DockerExecutor` container. The repo is bind-mounted at `/opt/megatron-lm`; -training data is mounted at `/mnt/artifacts`. +The GitHub Actions runner invokes `launch_nemo_run_workload.py`, which uses +**nemo-run** to launch a `DockerExecutor` container. The repo is bind-mounted +at `/opt/megatron-lm`; training data is mounted at `/mnt/artifacts`. **Unit tests** are dispatched through `torch.distributed.run`: @@ -216,7 +48,9 @@ pytest validation step; training output from all ranks is uploaded as an artifac **3 times** for known transient patterns (NCCL timeout, ECC error, segfault, HuggingFace connectivity, …) before declaring a genuine failure. -### Recipe YAML Structure +--- + +## Recipe YAML Structure Recipes live in `tests/test_utils/recipes/` and are parsed by `tests/test_utils/python_scripts/recipe_parser.py`. Each file expands a @@ -225,34 +59,89 @@ cartesian `products` block into individual workload specs: ```yaml type: basic format_version: 1 +maintainers: [mcore] +loggers: [stdout] spec: - name: "{test_case}_{environment}_{platforms}_{tag}" - model: gpt + name: "{test_case}_{environment}_{platforms}" + model: gpt # maps to tests/functional_tests/test_cases/{model}/ + build: mcore-pyt-{environment} nodes: 1 gpus: 8 + n_repeat: 5 platforms: dgx_h100 time_limit: 1800 script_setup: | ... script: |- - bash tests/unit_tests/run_ci_test.sh \ - --tag {tag} \ - --environment {environment} \ - --bucket "tests/unit_tests/models/**/*.py" \ - --log-dir {assets_dir}/logs/1/ + bash tests/functional_tests/shell_test_utils/run_ci_test.sh ... products: - test_case: [my_test] - environment: [dev, lts] - tag: [latest, legacy] - scope: [mr-github] - n_repeat: [1] - time_limit: [1800] + products: + - environment: [dev, lts] + scope: [mr-github] + platforms: [dgx_h100] ``` Key runtime placeholders: `{assets_dir}`, `{artifacts_dir}`, `{test_case}`, -`{environment}`, `{platforms}`, `{tag}`, `{n_repeat}`. +`{environment}`, `{platforms}`, `{n_repeat}`. + +### CI Test Scope Labels + +The CI pipeline reads PR labels to decide test scope, n_repeat, and container image. + +**Decision tree (first match wins):** -### Adding a Unit Test +| Condition | `scope` | `n_repeat` | `lightweight` | Notes | +|-----------|---------|-----------|---------------|-------| +| Merge group | `mr-github` | 1 | false | Automatic, no label needed | +| Label: **`Run tests`** | `mr-github` | 1 | **true** | Trains 4 steps, no golden-value compare | +| Label: **`Run functional tests`** | `mr-github` | 5 | **false** | Trains 100 steps, golden-value compare | +| _(no label)_ | `mr-github-slim` | 5 | false | Slim subset only | + +**Orthogonal image label:** + +| Label | Effect | +|-------|--------| +| **`container::lts`** | Use the LTS base image instead of `dev` (combinable with any scope label) | +| **`Run MBridge tests`** | Also triggers the MBridge L1 test suite | + +### Disabling a Test Without Deleting It + +To temporarily disable a test case in a recipe YAML, suffix its `scope` value +with `-broken` — **do not delete the entry**: + +```yaml +# before (test runs in CI) +scope: [mr-github] + +# after (test is skipped; entry preserved for easy re-enable) +scope: [mr-github-broken] +``` + +This applies to any scope token (`mr-github`, `mr-github-slim`, `mr-gitlab`, +etc.). Deleting the entry entirely would require recreating the test case +definition when the fix lands. + +### Which label to attach when opening a PR + +Apply this logic based on what the PR changes: + +| Changed paths / nature of change | Label to attach | +|----------------------------------|-----------------| +| Docs only (`docs/`, `*.md`, docstrings) | _(none)_ | +| CI/tooling only (`.github/`, `tools/`, `Makefile`) | _(none)_ | +| Test files only (`tests/`) — existing tests, no new golden values | `Run tests` | +| **New test cases added** (no golden values exist yet) | `Run functional tests` | +| Non-numerical library code (logging, error handling, CLI flags, refactors) | `Run tests` | +| Could affect training numerics (model arch, attention, optimizer, distributed, MoE routing) | `Run functional tests` | +| Container or dependency changes (`docker/`, `pyproject.toml`, `uv.lock`) | `Run tests` + `container::lts` | +| Touches MBridge integration | add `Run MBridge tests` | + +**Rule of thumb:** default to `Run tests`. Always use `Run functional tests` when the PR adds new test cases (golden values must be generated) or when the change could plausibly shift loss curves. + +--- + +## Adding a Unit Test 1. Create `tests/unit_tests//test_.py`. 2. Use fixtures from `tests/unit_tests/conftest.py`. @@ -260,7 +149,7 @@ Key runtime placeholders: `{assets_dir}`, `{artifacts_dir}`, `{test_case}`, - `@pytest.mark.internal` — skipped on `legacy` tag - `@pytest.mark.flaky` — skipped in `lts` environment - `@pytest.mark.experimental` — `latest` tag only -4. Verify the test runs locally inside the container: +4. Verify locally inside the container: ```bash pytest -xvs tests/unit_tests//test_.py @@ -269,10 +158,12 @@ Key runtime placeholders: `{assets_dir}`, `{artifacts_dir}`, `{test_case}`, 5. If the test needs a dedicated CI bucket, add an entry to `tests/test_utils/recipes/h100/unit-tests.yaml`. -### Adding a Functional / Integration Test +--- + +## Adding a Functional / Integration Test 1. Create `tests/functional_tests/test_cases///`. -2. Write the shell test script; use `{assets_dir}` for all output paths. +2. Write `model_config.yaml` with `MODEL_ARGS`, `ENV_VARS`, and `TEST_TYPE`. 3. Add a YAML recipe under `tests/test_utils/recipes/h100/` (and `gb200/` if needed). Required fields: `scope`, `environment`, `platform`, `n_repeat`, `time_limit`. @@ -286,14 +177,38 @@ Key runtime placeholders: `{assets_dir}`, `{artifacts_dir}`, `{test_case}`, 6. Commit the downloaded golden values. -### CI Test Scope Labels +--- + +## Triggering Internal CI + +Use `tools/trigger_internal_ci.py` to push the current branch to the internal +GitLab remote and trigger a pipeline — without touching the GitLab UI. +Full setup and usage details: `tools/trigger_internal_ci.md`. + +**Prerequisites** (one-time): + +```bash +# 1. Add the internal GitLab remote +git remote add gitlab git@:ADLR/Megatron-LM.git + +# 2. Create a personal access token with 'api' scope on your GitLab profile, +# then store it: +export GITLAB_TOKEN=glpat- +``` + +**Usage:** + +```bash +python tools/trigger_internal_ci.py \ + --gitlab-origin gitlab \ + [--functional-test-scope mr] \ + [--functional-test-repeat 5] \ + [--functional-test-cases all] \ + [--dry-run] +``` -| PR label | Scope | Behaviour | -|----------|-------|-----------| -| _(none)_ | `mr-github-slim` | Lightweight subset, fast feedback | -| `Run tests` | `mr-github` | Full suite, lightweight mode (4 steps, no golden compare) | -| `Run functional tests` | `mr-github` | Full suite, 100-step training + golden compare, n_repeat=5 | -| `container::lts` | _(any)_ | Use the LTS base image instead of dev | +The script force-pushes the current branch as `pull-request/` and +prints the resulting pipeline URL. --- @@ -450,10 +365,7 @@ PR's changes or is a pre-existing issue on `main`. | Problem | Cause | Fix | |---------|-------|-----| -| `uv sync --locked` fails | Dependency conflict or stale `uv.lock` | Re-run `uv lock` inside the container and commit updated lock | -| `ModuleNotFoundError` after pip install | pip installed outside the uv-managed venv | Use `uv add` and `uv sync`, never bare `pip install` | -| `uv: command not found` inside container | Wrong container image | Use the `megatron-lm` image built from `Dockerfile.ci.dev` | -| `No space left on device` during uv ops | Cache fills container's `/root/.cache/` | Mount a host cache dir via `-v $HOME/.cache/uv:/root/.cache/uv` | -| Pre-commit fails with linting errors | Code style violations | Run `BASE_REF=main CHECK_ONLY=false bash tools/autoformat.sh` | | Port collision on multi-GPU runs | torchrun binding conflicts | Use `torch.distributed.run` via the container entry point | -| `docker build` fails with secret-related error | `Dockerfile.ci.dev` has a `jet` stage that requires an internal secret | Add `--target main` to stop before the `jet` stage | +| Test passes locally but fails in CI | Different environment or data path | Check `DATA_PATH`, `DATA_CACHE_PATH`, and the `environment` tag (`dev` vs `lts`) | +| Golden value mismatch after a code change | Numerical regression | Download new golden values via `download_golden_values.py` after a clean run | +| `cicd-integration-tests-gb200` not triggered | GB200 jobs require maintainer status | Ask a maintainer to trigger, or add the `Run functional tests` label | diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_reruns_persistent_1_1node/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_reruns_persistent_1_1node/model_config.yaml new file mode 100644 index 00000000000..daa5093f0a4 --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_reruns_persistent_1_1node/model_config.yaml @@ -0,0 +1,90 @@ +ENV_VARS: + SKIP_PYTEST: 1 + CUDA_DEVICE_MAX_CONNECTIONS: 1 + NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0 + NCCL_ALGO: Ring + CUBLAS_WORKSPACE_CONFIG: :4096:8 +MODEL_ARGS: + # generic training settings + --deterministic-mode: true + --bf16: true + --train-iters: 50 + --eval-iters: 0 + --manual-gc: true + --use-mcore-models: true + --distributed-backend: nccl + # parallelism settings + --sequence-parallel: true + --tensor-model-parallel-size: 1 + --pipeline-model-parallel-size: 2 + --micro-batch-size: 1 + --global-batch-size: 128 + # embedding settings + --untie-embeddings-and-output-weights: true + --position-embedding-type: rope + --rotary-percent: 1.0 + --max-position-embeddings: 4096 + # transformer settings + --num-layers: 32 + --hidden-size: 3072 + --ffn-hidden-size: 8192 + --num-attention-heads: 32 + --num-query-groups: 8 + --seq-length: 4096 + --kv-channels: 128 + --ffn-hidden-size: 8192 + --group-query-attention: true + --normalization: RMSNorm + --swiglu: true + --attention-dropout: 0.0 + --hidden-dropout: 0.0 + --no-create-attention-mask-in-dataloader: true + --transformer-impl: transformer_engine + --disable-bias-linear: true + # gradient & optimizer settings + --clip-grad: 1.0 + --overlap-grad-reduce: true + --overlap-param-gather: true + --lr: 3e-4 + --lr-warmup-samples: 0 + --adam-beta1: 0.9 + --adam-beta2: 0.95 + --adam-eps: 1e-8 + --use-distributed-optimizer: true + --split: 949,50,1 + --no-gradient-accumulation-fusion: true + # checkpoint settings + --save-interval: 10000 + --eval-interval: 1000 + --ckpt-format: torch_dist + --dist-ckpt-strictness: log_all # backward compatibility for TE changes + --save: ${CHECKPOINT_SAVE_PATH} + --load: ${CHECKPOINT_LOAD_PATH} + # data settings + --data-cache-path: ${DATA_CACHE_PATH} + --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document + --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json + --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt + # logging settings + --tensorboard-dir: ${TENSORBOARD_PATH} + --timing-log-level: 0 + --log-interval: 1 + --log-params-norm: true + --log-num-zeros-in-grad: true + --log-validation-ppl-to-tensorboard: true + --log-timers-to-tensorboard: true + --log-memory-to-tensorboard: true + # rerun settings + --rerun-mode: validate_results + --error-injection-rate: 100 + --error-injection-type: persistent_error + --async-save: true + --use-persistent-ckpt-worker: true +AFTER_SCRIPT: | + check_log() { if [[ -z $(grep -r $1 "$2" $LOG_DIR) ]]; then exit 1; else echo OK; fi } + check_log -F "Result validation enabled" + check_log -F "Injecting error type Persistent error" + check_log -F "First rerun: unexpected result is reproducible within the tolerance" + check_log -F "Saving a checkpoint and exiting now. Please resume the job from the checkpoint to rerun the last iteration and establish a diagnostic" + EXIT_CODE=0 +TEST_TYPE: regular diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_dist_optimizer_fim_dataset_1node/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_dist_optimizer_fim_dataset_1node/golden_values_dev_dgx_gb200.json new file mode 100644 index 00000000000..4a618a0422f --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_dist_optimizer_fim_dataset_1node/golden_values_dev_dgx_gb200.json @@ -0,0 +1,287 @@ +{ + "lm loss": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 10.88433, + "2": 10.88455, + "3": 10.88309, + "4": 10.88497, + "5": 10.87886, + "6": 10.87416, + "7": 10.88029, + "8": 10.87382, + "9": 10.88138, + "10": 10.88114, + "11": 10.87549, + "12": 10.86262, + "13": 10.87418, + "14": 10.86552, + "15": 10.83746, + "16": 10.83285, + "17": 10.83752, + "18": 10.82158, + "19": 10.83534, + "20": 10.73039, + "21": 10.73845, + "22": 10.72167, + "23": 10.71282, + "24": 10.68134, + "25": 10.67711, + "26": 10.67859, + "27": 10.63636, + "28": 10.57712, + "29": 10.54434, + "30": 10.52198, + "31": 10.51451, + "32": 10.49261, + "33": 10.47084, + "34": 10.43254, + "35": 10.44114, + "36": 10.41271, + "37": 10.37829, + "38": 10.39605, + "39": 10.35296, + "40": 10.34738, + "41": 10.32554, + "42": 10.29871, + "43": 10.2881, + "44": 10.24608, + "45": 10.27178, + "46": 10.22622, + "47": 10.22132, + "48": 10.17358, + "49": 10.17262, + "50": 10.17363 + } + }, + "num-zeros": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 1762.0, + "2": 1751.0, + "3": 1680.0, + "4": 1694.0, + "5": 1780.0, + "6": 1620.0, + "7": 1878.0, + "8": 1712.0, + "9": 1784.0, + "10": 1813.0, + "11": 1724.0, + "12": 1774.0, + "13": 1798.0, + "14": 1902.0, + "15": 1579.0, + "16": 1665.0, + "17": 1839.0, + "18": 1751.0, + "19": 1684.0, + "20": 1696.0, + "21": 1813.0, + "22": 1721.0, + "23": 1705.0, + "24": 1795.0, + "25": 1699.0, + "26": 1767.0, + "27": 1747.0, + "28": 1871.0, + "29": 1847.0, + "30": 1813.0, + "31": 2027.0, + "32": 2003.0, + "33": 2032.0, + "34": 2058.0, + "35": 2136.0, + "36": 2084.0, + "37": 2158.0, + "38": 2049.0, + "39": 2259.0, + "40": 2277.0, + "41": 2445.0, + "42": 2144.0, + "43": 2428.0, + "44": 2234.0, + "45": 2590.0, + "46": 2394.0, + "47": 2567.0, + "48": 2671.0, + "49": 2833.0, + "50": 2640.0 + } + }, + "mem-allocated-bytes": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 678556160.0, + "2": 678556160.0, + "3": 678556160.0, + "4": 678556160.0, + "5": 678556160.0, + "6": 678556160.0, + "7": 678556160.0, + "8": 678556160.0, + "9": 678556160.0, + "10": 678556160.0, + "11": 678556160.0, + "12": 678556160.0, + "13": 678556160.0, + "14": 678556160.0, + "15": 678556160.0, + "16": 678556160.0, + "17": 678556160.0, + "18": 678556160.0, + "19": 678556160.0, + "20": 678556160.0, + "21": 678556160.0, + "22": 678556160.0, + "23": 678556160.0, + "24": 678556160.0, + "25": 678556160.0, + "26": 678556160.0, + "27": 678556160.0, + "28": 678556160.0, + "29": 678556160.0, + "30": 678556160.0, + "31": 678556160.0, + "32": 678556160.0, + "33": 678556160.0, + "34": 678556160.0, + "35": 678556160.0, + "36": 678556160.0, + "37": 678556160.0, + "38": 678556160.0, + "39": 678556160.0, + "40": 678556160.0, + "41": 678556160.0, + "42": 678556160.0, + "43": 678556160.0, + "44": 678556160.0, + "45": 678556160.0, + "46": 678556160.0, + "47": 678556160.0, + "48": 678556160.0, + "49": 678556160.0, + "50": 678556160.0 + } + }, + "mem-max-allocated-bytes": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 4671278592.0, + "2": 4801303040.0, + "3": 4801303040.0, + "4": 4801303040.0, + "5": 4801303040.0, + "6": 4801303040.0, + "7": 4801303040.0, + "8": 4801303040.0, + "9": 4801303040.0, + "10": 4801303040.0, + "11": 4801303040.0, + "12": 4801303040.0, + "13": 4801303040.0, + "14": 4801303040.0, + "15": 4801303040.0, + "16": 4801303040.0, + "17": 4801303040.0, + "18": 4801303040.0, + "19": 4801303040.0, + "20": 4801303040.0, + "21": 4801303040.0, + "22": 4801303040.0, + "23": 4801303040.0, + "24": 4801303040.0, + "25": 4801303040.0, + "26": 4801303040.0, + "27": 4801303040.0, + "28": 4801303040.0, + "29": 4801303040.0, + "30": 4801303040.0, + "31": 4801303040.0, + "32": 4801303040.0, + "33": 4801303040.0, + "34": 4801303040.0, + "35": 4801303040.0, + "36": 4801303040.0, + "37": 4801303040.0, + "38": 4801303040.0, + "39": 4801303040.0, + "40": 4801303040.0, + "41": 4801303040.0, + "42": 4801303040.0, + "43": 4801303040.0, + "44": 4801303040.0, + "45": 4801303040.0, + "46": 4801303040.0, + "47": 4801303040.0, + "48": 4801303040.0, + "49": 4801303040.0, + "50": 4801303040.0 + } + }, + "iteration-time": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": "nan", + "2": 3.6133, + "3": 0.17103, + "4": 0.49157, + "5": 0.40377, + "6": 0.59575, + "7": 0.77479, + "8": 0.87225, + "9": 0.57554, + "10": 0.30962, + "11": 0.91119, + "12": 0.26573, + "13": 0.65816, + "14": 0.47101, + "15": 0.45011, + "16": 0.53162, + "17": 0.46989, + "18": 0.78826, + "19": 0.37436, + "20": 0.42871, + "21": 0.66093, + "22": 0.50714, + "23": 0.56508, + "24": 0.41589, + "25": 0.44038, + "26": 0.80327, + "27": 0.16697, + "28": 0.53239, + "29": 0.67375, + "30": 0.1689, + "31": 0.16802, + "32": 0.98804, + "33": 0.50687, + "34": 0.23032, + "35": 0.68479, + "36": 0.41224, + "37": 0.5696, + "38": 0.40419, + "39": 0.5839, + "40": 0.68095, + "41": 0.52006, + "42": 0.43584, + "43": 0.47061, + "44": 0.55601, + "45": 0.56503, + "46": 0.41824, + "47": 0.49422, + "48": 0.70723, + "49": 0.4008, + "50": 0.45645 + } + } +} \ No newline at end of file diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_dist_optimizer_fim_dataset_1node/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_dist_optimizer_fim_dataset_1node/model_config.yaml new file mode 100644 index 00000000000..86039c4d7f2 --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_dist_optimizer_fim_dataset_1node/model_config.yaml @@ -0,0 +1,58 @@ +ENV_VARS: + CUDA_DEVICE_MAX_CONNECTIONS: 1 + NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0 + NCCL_ALGO: Ring + CUBLAS_WORKSPACE_CONFIG: :4096:8 +MODEL_ARGS: + --num-layers: 12 + --hidden-size: 512 + --num-attention-heads: 8 + --log-params-norm: true + --log-num-zeros-in-grad: true + --log-validation-ppl-to-tensorboard: true + --log-timers-to-tensorboard: true + --tensorboard-dir: ${TENSORBOARD_PATH} + --micro-batch-size: 4 + --global-batch-size: 32 + --seq-length: 1024 + --max-position-embeddings: 1024 + --train-iters: 50 + --timing-log-level: 0 + --lr-decay-iters: 320000 + --save: ${CHECKPOINT_SAVE_PATH} + --load: ${CHECKPOINT_LOAD_PATH} + --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document + --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json + --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt + --split: 949,50,1 + --distributed-backend: nccl + --lr: 0.00015 + --lr-decay-style: cosine + --min-lr: 1.0e-5 + --weight-decay: 1e-2 + --clip-grad: 1.0 + --lr-warmup-fraction: .01 + --log-interval: 1 + --save-interval: 10000 + --eval-interval: 1000 + --eval-iters: 10 + --transformer-impl: transformer_engine + --tensor-model-parallel-size: 1 + --pipeline-model-parallel-size: 1 + --use-distributed-optimizer: true + --deterministic-mode: true + --no-gradient-accumulation-fusion: true + --attention-softmax-in-fp32: true + --use-mcore-models: true + --ckpt-format: torch_dist + --dist-ckpt-strictness: log_all # backward compatibility for TE changes + --data-cache-path: ${DATA_CACHE_PATH} + --bf16: true + --attention-backend: unfused + --log-memory-to-tensorboard: true + --fim-data: true + --fim-rate: 0.5 + --fim-spm-rate: 0.5 + --async-save: true + --use-persistent-ckpt-worker: true +TEST_TYPE: regular diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_dist_optimizer_no_mmap_bin_files/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_dist_optimizer_no_mmap_bin_files/golden_values_dev_dgx_h100.json index 9497508565b..a99191f0165 100644 --- a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_dist_optimizer_no_mmap_bin_files/golden_values_dev_dgx_h100.json +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_dist_optimizer_no_mmap_bin_files/golden_values_dev_dgx_h100.json @@ -233,55 +233,55 @@ "step_interval": 1, "values": { "1": "nan", - "2": 3.79403, - "3": 0.07704, - "4": 0.05712, - "5": 0.05704, - "6": 0.0572, - "7": 0.05716, - "8": 0.05679, - "9": 0.05811, - "10": 0.05723, - "11": 0.05946, - "12": 0.0569, - "13": 0.05852, - "14": 0.05673, - "15": 0.0581, - "16": 0.05666, - "17": 0.0583, - "18": 0.05729, - "19": 0.05668, - "20": 0.056, - "21": 0.05597, - "22": 0.05598, - "23": 0.05712, - "24": 0.05623, - "25": 0.05687, - "26": 0.05583, - "27": 0.05644, - "28": 0.05633, - "29": 0.05634, - "30": 0.0561, - "31": 0.05593, - "32": 0.057, - "33": 0.05638, - "34": 0.05614, - "35": 0.05694, - "36": 0.05689, - "37": 0.05635, - "38": 0.05618, - "39": 0.05595, - "40": 0.05662, - "41": 0.05643, - "42": 0.05633, - "43": 0.05599, - "44": 0.05608, - "45": 0.05694, - "46": 0.05621, - "47": 0.05686, - "48": 0.05634, - "49": 0.05628, - "50": 0.05719 + "2": 2.00661, + "3": 0.08261, + "4": 0.0834, + "5": 0.08137, + "6": 0.08519, + "7": 0.08268, + "8": 0.08214, + "9": 0.07997, + "10": 0.08048, + "11": 0.07925, + "12": 0.08168, + "13": 0.08132, + "14": 0.08227, + "15": 0.08134, + "16": 0.08172, + "17": 0.07891, + "18": 0.08025, + "19": 0.07953, + "20": 0.07947, + "21": 0.08003, + "22": 0.07794, + "23": 0.08017, + "24": 0.08288, + "25": 0.08019, + "26": 0.08199, + "27": 0.07847, + "28": 0.08114, + "29": 0.08002, + "30": 0.08133, + "31": 0.07915, + "32": 0.081, + "33": 0.07866, + "34": 0.07863, + "35": 0.07899, + "36": 0.07772, + "37": 0.07875, + "38": 0.07884, + "39": 0.07928, + "40": 0.0791, + "41": 0.07779, + "42": 0.07865, + "43": 0.0781, + "44": 0.07786, + "45": 0.07858, + "46": 0.07875, + "47": 0.07855, + "48": 0.07924, + "49": 0.07823, + "50": 0.08055 } } } \ No newline at end of file diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_dist_optimizer_no_mmap_bin_files_1node/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_dist_optimizer_no_mmap_bin_files_1node/golden_values_dev_dgx_gb200.json new file mode 100644 index 00000000000..e60909f700d --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_dist_optimizer_no_mmap_bin_files_1node/golden_values_dev_dgx_gb200.json @@ -0,0 +1,287 @@ +{ + "lm loss": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 10.88533, + "2": 10.88323, + "3": 10.88389, + "4": 10.88685, + "5": 10.8789, + "6": 10.87638, + "7": 10.8797, + "8": 10.87233, + "9": 10.88488, + "10": 10.87864, + "11": 10.87362, + "12": 10.86361, + "13": 10.87879, + "14": 10.865, + "15": 10.83819, + "16": 10.83255, + "17": 10.84053, + "18": 10.82214, + "19": 10.83598, + "20": 10.72905, + "21": 10.73896, + "22": 10.72498, + "23": 10.71146, + "24": 10.68267, + "25": 10.67564, + "26": 10.67938, + "27": 10.63466, + "28": 10.57945, + "29": 10.54432, + "30": 10.52102, + "31": 10.5136, + "32": 10.49426, + "33": 10.46836, + "34": 10.43371, + "35": 10.43863, + "36": 10.41292, + "37": 10.37869, + "38": 10.39311, + "39": 10.35118, + "40": 10.34578, + "41": 10.32306, + "42": 10.29734, + "43": 10.28519, + "44": 10.24436, + "45": 10.27149, + "46": 10.22645, + "47": 10.22022, + "48": 10.1732, + "49": 10.17067, + "50": 10.17205 + } + }, + "num-zeros": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 1752.0, + "2": 1759.0, + "3": 1797.0, + "4": 1699.0, + "5": 1723.0, + "6": 1652.0, + "7": 1826.0, + "8": 1721.0, + "9": 1738.0, + "10": 1755.0, + "11": 1713.0, + "12": 1633.0, + "13": 1838.0, + "14": 1805.0, + "15": 1624.0, + "16": 1772.0, + "17": 1803.0, + "18": 1885.0, + "19": 1669.0, + "20": 1740.0, + "21": 1750.0, + "22": 1711.0, + "23": 1656.0, + "24": 1801.0, + "25": 1665.0, + "26": 1869.0, + "27": 1901.0, + "28": 1874.0, + "29": 1934.0, + "30": 1835.0, + "31": 2054.0, + "32": 1967.0, + "33": 1859.0, + "34": 1998.0, + "35": 2070.0, + "36": 2010.0, + "37": 2252.0, + "38": 2135.0, + "39": 2189.0, + "40": 2277.0, + "41": 2394.0, + "42": 2163.0, + "43": 2421.0, + "44": 2333.0, + "45": 2614.0, + "46": 2537.0, + "47": 2494.0, + "48": 2705.0, + "49": 2869.0, + "50": 2689.0 + } + }, + "mem-allocated-bytes": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 678556160.0, + "2": 678556160.0, + "3": 678556160.0, + "4": 678556160.0, + "5": 678556160.0, + "6": 678556160.0, + "7": 678556160.0, + "8": 678556160.0, + "9": 678556160.0, + "10": 678556160.0, + "11": 678556160.0, + "12": 678556160.0, + "13": 678556160.0, + "14": 678556160.0, + "15": 678556160.0, + "16": 678556160.0, + "17": 678556160.0, + "18": 678556160.0, + "19": 678556160.0, + "20": 678556160.0, + "21": 678556160.0, + "22": 678556160.0, + "23": 678556160.0, + "24": 678556160.0, + "25": 678556160.0, + "26": 678556160.0, + "27": 678556160.0, + "28": 678556160.0, + "29": 678556160.0, + "30": 678556160.0, + "31": 678556160.0, + "32": 678556160.0, + "33": 678556160.0, + "34": 678556160.0, + "35": 678556160.0, + "36": 678556160.0, + "37": 678556160.0, + "38": 678556160.0, + "39": 678556160.0, + "40": 678556160.0, + "41": 678556160.0, + "42": 678556160.0, + "43": 678556160.0, + "44": 678556160.0, + "45": 678556160.0, + "46": 678556160.0, + "47": 678556160.0, + "48": 678556160.0, + "49": 678556160.0, + "50": 678556160.0 + } + }, + "mem-max-allocated-bytes": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 4671278592.0, + "2": 4801303040.0, + "3": 4801303040.0, + "4": 4801303040.0, + "5": 4801303040.0, + "6": 4801303040.0, + "7": 4801303040.0, + "8": 4801303040.0, + "9": 4801303040.0, + "10": 4801303040.0, + "11": 4801303040.0, + "12": 4801303040.0, + "13": 4801303040.0, + "14": 4801303040.0, + "15": 4801303040.0, + "16": 4801303040.0, + "17": 4801303040.0, + "18": 4801303040.0, + "19": 4801303040.0, + "20": 4801303040.0, + "21": 4801303040.0, + "22": 4801303040.0, + "23": 4801303040.0, + "24": 4801303040.0, + "25": 4801303040.0, + "26": 4801303040.0, + "27": 4801303040.0, + "28": 4801303040.0, + "29": 4801303040.0, + "30": 4801303040.0, + "31": 4801303040.0, + "32": 4801303040.0, + "33": 4801303040.0, + "34": 4801303040.0, + "35": 4801303040.0, + "36": 4801303040.0, + "37": 4801303040.0, + "38": 4801303040.0, + "39": 4801303040.0, + "40": 4801303040.0, + "41": 4801303040.0, + "42": 4801303040.0, + "43": 4801303040.0, + "44": 4801303040.0, + "45": 4801303040.0, + "46": 4801303040.0, + "47": 4801303040.0, + "48": 4801303040.0, + "49": 4801303040.0, + "50": 4801303040.0 + } + }, + "iteration-time": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": "nan", + "2": 2.84568, + "3": 0.15652, + "4": 0.48767, + "5": 0.34488, + "6": 0.71772, + "7": 0.2477, + "8": 0.34728, + "9": 0.52259, + "10": 0.40927, + "11": 0.28366, + "12": 0.60837, + "13": 0.36099, + "14": 0.32234, + "15": 0.49702, + "16": 0.51084, + "17": 0.38911, + "18": 0.72421, + "19": 0.44678, + "20": 0.34575, + "21": 0.60318, + "22": 0.40983, + "23": 0.23078, + "24": 0.58724, + "25": 0.2653, + "26": 0.51071, + "27": 0.39903, + "28": 0.39811, + "29": 0.28492, + "30": 0.42322, + "31": 0.34464, + "32": 0.2598, + "33": 0.57199, + "34": 0.24442, + "35": 0.55219, + "36": 0.47258, + "37": 0.35639, + "38": 0.33421, + "39": 0.37645, + "40": 0.42537, + "41": 0.37631, + "42": 0.47084, + "43": 0.41687, + "44": 0.656, + "45": 0.31775, + "46": 0.25806, + "47": 0.44207, + "48": 0.6167, + "49": 0.28392, + "50": 0.40466 + } + } +} \ No newline at end of file diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_dist_optimizer_no_mmap_bin_files_1node/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_dist_optimizer_no_mmap_bin_files_1node/model_config.yaml new file mode 100644 index 00000000000..6a90ba0f943 --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_dist_optimizer_no_mmap_bin_files_1node/model_config.yaml @@ -0,0 +1,56 @@ +ENV_VARS: + CUDA_DEVICE_MAX_CONNECTIONS: 1 + NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0 + NCCL_ALGO: Ring + CUBLAS_WORKSPACE_CONFIG: :4096:8 +MODEL_ARGS: + --num-layers: 12 + --hidden-size: 512 + --num-attention-heads: 8 + --log-params-norm: true + --log-num-zeros-in-grad: true + --log-validation-ppl-to-tensorboard: true + --log-timers-to-tensorboard: true + --tensorboard-dir: ${TENSORBOARD_PATH} + --micro-batch-size: 4 + --global-batch-size: 32 + --seq-length: 1024 + --max-position-embeddings: 1024 + --train-iters: 50 + --timing-log-level: 0 + --lr-decay-iters: 320000 + --save: ${CHECKPOINT_SAVE_PATH} + --load: ${CHECKPOINT_LOAD_PATH} + --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document + --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json + --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt + --split: 949,50,1 + --distributed-backend: nccl + --lr: 0.00015 + --lr-decay-style: cosine + --min-lr: 1.0e-5 + --weight-decay: 1e-2 + --clip-grad: 1.0 + --lr-warmup-fraction: .01 + --log-interval: 1 + --save-interval: 10000 + --eval-interval: 1000 + --eval-iters: 10 + --transformer-impl: transformer_engine + --tensor-model-parallel-size: 1 + --pipeline-model-parallel-size: 1 + --use-distributed-optimizer: true + --no-mmap-bin-files: true + --deterministic-mode: true + --no-gradient-accumulation-fusion: true + --attention-softmax-in-fp32: true + --use-mcore-models: true + --ckpt-format: torch_dist + --dist-ckpt-strictness: log_all # backward compatibility for TE changes + --data-cache-path: ${DATA_CACHE_PATH} + --bf16: true + --attention-backend: unfused + --log-memory-to-tensorboard: true + --async-save: true + --use-persistent-ckpt-worker: true +TEST_TYPE: regular diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_resume_torch_dist_dist_optimizer_1node/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_resume_torch_dist_dist_optimizer_1node/golden_values_dev_dgx_gb200.json new file mode 100644 index 00000000000..2307d5176ee --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_resume_torch_dist_dist_optimizer_1node/golden_values_dev_dgx_gb200.json @@ -0,0 +1,537 @@ +{ + "lm loss": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 10.8851, + "2": 10.88271, + "3": 10.88387, + "4": 10.88623, + "5": 10.87916, + "6": 10.87659, + "7": 10.87962, + "8": 10.87212, + "9": 10.88437, + "10": 10.87861, + "11": 10.87351, + "12": 10.86322, + "13": 10.87875, + "14": 10.86597, + "15": 10.83786, + "16": 10.83261, + "17": 10.84042, + "18": 10.82308, + "19": 10.8357, + "20": 10.72914, + "21": 10.73909, + "22": 10.72511, + "23": 10.71158, + "24": 10.68285, + "25": 10.67542, + "26": 10.67984, + "27": 10.63495, + "28": 10.57987, + "29": 10.54429, + "30": 10.52094, + "31": 10.51407, + "32": 10.49438, + "33": 10.46828, + "34": 10.43398, + "35": 10.43852, + "36": 10.4131, + "37": 10.3791, + "38": 10.39374, + "39": 10.35128, + "40": 10.34595, + "41": 10.32312, + "42": 10.29759, + "43": 10.28543, + "44": 10.24463, + "45": 10.27145, + "46": 10.22653, + "47": 10.22023, + "48": 10.17334, + "49": 10.17076, + "50": 10.17217, + "51": 10.1777, + "52": 10.13082, + "53": 10.1398, + "54": 10.10412, + "55": 10.0701, + "56": 10.10629, + "57": 10.09822, + "58": 10.1116, + "59": 10.0548, + "60": 10.07318, + "61": 10.02711, + "62": 9.99676, + "63": 10.07084, + "64": 10.02892, + "65": 10.00122, + "66": 10.02707, + "67": 10.00255, + "68": 9.96484, + "69": 9.98873, + "70": 9.97572, + "71": 9.9981, + "72": 9.97566, + "73": 9.96388, + "74": 9.95687, + "75": 9.92479, + "76": 9.9637, + "77": 9.95559, + "78": 9.90277, + "79": 9.90952, + "80": 9.92737, + "81": 9.94386, + "82": 9.89003, + "83": 9.84869, + "84": 9.78551, + "85": 9.77693, + "86": 9.87452, + "87": 9.90795, + "88": 9.88609, + "89": 9.81901, + "90": 9.81049, + "91": 9.82156, + "92": 9.81513, + "93": 9.7511, + "94": 9.82585, + "95": 9.82052, + "96": 9.80342, + "97": 9.74064, + "98": 9.77418, + "99": 9.81987, + "100": 9.71431 + } + }, + "num-zeros": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 1709.0, + "2": 1738.0, + "3": 1755.0, + "4": 1816.0, + "5": 1708.0, + "6": 1634.0, + "7": 1838.0, + "8": 1695.0, + "9": 1767.0, + "10": 1770.0, + "11": 1646.0, + "12": 1694.0, + "13": 1827.0, + "14": 1836.0, + "15": 1668.0, + "16": 1771.0, + "17": 1710.0, + "18": 1810.0, + "19": 1626.0, + "20": 1637.0, + "21": 1755.0, + "22": 1610.0, + "23": 1699.0, + "24": 1736.0, + "25": 1665.0, + "26": 1825.0, + "27": 1873.0, + "28": 1873.0, + "29": 1950.0, + "30": 1801.0, + "31": 2007.0, + "32": 1957.0, + "33": 1989.0, + "34": 2056.0, + "35": 1989.0, + "36": 1986.0, + "37": 2275.0, + "38": 2141.0, + "39": 2165.0, + "40": 2317.0, + "41": 2404.0, + "42": 2080.0, + "43": 2426.0, + "44": 2321.0, + "45": 2630.0, + "46": 2483.0, + "47": 2580.0, + "48": 2699.0, + "49": 2889.0, + "50": 2667.0, + "51": 2617.0, + "52": 2850.0, + "53": 2625.0, + "54": 3028.0, + "55": 2634.0, + "56": 2766.0, + "57": 2237.0, + "58": 3612.0, + "59": 2973.0, + "60": 2999.0, + "61": 2852.0, + "62": 3267.0, + "63": 3413.0, + "64": 3691.0, + "65": 2757.0, + "66": 3276.0, + "67": 3939.0, + "68": 3751.0, + "69": 3041.0, + "70": 3416.0, + "71": 3101.0, + "72": 3182.0, + "73": 3409.0, + "74": 3380.0, + "75": 3196.0, + "76": 3354.0, + "77": 3776.0, + "78": 3164.0, + "79": 3305.0, + "80": 2976.0, + "81": 3507.0, + "82": 3083.0, + "83": 3097.0, + "84": 3179.0, + "85": 2726.0, + "86": 3155.0, + "87": 2915.0, + "88": 2976.0, + "89": 2918.0, + "90": 3555.0, + "91": 3025.0, + "92": 3105.0, + "93": 3130.0, + "94": 2975.0, + "95": 3522.0, + "96": 3262.0, + "97": 3671.0, + "98": 3711.0, + "99": 3330.0, + "100": 3251.0 + } + }, + "mem-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 678556160.0, + "2": 678556160.0, + "3": 678556160.0, + "4": 678556160.0, + "5": 678556160.0, + "6": 678556160.0, + "7": 678556160.0, + "8": 678556160.0, + "9": 678556160.0, + "10": 678556160.0, + "11": 678556160.0, + "12": 678556160.0, + "13": 678556160.0, + "14": 678556160.0, + "15": 678556160.0, + "16": 678556160.0, + "17": 678556160.0, + "18": 678556160.0, + "19": 678556160.0, + "20": 678556160.0, + "21": 678556160.0, + "22": 678556160.0, + "23": 678556160.0, + "24": 678556160.0, + "25": 678556160.0, + "26": 678556160.0, + "27": 678556160.0, + "28": 678556160.0, + "29": 678556160.0, + "30": 678556160.0, + "31": 678556160.0, + "32": 678556160.0, + "33": 678556160.0, + "34": 678556160.0, + "35": 678556160.0, + "36": 678556160.0, + "37": 678556160.0, + "38": 678556160.0, + "39": 678556160.0, + "40": 678556160.0, + "41": 678556160.0, + "42": 678556160.0, + "43": 678556160.0, + "44": 678556160.0, + "45": 678556160.0, + "46": 678556160.0, + "47": 678556160.0, + "48": 678556160.0, + "49": 678556160.0, + "50": 678556160.0, + "51": 678556160.0, + "52": 678556160.0, + "53": 678556160.0, + "54": 678556160.0, + "55": 678556160.0, + "56": 678556160.0, + "57": 678556160.0, + "58": 678556160.0, + "59": 678556160.0, + "60": 678556160.0, + "61": 678556160.0, + "62": 678556160.0, + "63": 678556160.0, + "64": 678556160.0, + "65": 678556160.0, + "66": 678556160.0, + "67": 678556160.0, + "68": 678556160.0, + "69": 678556160.0, + "70": 678556160.0, + "71": 678556160.0, + "72": 678556160.0, + "73": 678556160.0, + "74": 678556160.0, + "75": 678556160.0, + "76": 678556160.0, + "77": 678556160.0, + "78": 678556160.0, + "79": 678556160.0, + "80": 678556160.0, + "81": 678556160.0, + "82": 678556160.0, + "83": 678556160.0, + "84": 678556160.0, + "85": 678556160.0, + "86": 678556160.0, + "87": 678556160.0, + "88": 678556160.0, + "89": 678556160.0, + "90": 678556160.0, + "91": 678556160.0, + "92": 678556160.0, + "93": 678556160.0, + "94": 678556160.0, + "95": 678556160.0, + "96": 678556160.0, + "97": 678556160.0, + "98": 678556160.0, + "99": 678556160.0, + "100": 678556160.0 + } + }, + "mem-max-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 2710971904.0, + "2": 2838143488.0, + "3": 2838143488.0, + "4": 2838143488.0, + "5": 2838143488.0, + "6": 2838143488.0, + "7": 2838143488.0, + "8": 2838143488.0, + "9": 2838143488.0, + "10": 2838143488.0, + "11": 2838143488.0, + "12": 2838143488.0, + "13": 2838143488.0, + "14": 2838143488.0, + "15": 2838143488.0, + "16": 2838143488.0, + "17": 2838143488.0, + "18": 2838143488.0, + "19": 2838143488.0, + "20": 2838143488.0, + "21": 2838143488.0, + "22": 2838143488.0, + "23": 2838143488.0, + "24": 2838143488.0, + "25": 2838143488.0, + "26": 2838143488.0, + "27": 2838143488.0, + "28": 2838143488.0, + "29": 2838143488.0, + "30": 2838143488.0, + "31": 2838143488.0, + "32": 2838143488.0, + "33": 2838143488.0, + "34": 2838143488.0, + "35": 2838143488.0, + "36": 2838143488.0, + "37": 2838143488.0, + "38": 2838143488.0, + "39": 2838143488.0, + "40": 2838143488.0, + "41": 2838143488.0, + "42": 2838143488.0, + "43": 2838143488.0, + "44": 2838143488.0, + "45": 2838143488.0, + "46": 2838143488.0, + "47": 2838143488.0, + "48": 2838143488.0, + "49": 2838143488.0, + "50": 2838143488.0, + "51": 2839190528.0, + "52": 2839190528.0, + "53": 2839190528.0, + "54": 2839190528.0, + "55": 2839190528.0, + "56": 2839190528.0, + "57": 2839190528.0, + "58": 2839190528.0, + "59": 2839190528.0, + "60": 2839190528.0, + "61": 2839190528.0, + "62": 2839190528.0, + "63": 2839190528.0, + "64": 2839190528.0, + "65": 2839190528.0, + "66": 2839190528.0, + "67": 2839190528.0, + "68": 2839190528.0, + "69": 2839190528.0, + "70": 2839190528.0, + "71": 2839190528.0, + "72": 2839190528.0, + "73": 2839190528.0, + "74": 2839190528.0, + "75": 2839190528.0, + "76": 2839190528.0, + "77": 2839190528.0, + "78": 2839190528.0, + "79": 2839190528.0, + "80": 2839190528.0, + "81": 2839190528.0, + "82": 2839190528.0, + "83": 2839190528.0, + "84": 2839190528.0, + "85": 2839190528.0, + "86": 2839190528.0, + "87": 2839190528.0, + "88": 2839190528.0, + "89": 2839190528.0, + "90": 2839190528.0, + "91": 2839190528.0, + "92": 2839190528.0, + "93": 2839190528.0, + "94": 2839190528.0, + "95": 2839190528.0, + "96": 2839190528.0, + "97": 2839190528.0, + "98": 2839190528.0, + "99": 2839190528.0, + "100": 2839190528.0 + } + }, + "iteration-time": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": "nan", + "2": 3.64251, + "3": 0.14564, + "4": 0.69615, + "5": 0.38215, + "6": 0.6346, + "7": 0.74593, + "8": 0.89708, + "9": 0.55832, + "10": 0.28272, + "11": 0.83048, + "12": 0.33235, + "13": 0.61059, + "14": 0.52766, + "15": 0.48374, + "16": 0.47463, + "17": 0.51678, + "18": 0.84008, + "19": 0.31682, + "20": 0.41386, + "21": 0.61198, + "22": 0.53115, + "23": 0.56112, + "24": 0.64114, + "25": 0.14856, + "26": 0.83503, + "27": 0.19993, + "28": 0.50781, + "29": 0.64928, + "30": 0.14737, + "31": 0.26576, + "32": 0.93274, + "33": 0.45057, + "34": 0.64101, + "35": 0.30505, + "36": 0.54406, + "37": 0.44498, + "38": 0.53077, + "39": 0.57178, + "40": 0.6543, + "41": 0.50539, + "42": 0.46748, + "43": 0.50091, + "44": 0.58324, + "45": 0.57242, + "46": 0.3735, + "47": 0.4793, + "48": 0.61504, + "49": 0.42407, + "50": 0.47662, + "51": 0.17782, + "52": 0.4073, + "53": 0.44936, + "54": 0.50732, + "55": 0.61449, + "56": 0.58418, + "57": 0.34155, + "58": 0.63191, + "59": 0.39173, + "60": 0.60985, + "61": 0.39937, + "62": 0.88731, + "63": 0.18294, + "64": 0.87844, + "65": 0.37649, + "66": 0.66987, + "67": 0.56889, + "68": 0.14611, + "69": 0.51706, + "70": 0.93599, + "71": 0.19189, + "72": 0.60981, + "73": 0.51305, + "74": 0.38185, + "75": 0.82964, + "76": 0.37193, + "77": 0.39365, + "78": 1.25405, + "79": 0.34092, + "80": 0.51336, + "81": 0.55853, + "82": 0.2935, + "83": 0.6079, + "84": 0.68193, + "85": 0.15066, + "86": 0.36158, + "87": 0.83725, + "88": 0.45807, + "89": 0.70285, + "90": 0.32966, + "91": 0.4361, + "92": 0.54428, + "93": 0.23552, + "94": 0.40095, + "95": 0.86155, + "96": 0.3819, + "97": 0.50302, + "98": 0.447, + "99": 1.05681, + "100": 0.7726 + } + } +} \ No newline at end of file diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_resume_torch_dist_dist_optimizer_1node/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_resume_torch_dist_dist_optimizer_1node/model_config.yaml new file mode 100644 index 00000000000..5859b7461f9 --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_resume_torch_dist_dist_optimizer_1node/model_config.yaml @@ -0,0 +1,56 @@ +ENV_VARS: + CUDA_DEVICE_MAX_CONNECTIONS: 1 + NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0 + NCCL_ALGO: Ring + CUBLAS_WORKSPACE_CONFIG: :4096:8 +MODEL_ARGS: + --num-layers: 12 + --hidden-size: 512 + --num-attention-heads: 8 + --log-params-norm: true + --log-num-zeros-in-grad: true + --log-validation-ppl-to-tensorboard: true + --log-timers-to-tensorboard: true + --tensorboard-dir: ${TENSORBOARD_PATH} + --micro-batch-size: 4 + --global-batch-size: 32 + --seq-length: 1024 + --max-position-embeddings: 1024 + --train-iters: 100 + --timing-log-level: 0 + --lr-decay-iters: 320000 + --save: ${CHECKPOINT_SAVE_PATH} + --load: ${CHECKPOINT_LOAD_PATH} + --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document + --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json + --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt + --split: 949,50,1 + --distributed-backend: nccl + --lr: 0.00015 + --lr-decay-style: cosine + --min-lr: 1.0e-5 + --weight-decay: 1e-2 + --clip-grad: 1.0 + --lr-warmup-fraction: .01 + --log-interval: 1 + --save-interval: 50 + --eval-interval: 1000 + --eval-iters: 10 + --transformer-impl: transformer_engine + --tensor-model-parallel-size: 1 + --pipeline-model-parallel-size: 1 + --use-distributed-optimizer: true + --no-ckpt-fully-parallel-save: true + --async-save: true + --deterministic-mode: true + --no-gradient-accumulation-fusion: true + --attention-softmax-in-fp32: true + --use-checkpoint-opt_param-scheduler: true + --use-mcore-models: true + --ckpt-format: torch_dist + --dist-ckpt-strictness: log_all # backward compatibility for TE changes + --data-cache-path: ${DATA_CACHE_PATH} + --bf16: true + --log-memory-to-tensorboard: true + --use-persistent-ckpt-worker: true +TEST_TYPE: ckpt-resume diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_resume_torch_dist_uniform_full_recompute_1node/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_resume_torch_dist_uniform_full_recompute_1node/golden_values_dev_dgx_gb200.json new file mode 100644 index 00000000000..529a9e27376 --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_resume_torch_dist_uniform_full_recompute_1node/golden_values_dev_dgx_gb200.json @@ -0,0 +1,537 @@ +{ + "lm loss": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 10.8851, + "2": 10.88271, + "3": 10.88387, + "4": 10.88623, + "5": 10.87916, + "6": 10.87661, + "7": 10.87963, + "8": 10.87211, + "9": 10.88437, + "10": 10.87858, + "11": 10.87354, + "12": 10.86326, + "13": 10.87879, + "14": 10.86595, + "15": 10.83782, + "16": 10.83261, + "17": 10.84044, + "18": 10.82306, + "19": 10.83569, + "20": 10.72913, + "21": 10.7391, + "22": 10.72513, + "23": 10.71154, + "24": 10.68292, + "25": 10.67538, + "26": 10.67981, + "27": 10.63493, + "28": 10.57986, + "29": 10.54428, + "30": 10.5209, + "31": 10.51407, + "32": 10.49437, + "33": 10.46826, + "34": 10.43401, + "35": 10.43851, + "36": 10.41311, + "37": 10.3791, + "38": 10.39376, + "39": 10.3513, + "40": 10.34595, + "41": 10.32313, + "42": 10.29756, + "43": 10.28542, + "44": 10.24461, + "45": 10.27146, + "46": 10.22651, + "47": 10.22023, + "48": 10.1733, + "49": 10.17078, + "50": 10.17217, + "51": 10.17766, + "52": 10.13083, + "53": 10.13975, + "54": 10.10412, + "55": 10.0701, + "56": 10.10628, + "57": 10.09819, + "58": 10.11158, + "59": 10.0548, + "60": 10.07314, + "61": 10.0271, + "62": 9.99675, + "63": 10.07086, + "64": 10.02893, + "65": 10.00122, + "66": 10.02706, + "67": 10.00259, + "68": 9.96482, + "69": 9.9887, + "70": 9.97569, + "71": 9.99808, + "72": 9.97568, + "73": 9.96384, + "74": 9.95689, + "75": 9.92479, + "76": 9.9637, + "77": 9.95559, + "78": 9.90278, + "79": 9.90952, + "80": 9.92737, + "81": 9.94386, + "82": 9.89002, + "83": 9.84872, + "84": 9.78548, + "85": 9.77694, + "86": 9.87451, + "87": 9.90794, + "88": 9.88606, + "89": 9.81898, + "90": 9.81048, + "91": 9.82157, + "92": 9.8151, + "93": 9.7511, + "94": 9.82585, + "95": 9.8205, + "96": 9.80342, + "97": 9.74065, + "98": 9.77415, + "99": 9.81989, + "100": 9.71431 + } + }, + "num-zeros": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 1709.0, + "2": 1738.0, + "3": 1755.0, + "4": 1816.0, + "5": 1698.0, + "6": 1631.0, + "7": 1869.0, + "8": 1739.0, + "9": 1691.0, + "10": 1826.0, + "11": 1722.0, + "12": 1730.0, + "13": 1818.0, + "14": 1863.0, + "15": 1615.0, + "16": 1836.0, + "17": 1747.0, + "18": 1833.0, + "19": 1608.0, + "20": 1657.0, + "21": 1859.0, + "22": 1730.0, + "23": 1635.0, + "24": 1832.0, + "25": 1729.0, + "26": 1798.0, + "27": 1878.0, + "28": 1847.0, + "29": 1947.0, + "30": 1779.0, + "31": 2074.0, + "32": 1982.0, + "33": 2070.0, + "34": 1922.0, + "35": 2024.0, + "36": 2020.0, + "37": 2265.0, + "38": 2093.0, + "39": 2265.0, + "40": 2284.0, + "41": 2386.0, + "42": 2091.0, + "43": 2397.0, + "44": 2272.0, + "45": 2554.0, + "46": 2392.0, + "47": 2509.0, + "48": 2705.0, + "49": 2865.0, + "50": 2674.0, + "51": 2545.0, + "52": 2776.0, + "53": 2583.0, + "54": 3041.0, + "55": 2759.0, + "56": 2815.0, + "57": 2403.0, + "58": 3632.0, + "59": 2959.0, + "60": 3070.0, + "61": 2800.0, + "62": 3300.0, + "63": 3522.0, + "64": 3671.0, + "65": 2744.0, + "66": 3233.0, + "67": 3924.0, + "68": 3752.0, + "69": 3066.0, + "70": 3516.0, + "71": 3069.0, + "72": 3063.0, + "73": 3411.0, + "74": 3374.0, + "75": 3206.0, + "76": 3251.0, + "77": 3808.0, + "78": 3312.0, + "79": 3270.0, + "80": 3004.0, + "81": 3497.0, + "82": 3130.0, + "83": 3054.0, + "84": 3133.0, + "85": 2843.0, + "86": 3235.0, + "87": 2937.0, + "88": 2995.0, + "89": 2953.0, + "90": 3492.0, + "91": 2867.0, + "92": 3067.0, + "93": 3202.0, + "94": 3100.0, + "95": 3381.0, + "96": 3427.0, + "97": 3692.0, + "98": 3619.0, + "99": 3378.0, + "100": 3170.0 + } + }, + "mem-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 1259751936.0, + "2": 1259751936.0, + "3": 1259751936.0, + "4": 1259751936.0, + "5": 1259751936.0, + "6": 1259751936.0, + "7": 1259751936.0, + "8": 1259751936.0, + "9": 1259751936.0, + "10": 1259751936.0, + "11": 1259751936.0, + "12": 1259751936.0, + "13": 1259751936.0, + "14": 1259751936.0, + "15": 1259751936.0, + "16": 1259751936.0, + "17": 1259751936.0, + "18": 1259751936.0, + "19": 1259751936.0, + "20": 1259751936.0, + "21": 1259751936.0, + "22": 1259751936.0, + "23": 1259751936.0, + "24": 1259751936.0, + "25": 1259751936.0, + "26": 1259751936.0, + "27": 1259751936.0, + "28": 1259751936.0, + "29": 1259751936.0, + "30": 1259751936.0, + "31": 1259751936.0, + "32": 1259751936.0, + "33": 1259751936.0, + "34": 1259751936.0, + "35": 1259751936.0, + "36": 1259751936.0, + "37": 1259751936.0, + "38": 1259751936.0, + "39": 1259751936.0, + "40": 1259751936.0, + "41": 1259751936.0, + "42": 1259751936.0, + "43": 1259751936.0, + "44": 1259751936.0, + "45": 1259751936.0, + "46": 1259751936.0, + "47": 1259751936.0, + "48": 1259751936.0, + "49": 1259751936.0, + "50": 1259751936.0, + "51": 1259751936.0, + "52": 1259751936.0, + "53": 1259751936.0, + "54": 1259751936.0, + "55": 1259751936.0, + "56": 1259751936.0, + "57": 1259751936.0, + "58": 1259751936.0, + "59": 1259751936.0, + "60": 1259751936.0, + "61": 1259751936.0, + "62": 1259751936.0, + "63": 1259751936.0, + "64": 1259751936.0, + "65": 1259751936.0, + "66": 1259751936.0, + "67": 1259751936.0, + "68": 1259751936.0, + "69": 1259751936.0, + "70": 1259751936.0, + "71": 1259751936.0, + "72": 1259751936.0, + "73": 1259751936.0, + "74": 1259751936.0, + "75": 1259751936.0, + "76": 1259751936.0, + "77": 1259751936.0, + "78": 1259751936.0, + "79": 1259751936.0, + "80": 1259751936.0, + "81": 1259751936.0, + "82": 1259751936.0, + "83": 1259751936.0, + "84": 1259751936.0, + "85": 1259751936.0, + "86": 1259751936.0, + "87": 1259751936.0, + "88": 1259751936.0, + "89": 1259751936.0, + "90": 1259751936.0, + "91": 1259751936.0, + "92": 1259751936.0, + "93": 1259751936.0, + "94": 1259751936.0, + "95": 1259751936.0, + "96": 1259751936.0, + "97": 1259751936.0, + "98": 1259751936.0, + "99": 1259751936.0, + "100": 1259751936.0 + } + }, + "mem-max-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 2047409664.0, + "2": 2561335808.0, + "3": 2561335808.0, + "4": 2561335808.0, + "5": 2561335808.0, + "6": 2561335808.0, + "7": 2561335808.0, + "8": 2561335808.0, + "9": 2561335808.0, + "10": 2561335808.0, + "11": 2561335808.0, + "12": 2561335808.0, + "13": 2561335808.0, + "14": 2561335808.0, + "15": 2561335808.0, + "16": 2561335808.0, + "17": 2561335808.0, + "18": 2561335808.0, + "19": 2561335808.0, + "20": 2561335808.0, + "21": 2561335808.0, + "22": 2561335808.0, + "23": 2561335808.0, + "24": 2561335808.0, + "25": 2561335808.0, + "26": 2561335808.0, + "27": 2561335808.0, + "28": 2561335808.0, + "29": 2561335808.0, + "30": 2561335808.0, + "31": 2561335808.0, + "32": 2561335808.0, + "33": 2561335808.0, + "34": 2561335808.0, + "35": 2561335808.0, + "36": 2561335808.0, + "37": 2561335808.0, + "38": 2561335808.0, + "39": 2561335808.0, + "40": 2561335808.0, + "41": 2561335808.0, + "42": 2561335808.0, + "43": 2561335808.0, + "44": 2561335808.0, + "45": 2561335808.0, + "46": 2561335808.0, + "47": 2561335808.0, + "48": 2561335808.0, + "49": 2561335808.0, + "50": 2561335808.0, + "51": 2562384384.0, + "52": 2562384384.0, + "53": 2562384384.0, + "54": 2562384384.0, + "55": 2562384384.0, + "56": 2562384384.0, + "57": 2562384384.0, + "58": 2562384384.0, + "59": 2562384384.0, + "60": 2562384384.0, + "61": 2562384384.0, + "62": 2562384384.0, + "63": 2562384384.0, + "64": 2562384384.0, + "65": 2562384384.0, + "66": 2562384384.0, + "67": 2562384384.0, + "68": 2562384384.0, + "69": 2562384384.0, + "70": 2562384384.0, + "71": 2562384384.0, + "72": 2562384384.0, + "73": 2562384384.0, + "74": 2562384384.0, + "75": 2562384384.0, + "76": 2562384384.0, + "77": 2562384384.0, + "78": 2562384384.0, + "79": 2562384384.0, + "80": 2562384384.0, + "81": 2562384384.0, + "82": 2562384384.0, + "83": 2562384384.0, + "84": 2562384384.0, + "85": 2562384384.0, + "86": 2562384384.0, + "87": 2562384384.0, + "88": 2562384384.0, + "89": 2562384384.0, + "90": 2562384384.0, + "91": 2562384384.0, + "92": 2562384384.0, + "93": 2562384384.0, + "94": 2562384384.0, + "95": 2562384384.0, + "96": 2562384384.0, + "97": 2562384384.0, + "98": 2562384384.0, + "99": 2562384384.0, + "100": 2562384384.0 + } + }, + "iteration-time": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": "nan", + "2": 3.57509, + "3": 0.20178, + "4": 0.82258, + "5": 0.39919, + "6": 0.69081, + "7": 0.98895, + "8": 0.98378, + "9": 0.69702, + "10": 0.29157, + "11": 1.19804, + "12": 0.30424, + "13": 0.73918, + "14": 0.62143, + "15": 0.47407, + "16": 0.73703, + "17": 0.53845, + "18": 0.83743, + "19": 0.43301, + "20": 0.39515, + "21": 0.73791, + "22": 0.6433, + "23": 0.60787, + "24": 0.7222, + "25": 0.20582, + "26": 0.94288, + "27": 0.20499, + "28": 0.76266, + "29": 0.70845, + "30": 0.22175, + "31": 0.21449, + "32": 0.9657, + "33": 0.53192, + "34": 0.32966, + "35": 0.63851, + "36": 0.57629, + "37": 0.73158, + "38": 0.36522, + "39": 0.59696, + "40": 0.71549, + "41": 1.01974, + "42": 0.41239, + "43": 0.60102, + "44": 0.45107, + "45": 0.5768, + "46": 0.68473, + "47": 0.43513, + "48": 0.49857, + "49": 0.59588, + "50": 0.52239, + "51": 0.24529, + "52": 0.49446, + "53": 0.45003, + "54": 0.51564, + "55": 0.66543, + "56": 0.67498, + "57": 0.48678, + "58": 0.54374, + "59": 0.52969, + "60": 0.66159, + "61": 0.42186, + "62": 0.72921, + "63": 0.39725, + "64": 0.75179, + "65": 0.5659, + "66": 0.64875, + "67": 0.5554, + "68": 0.21749, + "69": 0.59304, + "70": 0.9711, + "71": 0.21399, + "72": 0.61448, + "73": 0.5583, + "74": 0.65536, + "75": 0.99327, + "76": 0.36856, + "77": 0.38857, + "78": 1.32148, + "79": 0.31772, + "80": 0.56325, + "81": 0.54162, + "82": 0.35743, + "83": 0.65589, + "84": 0.62591, + "85": 0.2224, + "86": 0.8255, + "87": 0.5126, + "88": 0.52961, + "89": 0.67579, + "90": 0.29558, + "91": 0.44578, + "92": 0.65353, + "93": 0.24317, + "94": 0.69969, + "95": 0.67753, + "96": 0.5725, + "97": 0.41921, + "98": 0.50783, + "99": 1.17035, + "100": 0.27449 + } + } +} \ No newline at end of file diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_resume_torch_dist_uniform_full_recompute_1node/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_resume_torch_dist_uniform_full_recompute_1node/model_config.yaml new file mode 100644 index 00000000000..c3ca9477dd7 --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp1_resume_torch_dist_uniform_full_recompute_1node/model_config.yaml @@ -0,0 +1,57 @@ +ENV_VARS: + CUDA_DEVICE_MAX_CONNECTIONS: 1 + NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0 + NCCL_ALGO: Ring + CUBLAS_WORKSPACE_CONFIG: :4096:8 +MODEL_ARGS: + --num-layers: 12 + --hidden-size: 512 + --num-attention-heads: 8 + --log-params-norm: true + --log-num-zeros-in-grad: true + --log-validation-ppl-to-tensorboard: true + --log-timers-to-tensorboard: true + --tensorboard-dir: ${TENSORBOARD_PATH} + --micro-batch-size: 4 + --global-batch-size: 32 + --seq-length: 1024 + --max-position-embeddings: 1024 + --train-iters: 100 + --timing-log-level: 0 + --lr-decay-iters: 320000 + --save: ${CHECKPOINT_SAVE_PATH} + --load: ${CHECKPOINT_LOAD_PATH} + --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document + --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json + --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt + --split: 949,50,1 + --distributed-backend: nccl + --lr: 0.00015 + --lr-decay-style: cosine + --min-lr: 1.0e-5 + --weight-decay: 1e-2 + --clip-grad: 1.0 + --lr-warmup-fraction: .01 + --log-interval: 1 + --save-interval: 50 + --eval-interval: 1000 + --eval-iters: 10 + --transformer-impl: transformer_engine + --tensor-model-parallel-size: 1 + --pipeline-model-parallel-size: 1 + --recompute-granularity: full + --recompute-method: uniform + --recompute-num-layers: 1 + --deterministic-mode: true + --no-gradient-accumulation-fusion: true + --attention-softmax-in-fp32: true + --use-checkpoint-opt_param-scheduler: true + --use-mcore-models: true + --ckpt-format: torch_dist + --dist-ckpt-strictness: log_all # backward compatibility for TE changes + --data-cache-path: ${DATA_CACHE_PATH} + --bf16: true + --log-memory-to-tensorboard: true + --async-save: true + --use-persistent-ckpt-worker: true +TEST_TYPE: ckpt-resume diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp2_resume_torch_dist_rope_embeddings_1node/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp2_resume_torch_dist_rope_embeddings_1node/golden_values_dev_dgx_gb200.json new file mode 100644 index 00000000000..5e8a1e44b89 --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp2_resume_torch_dist_rope_embeddings_1node/golden_values_dev_dgx_gb200.json @@ -0,0 +1,537 @@ +{ + "lm loss": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 10.90185, + "2": 10.90476, + "3": 10.91298, + "4": 10.90732, + "5": 10.90338, + "6": 10.88352, + "7": 10.89466, + "8": 10.9003, + "9": 10.90241, + "10": 10.90124, + "11": 10.89365, + "12": 10.88403, + "13": 10.87658, + "14": 10.87083, + "15": 10.86053, + "16": 10.85047, + "17": 10.85366, + "18": 10.83505, + "19": 10.83504, + "20": 10.75556, + "21": 10.75273, + "22": 10.74048, + "23": 10.73225, + "24": 10.6904, + "25": 10.70403, + "26": 10.68252, + "27": 10.64963, + "28": 10.5671, + "29": 10.54339, + "30": 10.51454, + "31": 10.51312, + "32": 10.49085, + "33": 10.46425, + "34": 10.41243, + "35": 10.41921, + "36": 10.40599, + "37": 10.35423, + "38": 10.36112, + "39": 10.32243, + "40": 10.32058, + "41": 10.28482, + "42": 10.27525, + "43": 10.24427, + "44": 10.21985, + "45": 10.23687, + "46": 10.19624, + "47": 10.18389, + "48": 10.1323, + "49": 10.13047, + "50": 10.14436, + "51": 10.14123, + "52": 10.09131, + "53": 10.08918, + "54": 10.06284, + "55": 10.01711, + "56": 10.07149, + "57": 10.03622, + "58": 10.06524, + "59": 10.00246, + "60": 10.01718, + "61": 9.98392, + "62": 9.93675, + "63": 10.03256, + "64": 9.96969, + "65": 9.93232, + "66": 9.97442, + "67": 9.94335, + "68": 9.8968, + "69": 9.9134, + "70": 9.9038, + "71": 9.93369, + "72": 9.89264, + "73": 9.87323, + "74": 9.87541, + "75": 9.84114, + "76": 9.89728, + "77": 9.8872, + "78": 9.82611, + "79": 9.83323, + "80": 9.8524, + "81": 9.87595, + "82": 9.82806, + "83": 9.7717, + "84": 9.70894, + "85": 9.69359, + "86": 9.80476, + "87": 9.8523, + "88": 9.82792, + "89": 9.74787, + "90": 9.74092, + "91": 9.7535, + "92": 9.74928, + "93": 9.67428, + "94": 9.75078, + "95": 9.75704, + "96": 9.73734, + "97": 9.6548, + "98": 9.69864, + "99": 9.75611, + "100": 9.63811 + } + }, + "num-zeros": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 1985.0, + "2": 1849.0, + "3": 1918.0, + "4": 1845.0, + "5": 1981.0, + "6": 1867.0, + "7": 2134.0, + "8": 1795.0, + "9": 1915.0, + "10": 2000.0, + "11": 1951.0, + "12": 1871.0, + "13": 1991.0, + "14": 2057.0, + "15": 1826.0, + "16": 1941.0, + "17": 2086.0, + "18": 2095.0, + "19": 2002.0, + "20": 1853.0, + "21": 2090.0, + "22": 1944.0, + "23": 1847.0, + "24": 2008.0, + "25": 1831.0, + "26": 2011.0, + "27": 1982.0, + "28": 1928.0, + "29": 2119.0, + "30": 1923.0, + "31": 2072.0, + "32": 1955.0, + "33": 2018.0, + "34": 2088.0, + "35": 2066.0, + "36": 2037.0, + "37": 2227.0, + "38": 2208.0, + "39": 2078.0, + "40": 2359.0, + "41": 2345.0, + "42": 2083.0, + "43": 2417.0, + "44": 2281.0, + "45": 2524.0, + "46": 2404.0, + "47": 2372.0, + "48": 2558.0, + "49": 2718.0, + "50": 2502.0, + "51": 2528.0, + "52": 2618.0, + "53": 2473.0, + "54": 2778.0, + "55": 2518.0, + "56": 2819.0, + "57": 2354.0, + "58": 3243.0, + "59": 2808.0, + "60": 2877.0, + "61": 2627.0, + "62": 2989.0, + "63": 3013.0, + "64": 3270.0, + "65": 2533.0, + "66": 2795.0, + "67": 3328.0, + "68": 3128.0, + "69": 3087.0, + "70": 2901.0, + "71": 3034.0, + "72": 3116.0, + "73": 3270.0, + "74": 3013.0, + "75": 3097.0, + "76": 3258.0, + "77": 3142.0, + "78": 3204.0, + "79": 2963.0, + "80": 3126.0, + "81": 3222.0, + "82": 3247.0, + "83": 2893.0, + "84": 2910.0, + "85": 2985.0, + "86": 3218.0, + "87": 3275.0, + "88": 3179.0, + "89": 3082.0, + "90": 3491.0, + "91": 2820.0, + "92": 2940.0, + "93": 2944.0, + "94": 3371.0, + "95": 3239.0, + "96": 3199.0, + "97": 3432.0, + "98": 3457.0, + "99": 3228.0, + "100": 3028.0 + } + }, + "mem-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 923751424.0, + "2": 923751424.0, + "3": 923751424.0, + "4": 923751424.0, + "5": 923751424.0, + "6": 923751424.0, + "7": 923751424.0, + "8": 923751424.0, + "9": 923751424.0, + "10": 923751424.0, + "11": 923751424.0, + "12": 923751424.0, + "13": 923751424.0, + "14": 923751424.0, + "15": 923751424.0, + "16": 923751424.0, + "17": 923751424.0, + "18": 923751424.0, + "19": 923751424.0, + "20": 923751424.0, + "21": 923751424.0, + "22": 923751424.0, + "23": 923751424.0, + "24": 923751424.0, + "25": 923751424.0, + "26": 923751424.0, + "27": 923751424.0, + "28": 923751424.0, + "29": 923751424.0, + "30": 923751424.0, + "31": 923751424.0, + "32": 923751424.0, + "33": 923751424.0, + "34": 923751424.0, + "35": 923751424.0, + "36": 923751424.0, + "37": 923751424.0, + "38": 923751424.0, + "39": 923751424.0, + "40": 923751424.0, + "41": 923751424.0, + "42": 923751424.0, + "43": 923751424.0, + "44": 923751424.0, + "45": 923751424.0, + "46": 923751424.0, + "47": 923751424.0, + "48": 923751424.0, + "49": 923751424.0, + "50": 923751424.0, + "51": 923751424.0, + "52": 923751424.0, + "53": 923751424.0, + "54": 923751424.0, + "55": 923751424.0, + "56": 923751424.0, + "57": 923751424.0, + "58": 923751424.0, + "59": 923751424.0, + "60": 923751424.0, + "61": 923751424.0, + "62": 923751424.0, + "63": 923751424.0, + "64": 923751424.0, + "65": 923751424.0, + "66": 923751424.0, + "67": 923751424.0, + "68": 923751424.0, + "69": 923751424.0, + "70": 923751424.0, + "71": 923751424.0, + "72": 923751424.0, + "73": 923751424.0, + "74": 923751424.0, + "75": 923751424.0, + "76": 923751424.0, + "77": 923751424.0, + "78": 923751424.0, + "79": 923751424.0, + "80": 923751424.0, + "81": 923751424.0, + "82": 923751424.0, + "83": 923751424.0, + "84": 923751424.0, + "85": 923751424.0, + "86": 923751424.0, + "87": 923751424.0, + "88": 923751424.0, + "89": 923751424.0, + "90": 923751424.0, + "91": 923751424.0, + "92": 923751424.0, + "93": 923751424.0, + "94": 923751424.0, + "95": 923751424.0, + "96": 923751424.0, + "97": 923751424.0, + "98": 923751424.0, + "99": 923751424.0, + "100": 923751424.0 + } + }, + "mem-max-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 2262890496.0, + "2": 2627599360.0, + "3": 2627599360.0, + "4": 2627599360.0, + "5": 2627599360.0, + "6": 2627599360.0, + "7": 2627599360.0, + "8": 2627599360.0, + "9": 2627599360.0, + "10": 2627599360.0, + "11": 2627599360.0, + "12": 2627599360.0, + "13": 2627599360.0, + "14": 2627599360.0, + "15": 2627599360.0, + "16": 2627599360.0, + "17": 2627599360.0, + "18": 2627599360.0, + "19": 2627599360.0, + "20": 2627599360.0, + "21": 2627599360.0, + "22": 2627599360.0, + "23": 2627599360.0, + "24": 2627599360.0, + "25": 2627599360.0, + "26": 2627599360.0, + "27": 2627599360.0, + "28": 2627599360.0, + "29": 2627599360.0, + "30": 2627599360.0, + "31": 2627599360.0, + "32": 2627599360.0, + "33": 2627599360.0, + "34": 2627599360.0, + "35": 2627599360.0, + "36": 2627599360.0, + "37": 2627599360.0, + "38": 2627599360.0, + "39": 2627599360.0, + "40": 2627599360.0, + "41": 2627599360.0, + "42": 2627599360.0, + "43": 2627599360.0, + "44": 2627599360.0, + "45": 2627599360.0, + "46": 2627599360.0, + "47": 2627599360.0, + "48": 2627599360.0, + "49": 2627599360.0, + "50": 2627599360.0, + "51": 2627599360.0, + "52": 2627599360.0, + "53": 2627599360.0, + "54": 2627599360.0, + "55": 2627599360.0, + "56": 2627599360.0, + "57": 2627599360.0, + "58": 2627599360.0, + "59": 2627599360.0, + "60": 2627599360.0, + "61": 2627599360.0, + "62": 2627599360.0, + "63": 2627599360.0, + "64": 2627599360.0, + "65": 2627599360.0, + "66": 2627599360.0, + "67": 2627599360.0, + "68": 2627599360.0, + "69": 2627599360.0, + "70": 2627599360.0, + "71": 2627599360.0, + "72": 2627599360.0, + "73": 2627599360.0, + "74": 2627599360.0, + "75": 2627599360.0, + "76": 2627599360.0, + "77": 2627599360.0, + "78": 2627599360.0, + "79": 2627599360.0, + "80": 2627599360.0, + "81": 2627599360.0, + "82": 2627599360.0, + "83": 2627599360.0, + "84": 2627599360.0, + "85": 2627599360.0, + "86": 2627599360.0, + "87": 2627599360.0, + "88": 2627599360.0, + "89": 2627599360.0, + "90": 2627599360.0, + "91": 2627599360.0, + "92": 2627599360.0, + "93": 2627599360.0, + "94": 2627599360.0, + "95": 2627599360.0, + "96": 2627599360.0, + "97": 2627599360.0, + "98": 2627599360.0, + "99": 2627599360.0, + "100": 2627599360.0 + } + }, + "iteration-time": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": "nan", + "2": 3.73641, + "3": 1.07133, + "4": 0.77912, + "5": 0.63046, + "6": 0.7537, + "7": 0.93703, + "8": 0.9306, + "9": 0.89884, + "10": 0.56496, + "11": 0.8365, + "12": 0.90593, + "13": 0.70214, + "14": 1.01127, + "15": 0.52752, + "16": 0.8363, + "17": 1.16065, + "18": 0.99854, + "19": 0.85041, + "20": 0.98386, + "21": 0.78951, + "22": 0.8023, + "23": 0.92475, + "24": 1.23701, + "25": 0.46994, + "26": 0.93265, + "27": 0.87922, + "28": 0.77761, + "29": 0.93158, + "30": 0.71698, + "31": 0.68337, + "32": 1.2395, + "33": 0.97382, + "34": 0.35633, + "35": 1.23896, + "36": 0.91933, + "37": 0.82472, + "38": 0.73137, + "39": 0.6832, + "40": 1.20931, + "41": 0.72399, + "42": 0.91417, + "43": 0.68297, + "44": 1.08135, + "45": 0.80831, + "46": 0.8797, + "47": 0.99429, + "48": 0.95444, + "49": 0.87085, + "50": 1.10394, + "51": 0.39822, + "52": 0.54513, + "53": 0.93496, + "54": 0.9221, + "55": 0.89089, + "56": 0.77753, + "57": 0.75592, + "58": 0.69142, + "59": 0.86385, + "60": 0.76462, + "61": 0.58373, + "62": 1.17109, + "63": 0.81784, + "64": 0.9365, + "65": 0.68267, + "66": 1.11481, + "67": 1.17844, + "68": 0.70035, + "69": 1.21913, + "70": 0.76552, + "71": 1.20082, + "72": 0.94089, + "73": 0.75625, + "74": 0.65162, + "75": 0.89315, + "76": 0.8307, + "77": 0.8403, + "78": 0.88644, + "79": 1.13763, + "80": 0.98328, + "81": 0.64334, + "82": 1.15935, + "83": 0.90091, + "84": 0.69281, + "85": 0.42082, + "86": 0.75252, + "87": 1.26204, + "88": 0.6022, + "89": 0.66422, + "90": 0.85466, + "91": 0.69881, + "92": 1.23749, + "93": 0.77535, + "94": 0.88019, + "95": 0.82023, + "96": 0.60602, + "97": 0.7628, + "98": 0.97336, + "99": 1.20363, + "100": 0.7509 + } + } +} \ No newline at end of file diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp2_resume_torch_dist_rope_embeddings_1node/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp2_resume_torch_dist_rope_embeddings_1node/model_config.yaml new file mode 100644 index 00000000000..9cde3247944 --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp2_resume_torch_dist_rope_embeddings_1node/model_config.yaml @@ -0,0 +1,55 @@ +ENV_VARS: + CUDA_DEVICE_MAX_CONNECTIONS: 1 + NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0 + NCCL_ALGO: Ring + CUBLAS_WORKSPACE_CONFIG: :4096:8 +MODEL_ARGS: + --num-layers: 12 + --hidden-size: 512 + --num-attention-heads: 8 + --log-params-norm: true + --log-num-zeros-in-grad: true + --log-validation-ppl-to-tensorboard: true + --log-timers-to-tensorboard: true + --tensorboard-dir: ${TENSORBOARD_PATH} + --micro-batch-size: 4 + --global-batch-size: 32 + --seq-length: 1024 + --max-position-embeddings: 1024 + --train-iters: 100 + --timing-log-level: 0 + --lr-decay-iters: 320000 + --save: ${CHECKPOINT_SAVE_PATH} + --load: ${CHECKPOINT_LOAD_PATH} + --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document + --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json + --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt + --split: 949,50,1 + --distributed-backend: nccl + --lr: 0.00015 + --lr-decay-style: cosine + --min-lr: 1.0e-5 + --weight-decay: 1e-2 + --clip-grad: 1.0 + --lr-warmup-fraction: .01 + --log-interval: 1 + --save-interval: 50 + --eval-interval: 1000 + --eval-iters: 10 + --transformer-impl: transformer_engine + --tensor-model-parallel-size: 1 + --pipeline-model-parallel-size: 2 + --position-embedding-type: rope + --deterministic-mode: true + --no-gradient-accumulation-fusion: true + --attention-softmax-in-fp32: true + --use-checkpoint-opt_param-scheduler: true + --use-mcore-models: true + --ckpt-format: torch_dist + --dist-ckpt-strictness: log_all # backward compatibility for TE changes + --data-cache-path: ${DATA_CACHE_PATH} + --bf16: true + --log-memory-to-tensorboard: true + --async-save: true + --use-persistent-ckpt-worker: true +TEST_TYPE: ckpt-resume diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp2_resume_torch_dist_rope_embeddings_interleaved_no_fusion_1node/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp2_resume_torch_dist_rope_embeddings_interleaved_no_fusion_1node/golden_values_dev_dgx_gb200.json new file mode 100644 index 00000000000..4a78de1149f --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp2_resume_torch_dist_rope_embeddings_interleaved_no_fusion_1node/golden_values_dev_dgx_gb200.json @@ -0,0 +1,537 @@ +{ + "lm loss": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 10.90174, + "2": 10.90447, + "3": 10.91297, + "4": 10.90733, + "5": 10.90344, + "6": 10.88332, + "7": 10.89468, + "8": 10.90028, + "9": 10.9021, + "10": 10.90121, + "11": 10.89368, + "12": 10.88437, + "13": 10.87653, + "14": 10.87083, + "15": 10.86039, + "16": 10.85074, + "17": 10.85348, + "18": 10.83498, + "19": 10.83514, + "20": 10.75544, + "21": 10.7526, + "22": 10.74059, + "23": 10.73225, + "24": 10.69032, + "25": 10.70418, + "26": 10.6823, + "27": 10.64975, + "28": 10.56722, + "29": 10.54333, + "30": 10.51478, + "31": 10.5135, + "32": 10.49083, + "33": 10.46441, + "34": 10.41257, + "35": 10.41932, + "36": 10.40629, + "37": 10.35435, + "38": 10.36121, + "39": 10.32238, + "40": 10.3208, + "41": 10.28486, + "42": 10.2753, + "43": 10.24448, + "44": 10.22008, + "45": 10.23701, + "46": 10.19652, + "47": 10.184, + "48": 10.13231, + "49": 10.13082, + "50": 10.14454, + "51": 10.14126, + "52": 10.09149, + "53": 10.08937, + "54": 10.06294, + "55": 10.01717, + "56": 10.07169, + "57": 10.03655, + "58": 10.06554, + "59": 10.00273, + "60": 10.01727, + "61": 9.98398, + "62": 9.937, + "63": 10.03276, + "64": 9.96992, + "65": 9.93251, + "66": 9.97456, + "67": 9.94352, + "68": 9.89683, + "69": 9.91356, + "70": 9.904, + "71": 9.93385, + "72": 9.8928, + "73": 9.87347, + "74": 9.87565, + "75": 9.84138, + "76": 9.89744, + "77": 9.88738, + "78": 9.82628, + "79": 9.83349, + "80": 9.85242, + "81": 9.876, + "82": 9.82812, + "83": 9.77197, + "84": 9.70915, + "85": 9.6939, + "86": 9.80488, + "87": 9.85235, + "88": 9.82811, + "89": 9.74806, + "90": 9.74109, + "91": 9.75365, + "92": 9.74941, + "93": 9.67469, + "94": 9.75091, + "95": 9.75728, + "96": 9.73751, + "97": 9.65522, + "98": 9.69868, + "99": 9.75619, + "100": 9.63837 + } + }, + "num-zeros": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 1978.0, + "2": 1895.0, + "3": 1998.0, + "4": 1966.0, + "5": 1875.0, + "6": 1861.0, + "7": 2170.0, + "8": 1843.0, + "9": 1944.0, + "10": 1981.0, + "11": 1941.0, + "12": 1919.0, + "13": 1998.0, + "14": 2075.0, + "15": 1740.0, + "16": 1985.0, + "17": 2016.0, + "18": 2009.0, + "19": 1899.0, + "20": 1950.0, + "21": 2023.0, + "22": 2029.0, + "23": 1864.0, + "24": 2101.0, + "25": 1890.0, + "26": 1878.0, + "27": 2006.0, + "28": 1946.0, + "29": 2100.0, + "30": 1930.0, + "31": 1962.0, + "32": 1987.0, + "33": 2055.0, + "34": 2071.0, + "35": 2105.0, + "36": 2062.0, + "37": 2260.0, + "38": 2200.0, + "39": 2203.0, + "40": 2429.0, + "41": 2313.0, + "42": 2009.0, + "43": 2315.0, + "44": 2252.0, + "45": 2617.0, + "46": 2370.0, + "47": 2401.0, + "48": 2580.0, + "49": 2660.0, + "50": 2554.0, + "51": 2637.0, + "52": 2630.0, + "53": 2566.0, + "54": 2752.0, + "55": 2425.0, + "56": 2821.0, + "57": 2290.0, + "58": 3193.0, + "59": 2711.0, + "60": 2823.0, + "61": 2724.0, + "62": 2944.0, + "63": 3036.0, + "64": 3205.0, + "65": 2564.0, + "66": 2898.0, + "67": 3355.0, + "68": 3084.0, + "69": 3068.0, + "70": 2941.0, + "71": 2941.0, + "72": 3184.0, + "73": 3234.0, + "74": 3085.0, + "75": 3120.0, + "76": 3286.0, + "77": 3099.0, + "78": 3156.0, + "79": 2924.0, + "80": 3054.0, + "81": 3184.0, + "82": 3181.0, + "83": 2900.0, + "84": 2870.0, + "85": 2927.0, + "86": 3155.0, + "87": 3329.0, + "88": 3293.0, + "89": 3022.0, + "90": 3468.0, + "91": 2884.0, + "92": 2866.0, + "93": 3052.0, + "94": 3389.0, + "95": 3208.0, + "96": 3251.0, + "97": 3320.0, + "98": 3483.0, + "99": 3022.0, + "100": 3058.0 + } + }, + "mem-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 923751424.0, + "2": 923751424.0, + "3": 923751424.0, + "4": 923751424.0, + "5": 923751424.0, + "6": 923751424.0, + "7": 923751424.0, + "8": 923751424.0, + "9": 923751424.0, + "10": 923751424.0, + "11": 923751424.0, + "12": 923751424.0, + "13": 923751424.0, + "14": 923751424.0, + "15": 923751424.0, + "16": 923751424.0, + "17": 923751424.0, + "18": 923751424.0, + "19": 923751424.0, + "20": 923751424.0, + "21": 923751424.0, + "22": 923751424.0, + "23": 923751424.0, + "24": 923751424.0, + "25": 923751424.0, + "26": 923751424.0, + "27": 923751424.0, + "28": 923751424.0, + "29": 923751424.0, + "30": 923751424.0, + "31": 923751424.0, + "32": 923751424.0, + "33": 923751424.0, + "34": 923751424.0, + "35": 923751424.0, + "36": 923751424.0, + "37": 923751424.0, + "38": 923751424.0, + "39": 923751424.0, + "40": 923751424.0, + "41": 923751424.0, + "42": 923751424.0, + "43": 923751424.0, + "44": 923751424.0, + "45": 923751424.0, + "46": 923751424.0, + "47": 923751424.0, + "48": 923751424.0, + "49": 923751424.0, + "50": 923751424.0, + "51": 923751424.0, + "52": 923751424.0, + "53": 923751424.0, + "54": 923751424.0, + "55": 923751424.0, + "56": 923751424.0, + "57": 923751424.0, + "58": 923751424.0, + "59": 923751424.0, + "60": 923751424.0, + "61": 923751424.0, + "62": 923751424.0, + "63": 923751424.0, + "64": 923751424.0, + "65": 923751424.0, + "66": 923751424.0, + "67": 923751424.0, + "68": 923751424.0, + "69": 923751424.0, + "70": 923751424.0, + "71": 923751424.0, + "72": 923751424.0, + "73": 923751424.0, + "74": 923751424.0, + "75": 923751424.0, + "76": 923751424.0, + "77": 923751424.0, + "78": 923751424.0, + "79": 923751424.0, + "80": 923751424.0, + "81": 923751424.0, + "82": 923751424.0, + "83": 923751424.0, + "84": 923751424.0, + "85": 923751424.0, + "86": 923751424.0, + "87": 923751424.0, + "88": 923751424.0, + "89": 923751424.0, + "90": 923751424.0, + "91": 923751424.0, + "92": 923751424.0, + "93": 923751424.0, + "94": 923751424.0, + "95": 923751424.0, + "96": 923751424.0, + "97": 923751424.0, + "98": 923751424.0, + "99": 923751424.0, + "100": 923751424.0 + } + }, + "mem-max-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 2266036224.0, + "2": 2630745088.0, + "3": 2630745088.0, + "4": 2630745088.0, + "5": 2630745088.0, + "6": 2630745088.0, + "7": 2630745088.0, + "8": 2630745088.0, + "9": 2630745088.0, + "10": 2630745088.0, + "11": 2630745088.0, + "12": 2630745088.0, + "13": 2630745088.0, + "14": 2630745088.0, + "15": 2630745088.0, + "16": 2630745088.0, + "17": 2630745088.0, + "18": 2630745088.0, + "19": 2630745088.0, + "20": 2630745088.0, + "21": 2630745088.0, + "22": 2630745088.0, + "23": 2630745088.0, + "24": 2630745088.0, + "25": 2630745088.0, + "26": 2630745088.0, + "27": 2630745088.0, + "28": 2630745088.0, + "29": 2630745088.0, + "30": 2630745088.0, + "31": 2630745088.0, + "32": 2630745088.0, + "33": 2630745088.0, + "34": 2630745088.0, + "35": 2630745088.0, + "36": 2630745088.0, + "37": 2630745088.0, + "38": 2630745088.0, + "39": 2630745088.0, + "40": 2630745088.0, + "41": 2630745088.0, + "42": 2630745088.0, + "43": 2630745088.0, + "44": 2630745088.0, + "45": 2630745088.0, + "46": 2630745088.0, + "47": 2630745088.0, + "48": 2630745088.0, + "49": 2630745088.0, + "50": 2630745088.0, + "51": 2630745088.0, + "52": 2630745088.0, + "53": 2630745088.0, + "54": 2630745088.0, + "55": 2630745088.0, + "56": 2630745088.0, + "57": 2630745088.0, + "58": 2630745088.0, + "59": 2630745088.0, + "60": 2630745088.0, + "61": 2630745088.0, + "62": 2630745088.0, + "63": 2630745088.0, + "64": 2630745088.0, + "65": 2630745088.0, + "66": 2630745088.0, + "67": 2630745088.0, + "68": 2630745088.0, + "69": 2630745088.0, + "70": 2630745088.0, + "71": 2630745088.0, + "72": 2630745088.0, + "73": 2630745088.0, + "74": 2630745088.0, + "75": 2630745088.0, + "76": 2630745088.0, + "77": 2630745088.0, + "78": 2630745088.0, + "79": 2630745088.0, + "80": 2630745088.0, + "81": 2630745088.0, + "82": 2630745088.0, + "83": 2630745088.0, + "84": 2630745088.0, + "85": 2630745088.0, + "86": 2630745088.0, + "87": 2630745088.0, + "88": 2630745088.0, + "89": 2630745088.0, + "90": 2630745088.0, + "91": 2630745088.0, + "92": 2630745088.0, + "93": 2630745088.0, + "94": 2630745088.0, + "95": 2630745088.0, + "96": 2630745088.0, + "97": 2630745088.0, + "98": 2630745088.0, + "99": 2630745088.0, + "100": 2630745088.0 + } + }, + "iteration-time": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": "nan", + "2": 3.75585, + "3": 1.22887, + "4": 0.89437, + "5": 0.7097, + "6": 0.82379, + "7": 0.71934, + "8": 1.27771, + "9": 1.09445, + "10": 0.68914, + "11": 0.90417, + "12": 0.95738, + "13": 0.59106, + "14": 1.17819, + "15": 0.84828, + "16": 0.63637, + "17": 0.98456, + "18": 0.84786, + "19": 0.72655, + "20": 0.77842, + "21": 0.71996, + "22": 0.96397, + "23": 0.70114, + "24": 1.38488, + "25": 0.54712, + "26": 1.02075, + "27": 0.57509, + "28": 0.75067, + "29": 1.07444, + "30": 0.82259, + "31": 0.73618, + "32": 1.46436, + "33": 0.81503, + "34": 0.32153, + "35": 1.09799, + "36": 1.07849, + "37": 0.82266, + "38": 0.48851, + "39": 0.79108, + "40": 1.35691, + "41": 0.70674, + "42": 0.78456, + "43": 1.05378, + "44": 0.96238, + "45": 0.91653, + "46": 0.68307, + "47": 0.85659, + "48": 0.78411, + "49": 0.82818, + "50": 1.04866, + "51": 0.37336, + "52": 0.7845, + "53": 0.88546, + "54": 0.89871, + "55": 0.94756, + "56": 1.07823, + "57": 0.63618, + "58": 0.96951, + "59": 0.73593, + "60": 0.71921, + "61": 1.0793, + "62": 0.78166, + "63": 0.83449, + "64": 1.00959, + "65": 0.5949, + "66": 3.05082, + "67": 1.08646, + "68": 0.51947, + "69": 1.12208, + "70": 0.79343, + "71": 0.94914, + "72": 0.81531, + "73": 0.7217, + "74": 0.55725, + "75": 0.61983, + "76": 0.91319, + "77": 0.91216, + "78": 0.69786, + "79": 1.14791, + "80": 0.78017, + "81": 0.62833, + "82": 0.97476, + "83": 0.96088, + "84": 0.63944, + "85": 0.52443, + "86": 0.70965, + "87": 1.094, + "88": 0.60503, + "89": 0.86781, + "90": 0.91008, + "91": 0.83472, + "92": 1.29389, + "93": 0.68, + "94": 1.03141, + "95": 0.91085, + "96": 0.61253, + "97": 0.77848, + "98": 0.89504, + "99": 1.09421, + "100": 0.58777 + } + } +} \ No newline at end of file diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp2_resume_torch_dist_rope_embeddings_interleaved_no_fusion_1node/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp2_resume_torch_dist_rope_embeddings_interleaved_no_fusion_1node/model_config.yaml new file mode 100644 index 00000000000..b4e07b5b5e1 --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp2_resume_torch_dist_rope_embeddings_interleaved_no_fusion_1node/model_config.yaml @@ -0,0 +1,57 @@ +ENV_VARS: + CUDA_DEVICE_MAX_CONNECTIONS: 1 + NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0 + NCCL_ALGO: Ring + CUBLAS_WORKSPACE_CONFIG: :4096:8 +MODEL_ARGS: + --num-layers: 12 + --hidden-size: 512 + --num-attention-heads: 8 + --log-params-norm: true + --log-num-zeros-in-grad: true + --log-validation-ppl-to-tensorboard: true + --log-timers-to-tensorboard: true + --tensorboard-dir: ${TENSORBOARD_PATH} + --micro-batch-size: 4 + --global-batch-size: 32 + --seq-length: 1024 + --max-position-embeddings: 1024 + --train-iters: 100 + --timing-log-level: 0 + --lr-decay-iters: 320000 + --save: ${CHECKPOINT_SAVE_PATH} + --load: ${CHECKPOINT_LOAD_PATH} + --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document + --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json + --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt + --split: 949,50,1 + --distributed-backend: nccl + --lr: 0.00015 + --lr-decay-style: cosine + --min-lr: 1.0e-5 + --weight-decay: 1e-2 + --clip-grad: 1.0 + --lr-warmup-fraction: .01 + --log-interval: 1 + --save-interval: 50 + --eval-interval: 1000 + --eval-iters: 10 + --transformer-impl: transformer_engine + --tensor-model-parallel-size: 1 + --pipeline-model-parallel-size: 2 + --position-embedding-type: rope + --rotary-interleaved: true + --no-rope-fusion: true + --deterministic-mode: true + --no-gradient-accumulation-fusion: true + --attention-softmax-in-fp32: true + --use-checkpoint-opt_param-scheduler: true + --use-mcore-models: true + --ckpt-format: torch_dist + --dist-ckpt-strictness: log_all # backward compatibility for TE changes + --data-cache-path: ${DATA_CACHE_PATH} + --bf16: true + --log-memory-to-tensorboard: true + --async-save: true + --use-persistent-ckpt-worker: true +TEST_TYPE: ckpt-resume diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_disable_bias_linear_1node/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_disable_bias_linear_1node/golden_values_dev_dgx_gb200.json new file mode 100644 index 00000000000..c55077b5809 --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_disable_bias_linear_1node/golden_values_dev_dgx_gb200.json @@ -0,0 +1,537 @@ +{ + "lm loss": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 10.92753, + "2": 10.92559, + "3": 10.93114, + "4": 10.93741, + "5": 10.93016, + "6": 10.92355, + "7": 10.93863, + "8": 10.92774, + "9": 10.93076, + "10": 10.92802, + "11": 10.91411, + "12": 10.92298, + "13": 10.92014, + "14": 10.90614, + "15": 10.88338, + "16": 10.87951, + "17": 10.88868, + "18": 10.86451, + "19": 10.87355, + "20": 10.783, + "21": 10.78553, + "22": 10.76764, + "23": 10.764, + "24": 10.72663, + "25": 10.73401, + "26": 10.71265, + "27": 10.67569, + "28": 10.6012, + "29": 10.58219, + "30": 10.55702, + "31": 10.56167, + "32": 10.53487, + "33": 10.50348, + "34": 10.46525, + "35": 10.47887, + "36": 10.45601, + "37": 10.41808, + "38": 10.4167, + "39": 10.38178, + "40": 10.37615, + "41": 10.34328, + "42": 10.33161, + "43": 10.3077, + "44": 10.27713, + "45": 10.2988, + "46": 10.25974, + "47": 10.23655, + "48": 10.19378, + "49": 10.18543, + "50": 10.19728, + "51": 10.19367, + "52": 10.14697, + "53": 10.14982, + "54": 10.11733, + "55": 10.08476, + "56": 10.12202, + "57": 10.10466, + "58": 10.11902, + "59": 10.06425, + "60": 10.08087, + "61": 10.03845, + "62": 10.00507, + "63": 10.07872, + "64": 10.03202, + "65": 10.00346, + "66": 10.03491, + "67": 10.00571, + "68": 9.97032, + "69": 9.99153, + "70": 9.97866, + "71": 9.9998, + "72": 9.97531, + "73": 9.96566, + "74": 9.95983, + "75": 9.92549, + "76": 9.96566, + "77": 9.95606, + "78": 9.90446, + "79": 9.91127, + "80": 9.92478, + "81": 9.94432, + "82": 9.88863, + "83": 9.85093, + "84": 9.78899, + "85": 9.77531, + "86": 9.88013, + "87": 9.91309, + "88": 9.88639, + "89": 9.82366, + "90": 9.81326, + "91": 9.82136, + "92": 9.81011, + "93": 9.75216, + "94": 9.82571, + "95": 9.81912, + "96": 9.8055, + "97": 9.7436, + "98": 9.77248, + "99": 9.81661, + "100": 9.71126 + } + }, + "num-zeros": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 1770.0, + "2": 1703.0, + "3": 1799.0, + "4": 1823.0, + "5": 1737.0, + "6": 1733.0, + "7": 1952.0, + "8": 1738.0, + "9": 1687.0, + "10": 1780.0, + "11": 1673.0, + "12": 1677.0, + "13": 1724.0, + "14": 1837.0, + "15": 1570.0, + "16": 1727.0, + "17": 1788.0, + "18": 1780.0, + "19": 1660.0, + "20": 1694.0, + "21": 1732.0, + "22": 1727.0, + "23": 1703.0, + "24": 1799.0, + "25": 1588.0, + "26": 1817.0, + "27": 1684.0, + "28": 1866.0, + "29": 1939.0, + "30": 1970.0, + "31": 1940.0, + "32": 1908.0, + "33": 1954.0, + "34": 2074.0, + "35": 2035.0, + "36": 1969.0, + "37": 2171.0, + "38": 2085.0, + "39": 2307.0, + "40": 2354.0, + "41": 2281.0, + "42": 1999.0, + "43": 2430.0, + "44": 2201.0, + "45": 2656.0, + "46": 2281.0, + "47": 2477.0, + "48": 2552.0, + "49": 2811.0, + "50": 2570.0, + "51": 2414.0, + "52": 2753.0, + "53": 2603.0, + "54": 2894.0, + "55": 2730.0, + "56": 2607.0, + "57": 2214.0, + "58": 3601.0, + "59": 2895.0, + "60": 2853.0, + "61": 2642.0, + "62": 3082.0, + "63": 3315.0, + "64": 3551.0, + "65": 2620.0, + "66": 3064.0, + "67": 3946.0, + "68": 3280.0, + "69": 2902.0, + "70": 3285.0, + "71": 3061.0, + "72": 2994.0, + "73": 3480.0, + "74": 3147.0, + "75": 3252.0, + "76": 3151.0, + "77": 3721.0, + "78": 3131.0, + "79": 3305.0, + "80": 3197.0, + "81": 3396.0, + "82": 3230.0, + "83": 3266.0, + "84": 2830.0, + "85": 2720.0, + "86": 3098.0, + "87": 2969.0, + "88": 3044.0, + "89": 3029.0, + "90": 3586.0, + "91": 3189.0, + "92": 2969.0, + "93": 2896.0, + "94": 3171.0, + "95": 3403.0, + "96": 3597.0, + "97": 3484.0, + "98": 3353.0, + "99": 3320.0, + "100": 3368.0 + } + }, + "mem-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 745146880.0, + "2": 745146880.0, + "3": 745146880.0, + "4": 745146880.0, + "5": 745146880.0, + "6": 745146880.0, + "7": 745146880.0, + "8": 745146880.0, + "9": 745146880.0, + "10": 745146880.0, + "11": 745146880.0, + "12": 745146880.0, + "13": 745146880.0, + "14": 745146880.0, + "15": 745146880.0, + "16": 745146880.0, + "17": 745146880.0, + "18": 745146880.0, + "19": 745146880.0, + "20": 745146880.0, + "21": 745146880.0, + "22": 745146880.0, + "23": 745146880.0, + "24": 745146880.0, + "25": 745146880.0, + "26": 745146880.0, + "27": 745146880.0, + "28": 745146880.0, + "29": 745146880.0, + "30": 745146880.0, + "31": 745146880.0, + "32": 745146880.0, + "33": 745146880.0, + "34": 745146880.0, + "35": 745146880.0, + "36": 745146880.0, + "37": 745146880.0, + "38": 745146880.0, + "39": 745146880.0, + "40": 745146880.0, + "41": 745146880.0, + "42": 745146880.0, + "43": 745146880.0, + "44": 745146880.0, + "45": 745146880.0, + "46": 745146880.0, + "47": 745146880.0, + "48": 745146880.0, + "49": 745146880.0, + "50": 745146880.0, + "51": 745146880.0, + "52": 745146880.0, + "53": 745146880.0, + "54": 745146880.0, + "55": 745146880.0, + "56": 745146880.0, + "57": 745146880.0, + "58": 745146880.0, + "59": 745146880.0, + "60": 745146880.0, + "61": 745146880.0, + "62": 745146880.0, + "63": 745146880.0, + "64": 745146880.0, + "65": 745146880.0, + "66": 745146880.0, + "67": 745146880.0, + "68": 745146880.0, + "69": 745146880.0, + "70": 745146880.0, + "71": 745146880.0, + "72": 745146880.0, + "73": 745146880.0, + "74": 745146880.0, + "75": 745146880.0, + "76": 745146880.0, + "77": 745146880.0, + "78": 745146880.0, + "79": 745146880.0, + "80": 745146880.0, + "81": 745146880.0, + "82": 745146880.0, + "83": 745146880.0, + "84": 745146880.0, + "85": 745146880.0, + "86": 745146880.0, + "87": 745146880.0, + "88": 745146880.0, + "89": 745146880.0, + "90": 745146880.0, + "91": 745146880.0, + "92": 745146880.0, + "93": 745146880.0, + "94": 745146880.0, + "95": 745146880.0, + "96": 745146880.0, + "97": 745146880.0, + "98": 745146880.0, + "99": 745146880.0, + "100": 745146880.0 + } + }, + "mem-max-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 1938739200.0, + "2": 2222436352.0, + "3": 2222436352.0, + "4": 2222436352.0, + "5": 2222436352.0, + "6": 2222436352.0, + "7": 2222436352.0, + "8": 2222436352.0, + "9": 2222436352.0, + "10": 2222436352.0, + "11": 2222436352.0, + "12": 2222436352.0, + "13": 2222436352.0, + "14": 2222436352.0, + "15": 2222436352.0, + "16": 2222436352.0, + "17": 2222436352.0, + "18": 2222436352.0, + "19": 2222436352.0, + "20": 2222436352.0, + "21": 2222436352.0, + "22": 2222436352.0, + "23": 2222436352.0, + "24": 2222436352.0, + "25": 2222436352.0, + "26": 2222436352.0, + "27": 2222436352.0, + "28": 2222436352.0, + "29": 2222436352.0, + "30": 2222436352.0, + "31": 2222436352.0, + "32": 2222436352.0, + "33": 2222436352.0, + "34": 2222436352.0, + "35": 2222436352.0, + "36": 2222436352.0, + "37": 2222436352.0, + "38": 2222436352.0, + "39": 2222436352.0, + "40": 2222436352.0, + "41": 2222436352.0, + "42": 2222436352.0, + "43": 2222436352.0, + "44": 2222436352.0, + "45": 2222436352.0, + "46": 2222436352.0, + "47": 2222436352.0, + "48": 2222436352.0, + "49": 2222436352.0, + "50": 2222436352.0, + "51": 2222436352.0, + "52": 2222436352.0, + "53": 2222436352.0, + "54": 2222436352.0, + "55": 2222436352.0, + "56": 2222436352.0, + "57": 2222436352.0, + "58": 2222436352.0, + "59": 2222436352.0, + "60": 2222436352.0, + "61": 2222436352.0, + "62": 2222436352.0, + "63": 2222436352.0, + "64": 2222436352.0, + "65": 2222436352.0, + "66": 2222436352.0, + "67": 2222436352.0, + "68": 2222436352.0, + "69": 2222436352.0, + "70": 2222436352.0, + "71": 2222436352.0, + "72": 2222436352.0, + "73": 2222436352.0, + "74": 2222436352.0, + "75": 2222436352.0, + "76": 2222436352.0, + "77": 2222436352.0, + "78": 2222436352.0, + "79": 2222436352.0, + "80": 2222436352.0, + "81": 2222436352.0, + "82": 2222436352.0, + "83": 2222436352.0, + "84": 2222436352.0, + "85": 2222436352.0, + "86": 2222436352.0, + "87": 2222436352.0, + "88": 2222436352.0, + "89": 2222436352.0, + "90": 2222436352.0, + "91": 2222436352.0, + "92": 2222436352.0, + "93": 2222436352.0, + "94": 2222436352.0, + "95": 2222436352.0, + "96": 2222436352.0, + "97": 2222436352.0, + "98": 2222436352.0, + "99": 2222436352.0, + "100": 2222436352.0 + } + }, + "iteration-time": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": "nan", + "2": 4.55478, + "3": 2.12224, + "4": 1.88698, + "5": 1.4481, + "6": 1.63569, + "7": 1.48671, + "8": 2.19801, + "9": 2.19503, + "10": 1.87768, + "11": 2.09547, + "12": 1.63219, + "13": 1.89992, + "14": 1.8934, + "15": 1.63055, + "16": 1.66552, + "17": 2.14443, + "18": 1.80768, + "19": 1.89317, + "20": 1.61079, + "21": 1.95956, + "22": 1.4216, + "23": 1.99236, + "24": 1.82538, + "25": 3.01868, + "26": 2.07744, + "27": 1.97852, + "28": 2.00511, + "29": 1.65934, + "30": 1.89307, + "31": 1.55945, + "32": 2.74758, + "33": 1.66337, + "34": 1.53218, + "35": 2.28327, + "36": 2.25217, + "37": 1.78252, + "38": 1.48412, + "39": 2.06124, + "40": 1.68558, + "41": 1.92712, + "42": 2.19365, + "43": 1.90056, + "44": 1.57815, + "45": 1.97887, + "46": 1.65269, + "47": 1.81564, + "48": 1.82219, + "49": 1.53636, + "50": 2.04823, + "51": 0.89436, + "52": 1.97439, + "53": 2.07342, + "54": 2.08431, + "55": 1.8224, + "56": 2.3008, + "57": 1.13755, + "58": 1.90311, + "59": 1.63581, + "60": 1.54188, + "61": 1.97723, + "62": 1.98726, + "63": 1.63993, + "64": 1.79327, + "65": 1.72779, + "66": 2.1385, + "67": 1.98949, + "68": 1.43944, + "69": 2.08961, + "70": 1.84112, + "71": 2.27617, + "72": 1.70558, + "73": 1.91425, + "74": 1.57182, + "75": 1.71805, + "76": 1.99003, + "77": 1.95394, + "78": 1.98389, + "79": 2.2803, + "80": 1.94276, + "81": 2.4261, + "82": 2.30378, + "83": 2.01228, + "84": 1.86139, + "85": 1.74653, + "86": 1.56587, + "87": 2.02734, + "88": 1.86375, + "89": 1.87166, + "90": 1.88278, + "91": 1.85861, + "92": 1.89225, + "93": 1.42961, + "94": 1.96168, + "95": 1.97221, + "96": 1.7668, + "97": 1.73103, + "98": 2.07478, + "99": 2.33331, + "100": 1.84779 + } + } +} \ No newline at end of file diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_disable_bias_linear_1node/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_disable_bias_linear_1node/model_config.yaml new file mode 100644 index 00000000000..305e7eabf98 --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_disable_bias_linear_1node/model_config.yaml @@ -0,0 +1,55 @@ +ENV_VARS: + CUDA_DEVICE_MAX_CONNECTIONS: 1 + NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0 + NCCL_ALGO: Ring + CUBLAS_WORKSPACE_CONFIG: :4096:8 +MODEL_ARGS: + --num-layers: 12 + --hidden-size: 512 + --num-attention-heads: 8 + --log-params-norm: true + --log-num-zeros-in-grad: true + --log-validation-ppl-to-tensorboard: true + --log-timers-to-tensorboard: true + --tensorboard-dir: ${TENSORBOARD_PATH} + --micro-batch-size: 4 + --global-batch-size: 32 + --seq-length: 1024 + --max-position-embeddings: 1024 + --train-iters: 100 + --timing-log-level: 0 + --lr-decay-iters: 320000 + --save: ${CHECKPOINT_SAVE_PATH} + --load: ${CHECKPOINT_LOAD_PATH} + --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document + --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json + --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt + --split: 949,50,1 + --distributed-backend: nccl + --lr: 0.00015 + --lr-decay-style: cosine + --min-lr: 1.0e-5 + --weight-decay: 1e-2 + --clip-grad: 1.0 + --lr-warmup-fraction: .01 + --log-interval: 1 + --save-interval: 50 + --eval-interval: 1000 + --eval-iters: 10 + --transformer-impl: transformer_engine + --tensor-model-parallel-size: 1 + --pipeline-model-parallel-size: 4 + --disable-bias-linear: true + --async-save: true + --deterministic-mode: true + --no-gradient-accumulation-fusion: true + --attention-softmax-in-fp32: true + --use-checkpoint-opt_param-scheduler: true + --use-mcore-models: true + --ckpt-format: torch_dist + --dist-ckpt-strictness: log_all # backward compatibility for TE changes + --data-cache-path: ${DATA_CACHE_PATH} + --bf16: true + --log-memory-to-tensorboard: true + --use-persistent-ckpt-worker: true +TEST_TYPE: ckpt-resume diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_persistent_disable_bias_linear_1node/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_persistent_disable_bias_linear_1node/golden_values_dev_dgx_gb200.json new file mode 100644 index 00000000000..071b5fa17ad --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_persistent_disable_bias_linear_1node/golden_values_dev_dgx_gb200.json @@ -0,0 +1,537 @@ +{ + "lm loss": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 10.92753, + "2": 10.92559, + "3": 10.93114, + "4": 10.93741, + "5": 10.93016, + "6": 10.92355, + "7": 10.93863, + "8": 10.92774, + "9": 10.93076, + "10": 10.92802, + "11": 10.91411, + "12": 10.92298, + "13": 10.92014, + "14": 10.90614, + "15": 10.88338, + "16": 10.87951, + "17": 10.88868, + "18": 10.86451, + "19": 10.87355, + "20": 10.783, + "21": 10.78553, + "22": 10.76764, + "23": 10.764, + "24": 10.72663, + "25": 10.73401, + "26": 10.71265, + "27": 10.67569, + "28": 10.6012, + "29": 10.58219, + "30": 10.55702, + "31": 10.56167, + "32": 10.53487, + "33": 10.50348, + "34": 10.46525, + "35": 10.47887, + "36": 10.45601, + "37": 10.41808, + "38": 10.4167, + "39": 10.38178, + "40": 10.37615, + "41": 10.34328, + "42": 10.33161, + "43": 10.3077, + "44": 10.27713, + "45": 10.2988, + "46": 10.25974, + "47": 10.23655, + "48": 10.19378, + "49": 10.18543, + "50": 10.19728, + "51": 10.19367, + "52": 10.14697, + "53": 10.14982, + "54": 10.11733, + "55": 10.08476, + "56": 10.12202, + "57": 10.10466, + "58": 10.11902, + "59": 10.06425, + "60": 10.08087, + "61": 10.03845, + "62": 10.00507, + "63": 10.07872, + "64": 10.03202, + "65": 10.00346, + "66": 10.03491, + "67": 10.00571, + "68": 9.97032, + "69": 9.99153, + "70": 9.97866, + "71": 9.9998, + "72": 9.97531, + "73": 9.96566, + "74": 9.95983, + "75": 9.92549, + "76": 9.96566, + "77": 9.95606, + "78": 9.90446, + "79": 9.91127, + "80": 9.92478, + "81": 9.94432, + "82": 9.88863, + "83": 9.85093, + "84": 9.78899, + "85": 9.77531, + "86": 9.88013, + "87": 9.91309, + "88": 9.88639, + "89": 9.82366, + "90": 9.81326, + "91": 9.82136, + "92": 9.81011, + "93": 9.75216, + "94": 9.82571, + "95": 9.81912, + "96": 9.8055, + "97": 9.7436, + "98": 9.77248, + "99": 9.81661, + "100": 9.71126 + } + }, + "num-zeros": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 1770.0, + "2": 1703.0, + "3": 1799.0, + "4": 1823.0, + "5": 1737.0, + "6": 1733.0, + "7": 1952.0, + "8": 1738.0, + "9": 1687.0, + "10": 1780.0, + "11": 1673.0, + "12": 1677.0, + "13": 1724.0, + "14": 1837.0, + "15": 1570.0, + "16": 1727.0, + "17": 1788.0, + "18": 1780.0, + "19": 1660.0, + "20": 1694.0, + "21": 1732.0, + "22": 1727.0, + "23": 1703.0, + "24": 1799.0, + "25": 1588.0, + "26": 1817.0, + "27": 1684.0, + "28": 1866.0, + "29": 1939.0, + "30": 1970.0, + "31": 1940.0, + "32": 1908.0, + "33": 1954.0, + "34": 2074.0, + "35": 2035.0, + "36": 1969.0, + "37": 2171.0, + "38": 2085.0, + "39": 2307.0, + "40": 2354.0, + "41": 2281.0, + "42": 1999.0, + "43": 2430.0, + "44": 2201.0, + "45": 2656.0, + "46": 2281.0, + "47": 2477.0, + "48": 2552.0, + "49": 2811.0, + "50": 2570.0, + "51": 2414.0, + "52": 2753.0, + "53": 2603.0, + "54": 2894.0, + "55": 2730.0, + "56": 2607.0, + "57": 2214.0, + "58": 3601.0, + "59": 2895.0, + "60": 2853.0, + "61": 2642.0, + "62": 3082.0, + "63": 3315.0, + "64": 3551.0, + "65": 2620.0, + "66": 3064.0, + "67": 3946.0, + "68": 3280.0, + "69": 2902.0, + "70": 3285.0, + "71": 3061.0, + "72": 2994.0, + "73": 3480.0, + "74": 3147.0, + "75": 3252.0, + "76": 3151.0, + "77": 3721.0, + "78": 3131.0, + "79": 3305.0, + "80": 3197.0, + "81": 3396.0, + "82": 3230.0, + "83": 3266.0, + "84": 2830.0, + "85": 2720.0, + "86": 3098.0, + "87": 2969.0, + "88": 3044.0, + "89": 3029.0, + "90": 3586.0, + "91": 3189.0, + "92": 2969.0, + "93": 2896.0, + "94": 3171.0, + "95": 3403.0, + "96": 3597.0, + "97": 3484.0, + "98": 3353.0, + "99": 3320.0, + "100": 3368.0 + } + }, + "mem-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 745146880.0, + "2": 745146880.0, + "3": 745146880.0, + "4": 745146880.0, + "5": 745146880.0, + "6": 745146880.0, + "7": 745146880.0, + "8": 745146880.0, + "9": 745146880.0, + "10": 745146880.0, + "11": 745146880.0, + "12": 745146880.0, + "13": 745146880.0, + "14": 745146880.0, + "15": 745146880.0, + "16": 745146880.0, + "17": 745146880.0, + "18": 745146880.0, + "19": 745146880.0, + "20": 745146880.0, + "21": 745146880.0, + "22": 745146880.0, + "23": 745146880.0, + "24": 745146880.0, + "25": 745146880.0, + "26": 745146880.0, + "27": 745146880.0, + "28": 745146880.0, + "29": 745146880.0, + "30": 745146880.0, + "31": 745146880.0, + "32": 745146880.0, + "33": 745146880.0, + "34": 745146880.0, + "35": 745146880.0, + "36": 745146880.0, + "37": 745146880.0, + "38": 745146880.0, + "39": 745146880.0, + "40": 745146880.0, + "41": 745146880.0, + "42": 745146880.0, + "43": 745146880.0, + "44": 745146880.0, + "45": 745146880.0, + "46": 745146880.0, + "47": 745146880.0, + "48": 745146880.0, + "49": 745146880.0, + "50": 745146880.0, + "51": 745146880.0, + "52": 745146880.0, + "53": 745146880.0, + "54": 745146880.0, + "55": 745146880.0, + "56": 745146880.0, + "57": 745146880.0, + "58": 745146880.0, + "59": 745146880.0, + "60": 745146880.0, + "61": 745146880.0, + "62": 745146880.0, + "63": 745146880.0, + "64": 745146880.0, + "65": 745146880.0, + "66": 745146880.0, + "67": 745146880.0, + "68": 745146880.0, + "69": 745146880.0, + "70": 745146880.0, + "71": 745146880.0, + "72": 745146880.0, + "73": 745146880.0, + "74": 745146880.0, + "75": 745146880.0, + "76": 745146880.0, + "77": 745146880.0, + "78": 745146880.0, + "79": 745146880.0, + "80": 745146880.0, + "81": 745146880.0, + "82": 745146880.0, + "83": 745146880.0, + "84": 745146880.0, + "85": 745146880.0, + "86": 745146880.0, + "87": 745146880.0, + "88": 745146880.0, + "89": 745146880.0, + "90": 745146880.0, + "91": 745146880.0, + "92": 745146880.0, + "93": 745146880.0, + "94": 745146880.0, + "95": 745146880.0, + "96": 745146880.0, + "97": 745146880.0, + "98": 745146880.0, + "99": 745146880.0, + "100": 745146880.0 + } + }, + "mem-max-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 1938739200.0, + "2": 2222436352.0, + "3": 2222436352.0, + "4": 2222436352.0, + "5": 2222436352.0, + "6": 2222436352.0, + "7": 2222436352.0, + "8": 2222436352.0, + "9": 2222436352.0, + "10": 2222436352.0, + "11": 2222436352.0, + "12": 2222436352.0, + "13": 2222436352.0, + "14": 2222436352.0, + "15": 2222436352.0, + "16": 2222436352.0, + "17": 2222436352.0, + "18": 2222436352.0, + "19": 2222436352.0, + "20": 2222436352.0, + "21": 2222436352.0, + "22": 2222436352.0, + "23": 2222436352.0, + "24": 2222436352.0, + "25": 2222436352.0, + "26": 2222436352.0, + "27": 2222436352.0, + "28": 2222436352.0, + "29": 2222436352.0, + "30": 2222436352.0, + "31": 2222436352.0, + "32": 2222436352.0, + "33": 2222436352.0, + "34": 2222436352.0, + "35": 2222436352.0, + "36": 2222436352.0, + "37": 2222436352.0, + "38": 2222436352.0, + "39": 2222436352.0, + "40": 2222436352.0, + "41": 2222436352.0, + "42": 2222436352.0, + "43": 2222436352.0, + "44": 2222436352.0, + "45": 2222436352.0, + "46": 2222436352.0, + "47": 2222436352.0, + "48": 2222436352.0, + "49": 2222436352.0, + "50": 2222436352.0, + "51": 2222436352.0, + "52": 2222436352.0, + "53": 2222436352.0, + "54": 2222436352.0, + "55": 2222436352.0, + "56": 2222436352.0, + "57": 2222436352.0, + "58": 2222436352.0, + "59": 2222436352.0, + "60": 2222436352.0, + "61": 2222436352.0, + "62": 2222436352.0, + "63": 2222436352.0, + "64": 2222436352.0, + "65": 2222436352.0, + "66": 2222436352.0, + "67": 2222436352.0, + "68": 2222436352.0, + "69": 2222436352.0, + "70": 2222436352.0, + "71": 2222436352.0, + "72": 2222436352.0, + "73": 2222436352.0, + "74": 2222436352.0, + "75": 2222436352.0, + "76": 2222436352.0, + "77": 2222436352.0, + "78": 2222436352.0, + "79": 2222436352.0, + "80": 2222436352.0, + "81": 2222436352.0, + "82": 2222436352.0, + "83": 2222436352.0, + "84": 2222436352.0, + "85": 2222436352.0, + "86": 2222436352.0, + "87": 2222436352.0, + "88": 2222436352.0, + "89": 2222436352.0, + "90": 2222436352.0, + "91": 2222436352.0, + "92": 2222436352.0, + "93": 2222436352.0, + "94": 2222436352.0, + "95": 2222436352.0, + "96": 2222436352.0, + "97": 2222436352.0, + "98": 2222436352.0, + "99": 2222436352.0, + "100": 2222436352.0 + } + }, + "iteration-time": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": "nan", + "2": 4.8239, + "3": 1.49529, + "4": 1.52417, + "5": 1.42926, + "6": 1.35051, + "7": 1.41401, + "8": 1.81254, + "9": 1.71474, + "10": 1.60617, + "11": 1.90952, + "12": 1.3573, + "13": 1.72892, + "14": 1.81053, + "15": 1.11267, + "16": 1.48912, + "17": 2.0212, + "18": 1.58117, + "19": 1.65997, + "20": 1.52494, + "21": 1.6049, + "22": 1.38273, + "23": 1.31657, + "24": 1.63216, + "25": 1.7781, + "26": 1.74546, + "27": 1.46651, + "28": 1.41259, + "29": 1.21659, + "30": 1.24029, + "31": 1.1265, + "32": 1.88094, + "33": 1.20882, + "34": 1.43348, + "35": 1.93841, + "36": 1.87279, + "37": 1.27431, + "38": 1.25399, + "39": 1.6126, + "40": 1.69113, + "41": 1.69312, + "42": 1.90648, + "43": 1.3948, + "44": 1.42326, + "45": 1.88925, + "46": 1.49771, + "47": 1.71084, + "48": 1.74375, + "49": 1.37454, + "50": 1.74703, + "51": 1.08463, + "52": 1.2548, + "53": 1.66332, + "54": 1.6491, + "55": 1.52139, + "56": 1.91286, + "57": 1.16891, + "58": 1.56165, + "59": 1.2747, + "60": 1.20924, + "61": 1.46663, + "62": 1.52802, + "63": 1.32289, + "64": 1.37273, + "65": 1.29883, + "66": 2.06997, + "67": 1.73687, + "68": 1.28257, + "69": 1.75932, + "70": 1.58293, + "71": 2.06564, + "72": 1.3906, + "73": 2.18023, + "74": 1.92128, + "75": 1.50758, + "76": 1.53722, + "77": 1.61446, + "78": 1.75456, + "79": 1.91389, + "80": 1.83397, + "81": 2.10738, + "82": 1.9352, + "83": 1.41118, + "84": 1.35621, + "85": 1.47453, + "86": 1.11847, + "87": 1.76302, + "88": 1.49852, + "89": 1.55278, + "90": 1.70938, + "91": 1.72302, + "92": 1.97837, + "93": 1.42573, + "94": 1.69552, + "95": 1.5947, + "96": 1.84451, + "97": 1.49641, + "98": 1.8176, + "99": 1.98379, + "100": 1.60434 + } + } +} \ No newline at end of file diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_persistent_disable_bias_linear_1node/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_persistent_disable_bias_linear_1node/model_config.yaml new file mode 100644 index 00000000000..3cf79eaf7d2 --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_persistent_disable_bias_linear_1node/model_config.yaml @@ -0,0 +1,55 @@ +ENV_VARS: + CUDA_DEVICE_MAX_CONNECTIONS: 1 + NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0 + NCCL_ALGO: Ring + CUBLAS_WORKSPACE_CONFIG: :4096:8 +MODEL_ARGS: + --num-layers: 12 + --hidden-size: 512 + --num-attention-heads: 8 + --log-params-norm: true + --log-num-zeros-in-grad: true + --log-validation-ppl-to-tensorboard: true + --log-timers-to-tensorboard: true + --tensorboard-dir: ${TENSORBOARD_PATH} + --micro-batch-size: 4 + --global-batch-size: 32 + --seq-length: 1024 + --max-position-embeddings: 1024 + --train-iters: 100 + --timing-log-level: 0 + --lr-decay-iters: 320000 + --save: ${CHECKPOINT_SAVE_PATH} + --load: ${CHECKPOINT_LOAD_PATH} + --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document + --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json + --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt + --split: 949,50,1 + --distributed-backend: nccl + --lr: 0.00015 + --lr-decay-style: cosine + --min-lr: 1.0e-5 + --weight-decay: 1e-2 + --clip-grad: 1.0 + --lr-warmup-fraction: .01 + --log-interval: 1 + --save-interval: 50 + --eval-interval: 1000 + --eval-iters: 10 + --transformer-impl: transformer_engine + --tensor-model-parallel-size: 1 + --pipeline-model-parallel-size: 4 + --disable-bias-linear: true + --async-save: true + --use-persistent-ckpt-worker: true + --deterministic-mode: true + --no-gradient-accumulation-fusion: true + --attention-softmax-in-fp32: true + --use-checkpoint-opt_param-scheduler: true + --use-mcore-models: true + --ckpt-format: torch_dist + --dist-ckpt-optim-fully-reshardable: true + --data-cache-path: ${DATA_CACHE_PATH} + --bf16: true + --log-memory-to-tensorboard: true +TEST_TYPE: ckpt-resume diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_swiglu_1node/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_swiglu_1node/golden_values_dev_dgx_gb200.json new file mode 100644 index 00000000000..9e46a8a4f6e --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_swiglu_1node/golden_values_dev_dgx_gb200.json @@ -0,0 +1,537 @@ +{ + "lm loss": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 10.91715, + "2": 10.91532, + "3": 10.90493, + "4": 10.90847, + "5": 10.90691, + "6": 10.9055, + "7": 10.92235, + "8": 10.90215, + "9": 10.90498, + "10": 10.89605, + "11": 10.90018, + "12": 10.90163, + "13": 10.90201, + "14": 10.88651, + "15": 10.89684, + "16": 10.86925, + "17": 10.87976, + "18": 10.84695, + "19": 10.87292, + "20": 10.82075, + "21": 10.82485, + "22": 10.81217, + "23": 10.80616, + "24": 10.76755, + "25": 10.78042, + "26": 10.76321, + "27": 10.7656, + "28": 10.70637, + "29": 10.68618, + "30": 10.66655, + "31": 10.65301, + "32": 10.64628, + "33": 10.61261, + "34": 10.5899, + "35": 10.58928, + "36": 10.56372, + "37": 10.55028, + "38": 10.55748, + "39": 10.50946, + "40": 10.49801, + "41": 10.46839, + "42": 10.46037, + "43": 10.42389, + "44": 10.40559, + "45": 10.40957, + "46": 10.36962, + "47": 10.36339, + "48": 10.31729, + "49": 10.31454, + "50": 10.31019, + "51": 10.29764, + "52": 10.25606, + "53": 10.24571, + "54": 10.21517, + "55": 10.18762, + "56": 10.21176, + "57": 10.1873, + "58": 10.20309, + "59": 10.15361, + "60": 10.15374, + "61": 10.12073, + "62": 10.07994, + "63": 10.14838, + "64": 10.09518, + "65": 10.06418, + "66": 10.08944, + "67": 10.06057, + "68": 10.02184, + "69": 10.03115, + "70": 10.02659, + "71": 10.0429, + "72": 10.0198, + "73": 9.99838, + "74": 9.98812, + "75": 9.9503, + "76": 10.00097, + "77": 9.99194, + "78": 9.93894, + "79": 9.9475, + "80": 9.95199, + "81": 9.97687, + "82": 9.93072, + "83": 9.89056, + "84": 9.82765, + "85": 9.81182, + "86": 9.9222, + "87": 9.94021, + "88": 9.9185, + "89": 9.85561, + "90": 9.85341, + "91": 9.86676, + "92": 9.85465, + "93": 9.79188, + "94": 9.86491, + "95": 9.86283, + "96": 9.84804, + "97": 9.78916, + "98": 9.82351, + "99": 9.86502, + "100": 9.76301 + } + }, + "num-zeros": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 2974.0, + "2": 3007.0, + "3": 2774.0, + "4": 2880.0, + "5": 2917.0, + "6": 2797.0, + "7": 2954.0, + "8": 2881.0, + "9": 3016.0, + "10": 3002.0, + "11": 3025.0, + "12": 2876.0, + "13": 3098.0, + "14": 3056.0, + "15": 2793.0, + "16": 2980.0, + "17": 2895.0, + "18": 2749.0, + "19": 3152.0, + "20": 2925.0, + "21": 2901.0, + "22": 2845.0, + "23": 2802.0, + "24": 2832.0, + "25": 3001.0, + "26": 2853.0, + "27": 2791.0, + "28": 2843.0, + "29": 2785.0, + "30": 2711.0, + "31": 2699.0, + "32": 2647.0, + "33": 2622.0, + "34": 2823.0, + "35": 2677.0, + "36": 2661.0, + "37": 2791.0, + "38": 2749.0, + "39": 2543.0, + "40": 2735.0, + "41": 2547.0, + "42": 2688.0, + "43": 2594.0, + "44": 2635.0, + "45": 2613.0, + "46": 2635.0, + "47": 2571.0, + "48": 2605.0, + "49": 2710.0, + "50": 2619.0, + "51": 2728.0, + "52": 2683.0, + "53": 2654.0, + "54": 2822.0, + "55": 2512.0, + "56": 2942.0, + "57": 2461.0, + "58": 3258.0, + "59": 2858.0, + "60": 2770.0, + "61": 2863.0, + "62": 2811.0, + "63": 3004.0, + "64": 3008.0, + "65": 2584.0, + "66": 2961.0, + "67": 3223.0, + "68": 2963.0, + "69": 3022.0, + "70": 2992.0, + "71": 2761.0, + "72": 3101.0, + "73": 3109.0, + "74": 2917.0, + "75": 2949.0, + "76": 3185.0, + "77": 3265.0, + "78": 3091.0, + "79": 3067.0, + "80": 3041.0, + "81": 3235.0, + "82": 3487.0, + "83": 3187.0, + "84": 3089.0, + "85": 2761.0, + "86": 3019.0, + "87": 3151.0, + "88": 3283.0, + "89": 3203.0, + "90": 3469.0, + "91": 2839.0, + "92": 3056.0, + "93": 3055.0, + "94": 3245.0, + "95": 3213.0, + "96": 3280.0, + "97": 3508.0, + "98": 3586.0, + "99": 3138.0, + "100": 3337.0 + } + }, + "mem-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 745077248.0, + "2": 745077248.0, + "3": 745077248.0, + "4": 745077248.0, + "5": 745077248.0, + "6": 745077248.0, + "7": 745077248.0, + "8": 745077248.0, + "9": 745077248.0, + "10": 745077248.0, + "11": 745077248.0, + "12": 745077248.0, + "13": 745077248.0, + "14": 745077248.0, + "15": 745077248.0, + "16": 745077248.0, + "17": 745077248.0, + "18": 745077248.0, + "19": 745077248.0, + "20": 745077248.0, + "21": 745077248.0, + "22": 745077248.0, + "23": 745077248.0, + "24": 745077248.0, + "25": 745077248.0, + "26": 745077248.0, + "27": 745077248.0, + "28": 745077248.0, + "29": 745077248.0, + "30": 745077248.0, + "31": 745077248.0, + "32": 745077248.0, + "33": 745077248.0, + "34": 745077248.0, + "35": 745077248.0, + "36": 745077248.0, + "37": 745077248.0, + "38": 745077248.0, + "39": 745077248.0, + "40": 745077248.0, + "41": 745077248.0, + "42": 745077248.0, + "43": 745077248.0, + "44": 745077248.0, + "45": 745077248.0, + "46": 745077248.0, + "47": 745077248.0, + "48": 745077248.0, + "49": 745077248.0, + "50": 745077248.0, + "51": 745077248.0, + "52": 745077248.0, + "53": 745077248.0, + "54": 745077248.0, + "55": 745077248.0, + "56": 745077248.0, + "57": 745077248.0, + "58": 745077248.0, + "59": 745077248.0, + "60": 745077248.0, + "61": 745077248.0, + "62": 745077248.0, + "63": 745077248.0, + "64": 745077248.0, + "65": 745077248.0, + "66": 745077248.0, + "67": 745077248.0, + "68": 745077248.0, + "69": 745077248.0, + "70": 745077248.0, + "71": 745077248.0, + "72": 745077248.0, + "73": 745077248.0, + "74": 745077248.0, + "75": 745077248.0, + "76": 745077248.0, + "77": 745077248.0, + "78": 745077248.0, + "79": 745077248.0, + "80": 745077248.0, + "81": 745077248.0, + "82": 745077248.0, + "83": 745077248.0, + "84": 745077248.0, + "85": 745077248.0, + "86": 745077248.0, + "87": 745077248.0, + "88": 745077248.0, + "89": 745077248.0, + "90": 745077248.0, + "91": 745077248.0, + "92": 745077248.0, + "93": 745077248.0, + "94": 745077248.0, + "95": 745077248.0, + "96": 745077248.0, + "97": 745077248.0, + "98": 745077248.0, + "99": 745077248.0, + "100": 745077248.0 + } + }, + "mem-max-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 1939395584.0, + "2": 2220400640.0, + "3": 2220400640.0, + "4": 2220400640.0, + "5": 2220400640.0, + "6": 2220400640.0, + "7": 2220400640.0, + "8": 2220400640.0, + "9": 2220400640.0, + "10": 2220400640.0, + "11": 2220400640.0, + "12": 2220400640.0, + "13": 2220400640.0, + "14": 2220400640.0, + "15": 2220400640.0, + "16": 2220400640.0, + "17": 2220400640.0, + "18": 2220400640.0, + "19": 2220400640.0, + "20": 2220400640.0, + "21": 2220400640.0, + "22": 2220400640.0, + "23": 2220400640.0, + "24": 2220400640.0, + "25": 2220400640.0, + "26": 2220400640.0, + "27": 2220400640.0, + "28": 2220400640.0, + "29": 2220400640.0, + "30": 2220400640.0, + "31": 2220400640.0, + "32": 2220400640.0, + "33": 2220400640.0, + "34": 2220400640.0, + "35": 2220400640.0, + "36": 2220400640.0, + "37": 2220400640.0, + "38": 2220400640.0, + "39": 2220400640.0, + "40": 2220400640.0, + "41": 2220400640.0, + "42": 2220400640.0, + "43": 2220400640.0, + "44": 2220400640.0, + "45": 2220400640.0, + "46": 2220400640.0, + "47": 2220400640.0, + "48": 2220400640.0, + "49": 2220400640.0, + "50": 2220400640.0, + "51": 2220400640.0, + "52": 2220400640.0, + "53": 2220400640.0, + "54": 2220400640.0, + "55": 2220400640.0, + "56": 2220400640.0, + "57": 2220400640.0, + "58": 2220400640.0, + "59": 2220400640.0, + "60": 2220400640.0, + "61": 2220400640.0, + "62": 2220400640.0, + "63": 2220400640.0, + "64": 2220400640.0, + "65": 2220400640.0, + "66": 2220400640.0, + "67": 2220400640.0, + "68": 2220400640.0, + "69": 2220400640.0, + "70": 2220400640.0, + "71": 2220400640.0, + "72": 2220400640.0, + "73": 2220400640.0, + "74": 2220400640.0, + "75": 2220400640.0, + "76": 2220400640.0, + "77": 2220400640.0, + "78": 2220400640.0, + "79": 2220400640.0, + "80": 2220400640.0, + "81": 2220400640.0, + "82": 2220400640.0, + "83": 2220400640.0, + "84": 2220400640.0, + "85": 2220400640.0, + "86": 2220400640.0, + "87": 2220400640.0, + "88": 2220400640.0, + "89": 2220400640.0, + "90": 2220400640.0, + "91": 2220400640.0, + "92": 2220400640.0, + "93": 2220400640.0, + "94": 2220400640.0, + "95": 2220400640.0, + "96": 2220400640.0, + "97": 2220400640.0, + "98": 2220400640.0, + "99": 2220400640.0, + "100": 2220400640.0 + } + }, + "iteration-time": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": "nan", + "2": 6.49768, + "3": 1.78821, + "4": 1.31662, + "5": 1.19483, + "6": 1.32002, + "7": 1.19501, + "8": 1.95318, + "9": 1.86166, + "10": 1.39713, + "11": 1.59592, + "12": 1.25518, + "13": 1.49537, + "14": 2.04054, + "15": 1.56473, + "16": 1.57388, + "17": 2.03659, + "18": 1.44582, + "19": 1.40341, + "20": 1.43802, + "21": 1.57772, + "22": 1.32866, + "23": 1.41581, + "24": 1.48365, + "25": 1.9315, + "26": 2.00407, + "27": 1.31987, + "28": 1.21824, + "29": 1.23524, + "30": 1.42625, + "31": 1.18426, + "32": 2.18566, + "33": 1.21912, + "34": 1.26818, + "35": 1.72461, + "36": 1.99265, + "37": 1.64633, + "38": 1.43456, + "39": 1.78121, + "40": 1.74824, + "41": 1.76074, + "42": 2.03061, + "43": 1.62949, + "44": 1.40162, + "45": 2.01929, + "46": 1.52463, + "47": 1.52912, + "48": 1.64357, + "49": 1.32553, + "50": 1.62875, + "51": 1.42699, + "52": 1.7505, + "53": 1.56022, + "54": 2.1245, + "55": 1.76746, + "56": 2.10262, + "57": 1.09509, + "58": 2.00918, + "59": 2.27663, + "60": 1.95187, + "61": 1.86576, + "62": 1.70876, + "63": 1.05715, + "64": 1.69941, + "65": 1.5731, + "66": 1.6839, + "67": 1.35012, + "68": 1.48772, + "69": 1.28276, + "70": 1.77781, + "71": 1.98693, + "72": 1.24692, + "73": 1.56742, + "74": 1.47728, + "75": 1.29539, + "76": 1.42729, + "77": 1.36865, + "78": 1.32401, + "79": 1.95447, + "80": 1.51188, + "81": 1.61243, + "82": 1.89444, + "83": 1.26901, + "84": 1.18003, + "85": 1.48986, + "86": 1.57861, + "87": 1.9783, + "88": 1.88564, + "89": 1.47677, + "90": 1.65885, + "91": 1.48929, + "92": 1.87662, + "93": 1.36189, + "94": 1.30314, + "95": 1.94435, + "96": 1.70256, + "97": 1.37501, + "98": 1.71407, + "99": 2.408, + "100": 1.67129 + } + } +} \ No newline at end of file diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_swiglu_1node/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_swiglu_1node/model_config.yaml new file mode 100644 index 00000000000..a39ef7f4f78 --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_swiglu_1node/model_config.yaml @@ -0,0 +1,56 @@ +ENV_VARS: + CUDA_DEVICE_MAX_CONNECTIONS: 1 + NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0 + NCCL_ALGO: Ring + CUBLAS_WORKSPACE_CONFIG: :4096:8 +MODEL_ARGS: + --num-layers: 12 + --hidden-size: 512 + --num-attention-heads: 8 + --log-params-norm: true + --log-num-zeros-in-grad: true + --log-validation-ppl-to-tensorboard: true + --log-timers-to-tensorboard: true + --tensorboard-dir: ${TENSORBOARD_PATH} + --micro-batch-size: 4 + --global-batch-size: 32 + --seq-length: 1024 + --max-position-embeddings: 1024 + --train-iters: 100 + --timing-log-level: 0 + --lr-decay-iters: 320000 + --save: ${CHECKPOINT_SAVE_PATH} + --load: ${CHECKPOINT_LOAD_PATH} + --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document + --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json + --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt + --split: 949,50,1 + --distributed-backend: nccl + --lr: 0.00015 + --lr-decay-style: cosine + --min-lr: 1.0e-5 + --weight-decay: 1e-2 + --clip-grad: 1.0 + --lr-warmup-fraction: .01 + --log-interval: 1 + --save-interval: 50 + --eval-interval: 1000 + --eval-iters: 10 + --transformer-impl: transformer_engine + --tensor-model-parallel-size: 1 + --pipeline-model-parallel-size: 4 + --swiglu: true + --ckpt-fully-parallel-load: true + --async-save: true + --deterministic-mode: true + --no-gradient-accumulation-fusion: true + --attention-softmax-in-fp32: true + --use-checkpoint-opt_param-scheduler: true + --use-mcore-models: true + --ckpt-format: torch_dist + --dist-ckpt-strictness: log_all # backward compatibility for TE changes + --data-cache-path: ${DATA_CACHE_PATH} + --bf16: true + --log-memory-to-tensorboard: true + --use-persistent-ckpt-worker: true +TEST_TYPE: ckpt-resume diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_untie_embeddings_and_outputs_1node/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_untie_embeddings_and_outputs_1node/golden_values_dev_dgx_gb200.json new file mode 100644 index 00000000000..b6f8a75f92d --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_untie_embeddings_and_outputs_1node/golden_values_dev_dgx_gb200.json @@ -0,0 +1,537 @@ +{ + "lm loss": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 10.94455, + "2": 10.94743, + "3": 10.94818, + "4": 10.94206, + "5": 10.95026, + "6": 10.94824, + "7": 10.93493, + "8": 10.94753, + "9": 10.93762, + "10": 10.94101, + "11": 10.93294, + "12": 10.94364, + "13": 10.90622, + "14": 10.9172, + "15": 10.89665, + "16": 10.89474, + "17": 10.88785, + "18": 10.86864, + "19": 10.87541, + "20": 10.81655, + "21": 10.78013, + "22": 10.77103, + "23": 10.76886, + "24": 10.74831, + "25": 10.74345, + "26": 10.72258, + "27": 10.6734, + "28": 10.60315, + "29": 10.57246, + "30": 10.56303, + "31": 10.54422, + "32": 10.53105, + "33": 10.48751, + "34": 10.47659, + "35": 10.45645, + "36": 10.44068, + "37": 10.40765, + "38": 10.40511, + "39": 10.36532, + "40": 10.3672, + "41": 10.33527, + "42": 10.30512, + "43": 10.28352, + "44": 10.255, + "45": 10.2688, + "46": 10.22196, + "47": 10.20823, + "48": 10.18878, + "49": 10.17404, + "50": 10.1681, + "51": 10.16394, + "52": 10.12751, + "53": 10.12716, + "54": 10.09558, + "55": 10.06465, + "56": 10.07494, + "57": 10.07258, + "58": 10.08235, + "59": 10.02218, + "60": 10.04248, + "61": 10.00366, + "62": 9.97281, + "63": 10.05249, + "64": 10.00735, + "65": 9.98001, + "66": 9.99713, + "67": 9.96053, + "68": 9.93587, + "69": 9.9726, + "70": 9.94511, + "71": 9.96529, + "72": 9.94714, + "73": 9.92131, + "74": 9.91335, + "75": 9.88321, + "76": 9.92543, + "77": 9.90561, + "78": 9.86953, + "79": 9.86598, + "80": 9.87842, + "81": 9.89412, + "82": 9.82675, + "83": 9.80186, + "84": 9.74788, + "85": 9.72843, + "86": 9.84286, + "87": 9.85715, + "88": 9.83812, + "89": 9.77531, + "90": 9.76324, + "91": 9.78325, + "92": 9.75571, + "93": 9.70188, + "94": 9.77557, + "95": 9.76597, + "96": 9.7691, + "97": 9.70025, + "98": 9.72317, + "99": 9.76326, + "100": 9.67042 + } + }, + "num-zeros": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 22961734.0, + "2": 22849292.0, + "3": 22709450.0, + "4": 22778272.0, + "5": 22783344.0, + "6": 22745688.0, + "7": 22871616.0, + "8": 22611788.0, + "9": 22758488.0, + "10": 22489664.0, + "11": 22753930.0, + "12": 22640498.0, + "13": 23322660.0, + "14": 22991940.0, + "15": 22718150.0, + "16": 22822544.0, + "17": 22929420.0, + "18": 22995424.0, + "19": 23084828.0, + "20": 22723088.0, + "21": 22918220.0, + "22": 22945164.0, + "23": 22627484.0, + "24": 22862724.0, + "25": 22636220.0, + "26": 23004960.0, + "27": 22801468.0, + "28": 22999412.0, + "29": 22979382.0, + "30": 22943964.0, + "31": 22908184.0, + "32": 22665988.0, + "33": 22741296.0, + "34": 23076296.0, + "35": 22750520.0, + "36": 22695812.0, + "37": 23103140.0, + "38": 22964608.0, + "39": 22985704.0, + "40": 22749862.0, + "41": 23065512.0, + "42": 22687840.0, + "43": 22987484.0, + "44": 22704888.0, + "45": 22846928.0, + "46": 22734288.0, + "47": 22849852.0, + "48": 22833504.0, + "49": 22888668.0, + "50": 22644700.0, + "51": 22698624.0, + "52": 22812560.0, + "53": 22962504.0, + "54": 22786684.0, + "55": 22922392.0, + "56": 22660110.0, + "57": 23192904.0, + "58": 22688988.0, + "59": 22841898.0, + "60": 23013408.0, + "61": 22673272.0, + "62": 22732280.0, + "63": 22631808.0, + "64": 22999252.0, + "65": 23198302.0, + "66": 22694020.0, + "67": 22955048.0, + "68": 22924488.0, + "69": 23149376.0, + "70": 22822536.0, + "71": 22740440.0, + "72": 23117448.0, + "73": 23133530.0, + "74": 22941076.0, + "75": 22880362.0, + "76": 22702896.0, + "77": 22980008.0, + "78": 22977106.0, + "79": 22824536.0, + "80": 22930748.0, + "81": 22834604.0, + "82": 22731260.0, + "83": 22733720.0, + "84": 23099270.0, + "85": 22921714.0, + "86": 23069020.0, + "87": 22375336.0, + "88": 22555520.0, + "89": 22727918.0, + "90": 22765510.0, + "91": 22918544.0, + "92": 22672096.0, + "93": 22645440.0, + "94": 23128324.0, + "95": 22694938.0, + "96": 22845282.0, + "97": 22832220.0, + "98": 22874074.0, + "99": 22636662.0, + "100": 23000632.0 + } + }, + "mem-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 746444288.0, + "2": 746444288.0, + "3": 746444288.0, + "4": 746444288.0, + "5": 746444288.0, + "6": 746444288.0, + "7": 746444288.0, + "8": 746444288.0, + "9": 746444288.0, + "10": 746444288.0, + "11": 746444288.0, + "12": 746444288.0, + "13": 746444288.0, + "14": 746444288.0, + "15": 746444288.0, + "16": 746444288.0, + "17": 746444288.0, + "18": 746444288.0, + "19": 746444288.0, + "20": 746444288.0, + "21": 746444288.0, + "22": 746444288.0, + "23": 746444288.0, + "24": 746444288.0, + "25": 746444288.0, + "26": 746444288.0, + "27": 746444288.0, + "28": 746444288.0, + "29": 746444288.0, + "30": 746444288.0, + "31": 746444288.0, + "32": 746444288.0, + "33": 746444288.0, + "34": 746444288.0, + "35": 746444288.0, + "36": 746444288.0, + "37": 746444288.0, + "38": 746444288.0, + "39": 746444288.0, + "40": 746444288.0, + "41": 746444288.0, + "42": 746444288.0, + "43": 746444288.0, + "44": 746444288.0, + "45": 746444288.0, + "46": 746444288.0, + "47": 746444288.0, + "48": 746444288.0, + "49": 746444288.0, + "50": 746444288.0, + "51": 746444288.0, + "52": 746444288.0, + "53": 746444288.0, + "54": 746444288.0, + "55": 746444288.0, + "56": 746444288.0, + "57": 746444288.0, + "58": 746444288.0, + "59": 746444288.0, + "60": 746444288.0, + "61": 746444288.0, + "62": 746444288.0, + "63": 746444288.0, + "64": 746444288.0, + "65": 746444288.0, + "66": 746444288.0, + "67": 746444288.0, + "68": 746444288.0, + "69": 746444288.0, + "70": 746444288.0, + "71": 746444288.0, + "72": 746444288.0, + "73": 746444288.0, + "74": 746444288.0, + "75": 746444288.0, + "76": 746444288.0, + "77": 746444288.0, + "78": 746444288.0, + "79": 746444288.0, + "80": 746444288.0, + "81": 746444288.0, + "82": 746444288.0, + "83": 746444288.0, + "84": 746444288.0, + "85": 746444288.0, + "86": 746444288.0, + "87": 746444288.0, + "88": 746444288.0, + "89": 746444288.0, + "90": 746444288.0, + "91": 746444288.0, + "92": 746444288.0, + "93": 746444288.0, + "94": 746444288.0, + "95": 746444288.0, + "96": 746444288.0, + "97": 746444288.0, + "98": 746444288.0, + "99": 746444288.0, + "100": 746444288.0 + } + }, + "mem-max-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 1938877440.0, + "2": 2222686208.0, + "3": 2222686208.0, + "4": 2222686208.0, + "5": 2222686208.0, + "6": 2222686208.0, + "7": 2222686208.0, + "8": 2222686208.0, + "9": 2222686208.0, + "10": 2222686208.0, + "11": 2222686208.0, + "12": 2222686208.0, + "13": 2222686208.0, + "14": 2222686208.0, + "15": 2222686208.0, + "16": 2222686208.0, + "17": 2222686208.0, + "18": 2222686208.0, + "19": 2222686208.0, + "20": 2222686208.0, + "21": 2222686208.0, + "22": 2222686208.0, + "23": 2222686208.0, + "24": 2222686208.0, + "25": 2222686208.0, + "26": 2222686208.0, + "27": 2222686208.0, + "28": 2222686208.0, + "29": 2222686208.0, + "30": 2222686208.0, + "31": 2222686208.0, + "32": 2222686208.0, + "33": 2222686208.0, + "34": 2222686208.0, + "35": 2222686208.0, + "36": 2222686208.0, + "37": 2222686208.0, + "38": 2222686208.0, + "39": 2222686208.0, + "40": 2222686208.0, + "41": 2222686208.0, + "42": 2222686208.0, + "43": 2222686208.0, + "44": 2222686208.0, + "45": 2222686208.0, + "46": 2222686208.0, + "47": 2222686208.0, + "48": 2222686208.0, + "49": 2222686208.0, + "50": 2222686208.0, + "51": 2222686208.0, + "52": 2222686208.0, + "53": 2222686208.0, + "54": 2222686208.0, + "55": 2222686208.0, + "56": 2222686208.0, + "57": 2222686208.0, + "58": 2222686208.0, + "59": 2222686208.0, + "60": 2222686208.0, + "61": 2222686208.0, + "62": 2222686208.0, + "63": 2222686208.0, + "64": 2222686208.0, + "65": 2222686208.0, + "66": 2222686208.0, + "67": 2222686208.0, + "68": 2222686208.0, + "69": 2222686208.0, + "70": 2222686208.0, + "71": 2222686208.0, + "72": 2222686208.0, + "73": 2222686208.0, + "74": 2222686208.0, + "75": 2222686208.0, + "76": 2222686208.0, + "77": 2222686208.0, + "78": 2222686208.0, + "79": 2222686208.0, + "80": 2222686208.0, + "81": 2222686208.0, + "82": 2222686208.0, + "83": 2222686208.0, + "84": 2222686208.0, + "85": 2222686208.0, + "86": 2222686208.0, + "87": 2222686208.0, + "88": 2222686208.0, + "89": 2222686208.0, + "90": 2222686208.0, + "91": 2222686208.0, + "92": 2222686208.0, + "93": 2222686208.0, + "94": 2222686208.0, + "95": 2222686208.0, + "96": 2222686208.0, + "97": 2222686208.0, + "98": 2222686208.0, + "99": 2222686208.0, + "100": 2222686208.0 + } + }, + "iteration-time": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": "nan", + "2": 5.78729, + "3": 2.24593, + "4": 1.83183, + "5": 1.57676, + "6": 1.59317, + "7": 1.44459, + "8": 2.05369, + "9": 2.11079, + "10": 1.75987, + "11": 2.02062, + "12": 1.58261, + "13": 1.87239, + "14": 1.7855, + "15": 1.40992, + "16": 1.65375, + "17": 2.03396, + "18": 1.46988, + "19": 1.53364, + "20": 1.77488, + "21": 1.83552, + "22": 1.25109, + "23": 2.07946, + "24": 1.58607, + "25": 1.86531, + "26": 1.8462, + "27": 1.38898, + "28": 1.66522, + "29": 1.30907, + "30": 1.57189, + "31": 0.98475, + "32": 2.24937, + "33": 1.32033, + "34": 1.45553, + "35": 2.08489, + "36": 2.0601, + "37": 1.84359, + "38": 1.32724, + "39": 1.94447, + "40": 1.56725, + "41": 1.68103, + "42": 1.95667, + "43": 1.72618, + "44": 1.45886, + "45": 1.65175, + "46": 1.18404, + "47": 1.45046, + "48": 1.85105, + "49": 1.47348, + "50": 1.93026, + "51": 0.81202, + "52": 1.59001, + "53": 1.76018, + "54": 1.81656, + "55": 1.54517, + "56": 1.93145, + "57": 0.91332, + "58": 1.72071, + "59": 1.41859, + "60": 1.49901, + "61": 1.66662, + "62": 2.08743, + "63": 1.47041, + "64": 1.66759, + "65": 1.64256, + "66": 1.83296, + "67": 1.60697, + "68": 1.11002, + "69": 2.15917, + "70": 1.67762, + "71": 1.78505, + "72": 1.82282, + "73": 1.8359, + "74": 1.26679, + "75": 1.71494, + "76": 1.79402, + "77": 1.60458, + "78": 1.69908, + "79": 1.91726, + "80": 1.78962, + "81": 1.95792, + "82": 1.9786, + "83": 1.73791, + "84": 1.48612, + "85": 1.53875, + "86": 1.33002, + "87": 2.06743, + "88": 1.95336, + "89": 2.0672, + "90": 1.82135, + "91": 2.00225, + "92": 2.05255, + "93": 1.51115, + "94": 1.82912, + "95": 1.94449, + "96": 1.80767, + "97": 1.51291, + "98": 1.75561, + "99": 2.12919, + "100": 1.66687 + } + } +} \ No newline at end of file diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_untie_embeddings_and_outputs_1node/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_untie_embeddings_and_outputs_1node/model_config.yaml new file mode 100644 index 00000000000..5d1a4257402 --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_resume_torch_dist_untie_embeddings_and_outputs_1node/model_config.yaml @@ -0,0 +1,55 @@ +ENV_VARS: + CUDA_DEVICE_MAX_CONNECTIONS: 1 + NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0 + NCCL_ALGO: Ring + CUBLAS_WORKSPACE_CONFIG: :4096:8 +MODEL_ARGS: + --num-layers: 12 + --hidden-size: 512 + --num-attention-heads: 8 + --log-params-norm: true + --log-num-zeros-in-grad: true + --log-validation-ppl-to-tensorboard: true + --log-timers-to-tensorboard: true + --tensorboard-dir: ${TENSORBOARD_PATH} + --micro-batch-size: 4 + --global-batch-size: 32 + --seq-length: 1024 + --max-position-embeddings: 1024 + --train-iters: 100 + --timing-log-level: 0 + --lr-decay-iters: 320000 + --save: ${CHECKPOINT_SAVE_PATH} + --load: ${CHECKPOINT_LOAD_PATH} + --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document + --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json + --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt + --split: 949,50,1 + --distributed-backend: nccl + --lr: 0.00015 + --lr-decay-style: cosine + --min-lr: 1.0e-5 + --weight-decay: 1e-2 + --clip-grad: 1.0 + --lr-warmup-fraction: .01 + --log-interval: 1 + --save-interval: 50 + --eval-interval: 1000 + --eval-iters: 10 + --transformer-impl: transformer_engine + --tensor-model-parallel-size: 1 + --pipeline-model-parallel-size: 4 + --untie-embeddings-and-output-weights: true + --deterministic-mode: true + --no-gradient-accumulation-fusion: true + --attention-softmax-in-fp32: true + --use-checkpoint-opt_param-scheduler: true + --use-mcore-models: true + --ckpt-format: torch_dist + --dist-ckpt-strictness: log_all # backward compatibility for TE changes + --data-cache-path: ${DATA_CACHE_PATH} + --bf16: true + --log-memory-to-tensorboard: true + --async-save: true + --use-persistent-ckpt-worker: true +TEST_TYPE: ckpt-resume diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_1node/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_1node/model_config.yaml new file mode 100644 index 00000000000..8d764f5a87a --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_1node/model_config.yaml @@ -0,0 +1,57 @@ +ENV_VARS: + CUDA_DEVICE_MAX_CONNECTIONS: 1 + NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0 + NCCL_ALGO: Ring + CUBLAS_WORKSPACE_CONFIG: :4096:8 + PYTORCH_CUDA_ALLOC_CONF: expandable_segments:True +MODEL_ARGS: + --num-layers: 12 + --hidden-size: 512 + --num-attention-heads: 8 + --log-params-norm: true + --log-num-zeros-in-grad: true + --log-validation-ppl-to-tensorboard: true + --log-timers-to-tensorboard: true + --tensorboard-dir: ${TENSORBOARD_PATH} + --micro-batch-size: 4 + --global-batch-size: 32 + --seq-length: 1024 + --max-position-embeddings: 1024 + --train-iters: 50 + --timing-log-level: 0 + --lr-decay-iters: 320000 + --save: ${CHECKPOINT_SAVE_PATH} + --load: ${CHECKPOINT_LOAD_PATH} + --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document + --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json + --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt + --split: 949,50,1 + --distributed-backend: nccl + --lr: 0.00015 + --lr-decay-style: cosine + --min-lr: 1.0e-5 + --weight-decay: 1e-2 + --clip-grad: 1.0 + --lr-warmup-fraction: .01 + --log-interval: 1 + --save-interval: 10000 + --eval-interval: 1000 + --eval-iters: 10 + --transformer-impl: transformer_engine + --tensor-model-parallel-size: 1 + --pipeline-model-parallel-size: 4 + --num-layers-per-virtual-pipeline-stage: 1 + --deterministic-mode: true + --no-gradient-accumulation-fusion: true + --attention-softmax-in-fp32: true + --use-mcore-models: true + --ckpt-format: torch_dist + --dist-ckpt-optim-fully-reshardable: true + --dist-ckpt-strictness: log_all # backward compatibility for TE changes + --data-cache-path: ${DATA_CACHE_PATH} + --bf16: true + --attention-backend: unfused + --log-memory-to-tensorboard: true + --async-save: true + --use-persistent-ckpt-worker: true +TEST_TYPE: regular diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_dist_optimizer_overlap_grad_reduce_param_gather_overlap_optimizer_1node/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_dist_optimizer_overlap_grad_reduce_param_gather_overlap_optimizer_1node/golden_values_dev_dgx_gb200.json new file mode 100644 index 00000000000..93d7bced52f --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_dist_optimizer_overlap_grad_reduce_param_gather_overlap_optimizer_1node/golden_values_dev_dgx_gb200.json @@ -0,0 +1,287 @@ +{ + "lm loss": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 10.93805, + "2": 10.93722, + "3": 10.93622, + "4": 10.94202, + "5": 10.93341, + "6": 10.93239, + "7": 10.9378, + "8": 10.93336, + "9": 10.93523, + "10": 10.93708, + "11": 10.92541, + "12": 10.92478, + "13": 10.92498, + "14": 10.913, + "15": 10.89303, + "16": 10.88848, + "17": 10.89973, + "18": 10.87744, + "19": 10.8763, + "20": 10.79072, + "21": 10.78034, + "22": 10.77489, + "23": 10.77493, + "24": 10.73628, + "25": 10.74375, + "26": 10.72423, + "27": 10.69602, + "28": 10.62335, + "29": 10.60318, + "30": 10.58016, + "31": 10.56934, + "32": 10.54832, + "33": 10.51985, + "34": 10.48553, + "35": 10.4921, + "36": 10.47101, + "37": 10.43653, + "38": 10.44075, + "39": 10.40247, + "40": 10.39246, + "41": 10.36838, + "42": 10.34589, + "43": 10.32638, + "44": 10.28949, + "45": 10.30803, + "46": 10.26792, + "47": 10.25248, + "48": 10.20356, + "49": 10.20108, + "50": 10.20926 + } + }, + "num-zeros": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 1793.0, + "2": 1843.0, + "3": 1760.0, + "4": 1725.0, + "5": 1766.0, + "6": 1737.0, + "7": 2008.0, + "8": 1734.0, + "9": 1810.0, + "10": 1808.0, + "11": 1609.0, + "12": 1629.0, + "13": 1860.0, + "14": 1810.0, + "15": 1650.0, + "16": 1834.0, + "17": 1793.0, + "18": 1785.0, + "19": 1678.0, + "20": 1648.0, + "21": 1762.0, + "22": 1718.0, + "23": 1666.0, + "24": 1852.0, + "25": 1701.0, + "26": 1848.0, + "27": 1834.0, + "28": 1871.0, + "29": 2032.0, + "30": 1892.0, + "31": 2038.0, + "32": 1994.0, + "33": 2040.0, + "34": 2067.0, + "35": 2022.0, + "36": 1876.0, + "37": 2201.0, + "38": 2209.0, + "39": 2128.0, + "40": 2321.0, + "41": 2376.0, + "42": 1900.0, + "43": 2332.0, + "44": 2199.0, + "45": 2583.0, + "46": 2490.0, + "47": 2430.0, + "48": 2588.0, + "49": 2814.0, + "50": 2547.0 + } + }, + "mem-allocated-bytes": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 763730432.0, + "2": 763730432.0, + "3": 763730432.0, + "4": 764516864.0, + "5": 763730432.0, + "6": 763730432.0, + "7": 763730432.0, + "8": 763730432.0, + "9": 763730432.0, + "10": 763730432.0, + "11": 763730432.0, + "12": 763730432.0, + "13": 764516864.0, + "14": 763730432.0, + "15": 763730432.0, + "16": 763730432.0, + "17": 764516864.0, + "18": 763730432.0, + "19": 764516864.0, + "20": 763730432.0, + "21": 764516864.0, + "22": 763730432.0, + "23": 763730432.0, + "24": 763730432.0, + "25": 764516864.0, + "26": 764516864.0, + "27": 763730432.0, + "28": 764516864.0, + "29": 763730432.0, + "30": 763730432.0, + "31": 764516864.0, + "32": 763730432.0, + "33": 763730432.0, + "34": 764516864.0, + "35": 763730432.0, + "36": 763730432.0, + "37": 763730432.0, + "38": 763730432.0, + "39": 763730432.0, + "40": 763730432.0, + "41": 763730432.0, + "42": 763730432.0, + "43": 763730432.0, + "44": 763730432.0, + "45": 763730432.0, + "46": 763730432.0, + "47": 763730432.0, + "48": 763730432.0, + "49": 763730432.0, + "50": 763730432.0 + } + }, + "mem-max-allocated-bytes": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 3868768768.0, + "2": 4154409984.0, + "3": 4154672128.0, + "4": 4154935808.0, + "5": 4154935808.0, + "6": 4154935808.0, + "7": 4154935808.0, + "8": 4155982848.0, + "9": 4155982848.0, + "10": 4155982848.0, + "11": 4155982848.0, + "12": 4155982848.0, + "13": 4155982848.0, + "14": 4155982848.0, + "15": 4155982848.0, + "16": 4155982848.0, + "17": 4155982848.0, + "18": 4155982848.0, + "19": 4155982848.0, + "20": 4155982848.0, + "21": 4155982848.0, + "22": 4155982848.0, + "23": 4155982848.0, + "24": 4155982848.0, + "25": 4155982848.0, + "26": 4155982848.0, + "27": 4155982848.0, + "28": 4155982848.0, + "29": 4155982848.0, + "30": 4155982848.0, + "31": 4155982848.0, + "32": 4155982848.0, + "33": 4155982848.0, + "34": 4155982848.0, + "35": 4155982848.0, + "36": 4155982848.0, + "37": 4155982848.0, + "38": 4155982848.0, + "39": 4155982848.0, + "40": 4155982848.0, + "41": 4155982848.0, + "42": 4155982848.0, + "43": 4155982848.0, + "44": 4155982848.0, + "45": 4155982848.0, + "46": 4155982848.0, + "47": 4155982848.0, + "48": 4155982848.0, + "49": 4155982848.0, + "50": 4155982848.0 + } + }, + "iteration-time": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": "nan", + "2": 6.67085, + "3": 2.27723, + "4": 1.80262, + "5": 1.61714, + "6": 1.78363, + "7": 1.6405, + "8": 2.12362, + "9": 2.28434, + "10": 1.8231, + "11": 1.94456, + "12": 1.69709, + "13": 1.81126, + "14": 2.09463, + "15": 1.61338, + "16": 1.79589, + "17": 2.12675, + "18": 1.82379, + "19": 1.90477, + "20": 1.64325, + "21": 1.80888, + "22": 1.49346, + "23": 1.91354, + "24": 1.76427, + "25": 2.11804, + "26": 2.30057, + "27": 1.56207, + "28": 1.53178, + "29": 1.56776, + "30": 1.73768, + "31": 1.51481, + "32": 2.5118, + "33": 1.45539, + "34": 1.53555, + "35": 2.04047, + "36": 2.16766, + "37": 1.7105, + "38": 1.27212, + "39": 2.18734, + "40": 1.85407, + "41": 1.93582, + "42": 2.40428, + "43": 1.72451, + "44": 1.64997, + "45": 1.86952, + "46": 1.62908, + "47": 1.82968, + "48": 2.05522, + "49": 1.54742, + "50": 2.32501 + } + } +} \ No newline at end of file diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_dist_optimizer_overlap_grad_reduce_param_gather_overlap_optimizer_1node/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_dist_optimizer_overlap_grad_reduce_param_gather_overlap_optimizer_1node/model_config.yaml new file mode 100644 index 00000000000..28248046b44 --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_dist_optimizer_overlap_grad_reduce_param_gather_overlap_optimizer_1node/model_config.yaml @@ -0,0 +1,58 @@ +ENV_VARS: + CUDA_DEVICE_MAX_CONNECTIONS: 1 + NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0 + NCCL_ALGO: Ring + CUBLAS_WORKSPACE_CONFIG: :4096:8 +MODEL_ARGS: + --num-layers: 12 + --hidden-size: 512 + --num-attention-heads: 8 + --log-params-norm: true + --log-num-zeros-in-grad: true + --log-validation-ppl-to-tensorboard: true + --log-timers-to-tensorboard: true + --tensorboard-dir: ${TENSORBOARD_PATH} + --micro-batch-size: 4 + --global-batch-size: 32 + --seq-length: 1024 + --max-position-embeddings: 1024 + --train-iters: 50 + --timing-log-level: 0 + --lr-decay-iters: 320000 + --save: ${CHECKPOINT_SAVE_PATH} + --load: ${CHECKPOINT_LOAD_PATH} + --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document + --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json + --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt + --split: 949,50,1 + --distributed-backend: nccl + --lr: 0.00015 + --lr-decay-style: cosine + --min-lr: 1.0e-5 + --weight-decay: 1e-2 + --clip-grad: 1.0 + --lr-warmup-fraction: .01 + --log-interval: 1 + --save-interval: 10000 + --eval-interval: 1000 + --eval-iters: 10 + --transformer-impl: transformer_engine + --tensor-model-parallel-size: 1 + --pipeline-model-parallel-size: 4 + --num-layers-per-virtual-pipeline-stage: 1 + --use-distributed-optimizer: true + --overlap-grad-reduce: true + --overlap-param-gather: true + --overlap-param-gather-with-optimizer-step: true + --check-weight-hash-across-dp-replicas-interval: 10 + --ckpt-fully-parallel-load: true + --deterministic-mode: true + --no-gradient-accumulation-fusion: true + --attention-softmax-in-fp32: true + --use-mcore-models: true + --ckpt-format: torch + --data-cache-path: ${DATA_CACHE_PATH} + --bf16: true + --attention-backend: unfused + --log-memory-to-tensorboard: true +TEST_TYPE: regular diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_decoupled_lr_1node/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_decoupled_lr_1node/golden_values_dev_dgx_gb200.json new file mode 100644 index 00000000000..d956f1fde7f --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_decoupled_lr_1node/golden_values_dev_dgx_gb200.json @@ -0,0 +1,537 @@ +{ + "lm loss": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 10.93832, + "2": 10.93769, + "3": 10.93625, + "4": 10.94232, + "5": 10.93349, + "6": 10.93269, + "7": 10.93796, + "8": 10.93356, + "9": 10.93555, + "10": 10.93692, + "11": 10.92585, + "12": 10.92412, + "13": 10.92504, + "14": 10.91248, + "15": 10.89335, + "16": 10.88815, + "17": 10.89968, + "18": 10.87789, + "19": 10.87549, + "20": 10.79108, + "21": 10.78001, + "22": 10.77459, + "23": 10.77485, + "24": 10.73559, + "25": 10.74282, + "26": 10.72347, + "27": 10.6955, + "28": 10.62303, + "29": 10.60325, + "30": 10.57991, + "31": 10.56891, + "32": 10.54647, + "33": 10.51723, + "34": 10.48161, + "35": 10.48816, + "36": 10.46791, + "37": 10.43373, + "38": 10.43848, + "39": 10.40028, + "40": 10.3903, + "41": 10.36588, + "42": 10.34114, + "43": 10.3191, + "44": 10.28097, + "45": 10.2984, + "46": 10.25723, + "47": 10.24193, + "48": 10.19501, + "49": 10.19305, + "50": 10.20072, + "51": 10.19623, + "52": 10.14789, + "53": 10.15718, + "54": 10.12042, + "55": 10.09009, + "56": 10.1185, + "57": 10.10154, + "58": 10.11659, + "59": 10.06156, + "60": 10.07213, + "61": 10.02683, + "62": 9.9928, + "63": 10.06735, + "64": 10.02093, + "65": 9.99324, + "66": 10.0227, + "67": 9.98726, + "68": 9.95238, + "69": 9.97227, + "70": 9.95945, + "71": 9.97771, + "72": 9.94822, + "73": 9.94072, + "74": 9.9307, + "75": 9.89817, + "76": 9.9397, + "77": 9.9293, + "78": 9.87561, + "79": 9.88327, + "80": 9.89532, + "81": 9.91711, + "82": 9.85934, + "83": 9.81914, + "84": 9.75546, + "85": 9.74598, + "86": 9.847, + "87": 9.87922, + "88": 9.85037, + "89": 9.78552, + "90": 9.77222, + "91": 9.78185, + "92": 9.76913, + "93": 9.70722, + "94": 9.77765, + "95": 9.77216, + "96": 9.75877, + "97": 9.69519, + "98": 9.72098, + "99": 9.76711, + "100": 9.65463 + } + }, + "num-zeros": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 1792.0, + "2": 1809.0, + "3": 1823.0, + "4": 1751.0, + "5": 1765.0, + "6": 1747.0, + "7": 2058.0, + "8": 1609.0, + "9": 1791.0, + "10": 1746.0, + "11": 1671.0, + "12": 1691.0, + "13": 1884.0, + "14": 1910.0, + "15": 1701.0, + "16": 1714.0, + "17": 1896.0, + "18": 1865.0, + "19": 1790.0, + "20": 1786.0, + "21": 1826.0, + "22": 1753.0, + "23": 1655.0, + "24": 1889.0, + "25": 1720.0, + "26": 1854.0, + "27": 1720.0, + "28": 1872.0, + "29": 1931.0, + "30": 1873.0, + "31": 2004.0, + "32": 1975.0, + "33": 1992.0, + "34": 2001.0, + "35": 2033.0, + "36": 1907.0, + "37": 2178.0, + "38": 2172.0, + "39": 2174.0, + "40": 2324.0, + "41": 2413.0, + "42": 1986.0, + "43": 2354.0, + "44": 2233.0, + "45": 2482.0, + "46": 2411.0, + "47": 2288.0, + "48": 2670.0, + "49": 2785.0, + "50": 2757.0, + "51": 2528.0, + "52": 2712.0, + "53": 2618.0, + "54": 2827.0, + "55": 2526.0, + "56": 2672.0, + "57": 2172.0, + "58": 3466.0, + "59": 2770.0, + "60": 2893.0, + "61": 2653.0, + "62": 3098.0, + "63": 3284.0, + "64": 3400.0, + "65": 2751.0, + "66": 3072.0, + "67": 3801.0, + "68": 3389.0, + "69": 2897.0, + "70": 3096.0, + "71": 3056.0, + "72": 2962.0, + "73": 3260.0, + "74": 3118.0, + "75": 3090.0, + "76": 3252.0, + "77": 3597.0, + "78": 3083.0, + "79": 3233.0, + "80": 2958.0, + "81": 3295.0, + "82": 3257.0, + "83": 3143.0, + "84": 3047.0, + "85": 2752.0, + "86": 3251.0, + "87": 2866.0, + "88": 3149.0, + "89": 2803.0, + "90": 3609.0, + "91": 3035.0, + "92": 2997.0, + "93": 3106.0, + "94": 3118.0, + "95": 3399.0, + "96": 3534.0, + "97": 3611.0, + "98": 3729.0, + "99": 3121.0, + "100": 3262.0 + } + }, + "mem-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 759682560.0, + "2": 759682560.0, + "3": 759682560.0, + "4": 759682560.0, + "5": 759682560.0, + "6": 759682560.0, + "7": 759682560.0, + "8": 759682560.0, + "9": 759682560.0, + "10": 759682560.0, + "11": 759682560.0, + "12": 759682560.0, + "13": 759682560.0, + "14": 759682560.0, + "15": 759682560.0, + "16": 759682560.0, + "17": 759682560.0, + "18": 759682560.0, + "19": 759682560.0, + "20": 759682560.0, + "21": 759682560.0, + "22": 759682560.0, + "23": 759682560.0, + "24": 759682560.0, + "25": 759682560.0, + "26": 759682560.0, + "27": 759682560.0, + "28": 759682560.0, + "29": 759682560.0, + "30": 759682560.0, + "31": 759682560.0, + "32": 759682560.0, + "33": 759682560.0, + "34": 759682560.0, + "35": 759682560.0, + "36": 759682560.0, + "37": 759682560.0, + "38": 759682560.0, + "39": 759682560.0, + "40": 759682560.0, + "41": 759682560.0, + "42": 759682560.0, + "43": 759682560.0, + "44": 759682560.0, + "45": 759682560.0, + "46": 759682560.0, + "47": 759682560.0, + "48": 759682560.0, + "49": 759682560.0, + "50": 759682560.0, + "51": 759682560.0, + "52": 759682560.0, + "53": 759682560.0, + "54": 759682560.0, + "55": 759682560.0, + "56": 759682560.0, + "57": 759682560.0, + "58": 759682560.0, + "59": 759682560.0, + "60": 759682560.0, + "61": 759682560.0, + "62": 759682560.0, + "63": 759682560.0, + "64": 759682560.0, + "65": 759682560.0, + "66": 759682560.0, + "67": 759682560.0, + "68": 759682560.0, + "69": 759682560.0, + "70": 759682560.0, + "71": 759682560.0, + "72": 759682560.0, + "73": 759682560.0, + "74": 759682560.0, + "75": 759682560.0, + "76": 759682560.0, + "77": 759682560.0, + "78": 759682560.0, + "79": 759682560.0, + "80": 759682560.0, + "81": 759682560.0, + "82": 759682560.0, + "83": 759682560.0, + "84": 759682560.0, + "85": 759682560.0, + "86": 759682560.0, + "87": 759682560.0, + "88": 759682560.0, + "89": 759682560.0, + "90": 759682560.0, + "91": 759682560.0, + "92": 759682560.0, + "93": 759682560.0, + "94": 759682560.0, + "95": 759682560.0, + "96": 759682560.0, + "97": 759682560.0, + "98": 759682560.0, + "99": 759682560.0, + "100": 759682560.0 + } + }, + "mem-max-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 2395800576.0, + "2": 2677512192.0, + "3": 2677512192.0, + "4": 2677512192.0, + "5": 2677512192.0, + "6": 2677512192.0, + "7": 2677512192.0, + "8": 2677512192.0, + "9": 2677512192.0, + "10": 2677512192.0, + "11": 2677512192.0, + "12": 2677512192.0, + "13": 2677512192.0, + "14": 2677512192.0, + "15": 2677512192.0, + "16": 2677512192.0, + "17": 2677512192.0, + "18": 2677512192.0, + "19": 2677512192.0, + "20": 2677512192.0, + "21": 2677512192.0, + "22": 2677512192.0, + "23": 2677512192.0, + "24": 2677512192.0, + "25": 2677512192.0, + "26": 2677512192.0, + "27": 2677512192.0, + "28": 2677512192.0, + "29": 2677512192.0, + "30": 2677512192.0, + "31": 2677512192.0, + "32": 2677512192.0, + "33": 2677512192.0, + "34": 2677512192.0, + "35": 2677512192.0, + "36": 2677512192.0, + "37": 2677512192.0, + "38": 2677512192.0, + "39": 2677512192.0, + "40": 2677512192.0, + "41": 2677512192.0, + "42": 2677512192.0, + "43": 2677512192.0, + "44": 2677512192.0, + "45": 2677512192.0, + "46": 2677512192.0, + "47": 2677512192.0, + "48": 2677512192.0, + "49": 2677512192.0, + "50": 2677512192.0, + "51": 2677512192.0, + "52": 2677512192.0, + "53": 2677512192.0, + "54": 2677512192.0, + "55": 2677512192.0, + "56": 2677512192.0, + "57": 2677512192.0, + "58": 2677512192.0, + "59": 2677512192.0, + "60": 2677512192.0, + "61": 2677512192.0, + "62": 2677512192.0, + "63": 2677512192.0, + "64": 2677512192.0, + "65": 2677512192.0, + "66": 2677512192.0, + "67": 2677512192.0, + "68": 2677512192.0, + "69": 2677512192.0, + "70": 2677512192.0, + "71": 2677512192.0, + "72": 2677512192.0, + "73": 2677512192.0, + "74": 2677512192.0, + "75": 2677512192.0, + "76": 2677512192.0, + "77": 2677512192.0, + "78": 2677512192.0, + "79": 2677512192.0, + "80": 2677512192.0, + "81": 2677512192.0, + "82": 2677512192.0, + "83": 2677512192.0, + "84": 2677512192.0, + "85": 2677512192.0, + "86": 2677512192.0, + "87": 2677512192.0, + "88": 2677512192.0, + "89": 2677512192.0, + "90": 2677512192.0, + "91": 2677512192.0, + "92": 2677512192.0, + "93": 2677512192.0, + "94": 2677512192.0, + "95": 2677512192.0, + "96": 2677512192.0, + "97": 2677512192.0, + "98": 2677512192.0, + "99": 2677512192.0, + "100": 2677512192.0 + } + }, + "iteration-time": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": "nan", + "2": 5.65975, + "3": 1.71033, + "4": 1.61619, + "5": 1.62708, + "6": 1.61612, + "7": 1.30395, + "8": 1.74515, + "9": 2.07042, + "10": 1.68014, + "11": 1.86725, + "12": 1.66946, + "13": 1.86948, + "14": 2.18714, + "15": 1.67257, + "16": 1.58643, + "17": 1.85266, + "18": 1.65496, + "19": 1.82152, + "20": 1.70409, + "21": 1.69468, + "22": 1.53054, + "23": 1.62991, + "24": 1.30673, + "25": 1.81085, + "26": 2.17003, + "27": 1.60589, + "28": 1.55514, + "29": 1.44325, + "30": 1.55277, + "31": 1.47009, + "32": 2.35097, + "33": 1.09025, + "34": 1.11502, + "35": 1.83528, + "36": 1.554, + "37": 1.29958, + "38": 1.15238, + "39": 1.93822, + "40": 1.60324, + "41": 1.56651, + "42": 2.10001, + "43": 1.64771, + "44": 1.6673, + "45": 1.96364, + "46": 1.67512, + "47": 1.59582, + "48": 1.95553, + "49": 1.58441, + "50": 1.82438, + "51": 1.21939, + "52": 1.66814, + "53": 1.69296, + "54": 1.86224, + "55": 1.67675, + "56": 2.22109, + "57": 1.24105, + "58": 1.93032, + "59": 1.58486, + "60": 1.40785, + "61": 1.89223, + "62": 1.83955, + "63": 1.53497, + "64": 1.50703, + "65": 1.57318, + "66": 1.67471, + "67": 1.92086, + "68": 1.3281, + "69": 2.3025, + "70": 2.53112, + "71": 2.36935, + "72": 1.71173, + "73": 1.92702, + "74": 1.30777, + "75": 1.65513, + "76": 1.90433, + "77": 1.65825, + "78": 1.73538, + "79": 1.9881, + "80": 1.66939, + "81": 1.89802, + "82": 2.00859, + "83": 1.75269, + "84": 1.43574, + "85": 1.52728, + "86": 1.74408, + "87": 1.98598, + "88": 1.90844, + "89": 1.97691, + "90": 1.71849, + "91": 1.80292, + "92": 2.26616, + "93": 1.37073, + "94": 1.78929, + "95": 1.90907, + "96": 1.92355, + "97": 1.57863, + "98": 1.62191, + "99": 2.18341, + "100": 1.63888 + } + } +} \ No newline at end of file diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_decoupled_lr_1node/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_decoupled_lr_1node/model_config.yaml new file mode 100644 index 00000000000..6bcce43a2db --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_decoupled_lr_1node/model_config.yaml @@ -0,0 +1,54 @@ +ENV_VARS: + CUDA_DEVICE_MAX_CONNECTIONS: 1 + NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0 + NCCL_ALGO: Ring + CUBLAS_WORKSPACE_CONFIG: :4096:8 + PYTORCH_CUDA_ALLOC_CONF: expandable_segments:True +MODEL_ARGS: + --num-layers: 12 + --hidden-size: 512 + --num-attention-heads: 8 + --log-params-norm: true + --log-num-zeros-in-grad: true + --log-validation-ppl-to-tensorboard: true + --log-timers-to-tensorboard: true + --tensorboard-dir: ${TENSORBOARD_PATH} + --micro-batch-size: 4 + --global-batch-size: 32 + --seq-length: 1024 + --max-position-embeddings: 1024 + --train-iters: 100 + --timing-log-level: 0 + --lr-decay-iters: 320000 + --save: ${CHECKPOINT_SAVE_PATH} + --load: ${CHECKPOINT_LOAD_PATH} + --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document + --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json + --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt + --split: 949,50,1 + --distributed-backend: nccl + --lr: 0.00015 + --lr-decay-style: cosine + --min-lr: 1.0e-5 + --weight-decay: 1e-2 + --clip-grad: 1.0 + --lr-warmup-fraction: .01 + --log-interval: 1 + --save-interval: 50 + --eval-interval: 1000 + --eval-iters: 10 + --transformer-impl: transformer_engine + --tensor-model-parallel-size: 1 + --pipeline-model-parallel-size: 4 + --num-layers-per-virtual-pipeline-stage: 1 + --decoupled-lr: 0.0002 + --deterministic-mode: true + --no-gradient-accumulation-fusion: true + --attention-softmax-in-fp32: true + --use-checkpoint-opt_param-scheduler: true + --ckpt-format: torch + --dist-ckpt-strictness: log_all # backward compatibility for TE changes + --data-cache-path: ${DATA_CACHE_PATH} + --bf16: true + --log-memory-to-tensorboard: true +TEST_TYPE: regular # Usually ckpt-resume, but as a WAR to #513 set to regular diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist_calculate_per_token_loss_1node/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist_calculate_per_token_loss_1node/golden_values_dev_dgx_gb200.json new file mode 100644 index 00000000000..ce2166701f3 --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist_calculate_per_token_loss_1node/golden_values_dev_dgx_gb200.json @@ -0,0 +1,537 @@ +{ + "lm loss": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 10.93832, + "2": 10.93769, + "3": 10.93625, + "4": 10.9423, + "5": 10.93352, + "6": 10.93273, + "7": 10.93793, + "8": 10.93357, + "9": 10.93552, + "10": 10.93693, + "11": 10.92585, + "12": 10.92413, + "13": 10.9251, + "14": 10.91258, + "15": 10.89335, + "16": 10.88822, + "17": 10.89973, + "18": 10.87821, + "19": 10.87589, + "20": 10.79119, + "21": 10.78019, + "22": 10.77474, + "23": 10.77501, + "24": 10.7364, + "25": 10.74383, + "26": 10.72466, + "27": 10.69626, + "28": 10.62353, + "29": 10.60379, + "30": 10.58048, + "31": 10.56959, + "32": 10.5486, + "33": 10.52046, + "34": 10.48553, + "35": 10.4923, + "36": 10.47086, + "37": 10.43635, + "38": 10.44109, + "39": 10.40281, + "40": 10.3928, + "41": 10.36862, + "42": 10.34609, + "43": 10.32634, + "44": 10.28964, + "45": 10.3083, + "46": 10.2682, + "47": 10.2525, + "48": 10.20372, + "49": 10.20119, + "50": 10.20922, + "51": 10.20528, + "52": 10.15731, + "53": 10.16687, + "54": 10.13043, + "55": 10.10057, + "56": 10.13013, + "57": 10.11459, + "58": 10.13004, + "59": 10.07594, + "60": 10.08697, + "61": 10.04225, + "62": 10.00953, + "63": 10.08368, + "64": 10.03719, + "65": 10.00999, + "66": 10.04021, + "67": 10.00686, + "68": 9.97346, + "69": 9.99438, + "70": 9.98228, + "71": 10.00183, + "72": 9.97365, + "73": 9.96698, + "74": 9.95724, + "75": 9.92479, + "76": 9.96623, + "77": 9.95576, + "78": 9.90172, + "79": 9.90899, + "80": 9.92107, + "81": 9.94317, + "82": 9.88583, + "83": 9.84493, + "84": 9.7831, + "85": 9.77443, + "86": 9.87683, + "87": 9.91001, + "88": 9.882, + "89": 9.81985, + "90": 9.80796, + "91": 9.81902, + "92": 9.80779, + "93": 9.74751, + "94": 9.82029, + "95": 9.8157, + "96": 9.80356, + "97": 9.7406, + "98": 9.76894, + "99": 9.81594, + "100": 9.70555 + } + }, + "num-zeros": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 1792.0, + "2": 1809.0, + "3": 1721.0, + "4": 1751.0, + "5": 1756.0, + "6": 1766.0, + "7": 2096.0, + "8": 1769.0, + "9": 1799.0, + "10": 1745.0, + "11": 1650.0, + "12": 1774.0, + "13": 1839.0, + "14": 1831.0, + "15": 1653.0, + "16": 1694.0, + "17": 1777.0, + "18": 1814.0, + "19": 1660.0, + "20": 1774.0, + "21": 1735.0, + "22": 1698.0, + "23": 1720.0, + "24": 1857.0, + "25": 1696.0, + "26": 1895.0, + "27": 1727.0, + "28": 1790.0, + "29": 1978.0, + "30": 1883.0, + "31": 2011.0, + "32": 2009.0, + "33": 2065.0, + "34": 2005.0, + "35": 1998.0, + "36": 1949.0, + "37": 2233.0, + "38": 2134.0, + "39": 2029.0, + "40": 2320.0, + "41": 2310.0, + "42": 2012.0, + "43": 2229.0, + "44": 2253.0, + "45": 2455.0, + "46": 2413.0, + "47": 2499.0, + "48": 2654.0, + "49": 2695.0, + "50": 2556.0, + "51": 2497.0, + "52": 2697.0, + "53": 2566.0, + "54": 2737.0, + "55": 2430.0, + "56": 2681.0, + "57": 2235.0, + "58": 3500.0, + "59": 2810.0, + "60": 2847.0, + "61": 2634.0, + "62": 3234.0, + "63": 3180.0, + "64": 3344.0, + "65": 2659.0, + "66": 2923.0, + "67": 3727.0, + "68": 3419.0, + "69": 2980.0, + "70": 3155.0, + "71": 3102.0, + "72": 2985.0, + "73": 3265.0, + "74": 3183.0, + "75": 3040.0, + "76": 3010.0, + "77": 3807.0, + "78": 3133.0, + "79": 3258.0, + "80": 3010.0, + "81": 3341.0, + "82": 3261.0, + "83": 3212.0, + "84": 2927.0, + "85": 2838.0, + "86": 3179.0, + "87": 2957.0, + "88": 3042.0, + "89": 2842.0, + "90": 3453.0, + "91": 3015.0, + "92": 3062.0, + "93": 3085.0, + "94": 3098.0, + "95": 3397.0, + "96": 3546.0, + "97": 3660.0, + "98": 3603.0, + "99": 3169.0, + "100": 3271.0 + } + }, + "mem-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 765317632.0, + "2": 765317632.0, + "3": 765317632.0, + "4": 765317632.0, + "5": 765317632.0, + "6": 765317632.0, + "7": 765317632.0, + "8": 765317632.0, + "9": 765317632.0, + "10": 765317632.0, + "11": 765317632.0, + "12": 765317632.0, + "13": 765317632.0, + "14": 765317632.0, + "15": 765317632.0, + "16": 765317632.0, + "17": 765317632.0, + "18": 765317632.0, + "19": 765317632.0, + "20": 765317632.0, + "21": 765317632.0, + "22": 765317632.0, + "23": 765317632.0, + "24": 765317632.0, + "25": 765317632.0, + "26": 765317632.0, + "27": 765317632.0, + "28": 765317632.0, + "29": 765317632.0, + "30": 765317632.0, + "31": 765317632.0, + "32": 765317632.0, + "33": 765317632.0, + "34": 765317632.0, + "35": 765317632.0, + "36": 765317632.0, + "37": 765317632.0, + "38": 765317632.0, + "39": 765317632.0, + "40": 765317632.0, + "41": 765317632.0, + "42": 765317632.0, + "43": 765317632.0, + "44": 765317632.0, + "45": 765317632.0, + "46": 765317632.0, + "47": 765317632.0, + "48": 765317632.0, + "49": 765317632.0, + "50": 765317632.0, + "51": 765317632.0, + "52": 765317632.0, + "53": 765317632.0, + "54": 765317632.0, + "55": 765317632.0, + "56": 765317632.0, + "57": 765317632.0, + "58": 765317632.0, + "59": 765317632.0, + "60": 765317632.0, + "61": 765317632.0, + "62": 765317632.0, + "63": 765317632.0, + "64": 765317632.0, + "65": 765317632.0, + "66": 765317632.0, + "67": 765317632.0, + "68": 765317632.0, + "69": 765317632.0, + "70": 765317632.0, + "71": 765317632.0, + "72": 765317632.0, + "73": 765317632.0, + "74": 765317632.0, + "75": 765317632.0, + "76": 765317632.0, + "77": 765317632.0, + "78": 765317632.0, + "79": 765317632.0, + "80": 765317632.0, + "81": 765317632.0, + "82": 765317632.0, + "83": 765317632.0, + "84": 765317632.0, + "85": 765317632.0, + "86": 765317632.0, + "87": 765317632.0, + "88": 765317632.0, + "89": 765317632.0, + "90": 765841920.0, + "91": 765317632.0, + "92": 765317632.0, + "93": 765317632.0, + "94": 765317632.0, + "95": 765317632.0, + "96": 765317632.0, + "97": 765317632.0, + "98": 765317632.0, + "99": 765317632.0, + "100": 765317632.0 + } + }, + "mem-max-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 2398288384.0, + "2": 2683147264.0, + "3": 2683147264.0, + "4": 2683671552.0, + "5": 2683671552.0, + "6": 2683671552.0, + "7": 2683671552.0, + "8": 2683671552.0, + "9": 2683671552.0, + "10": 2683671552.0, + "11": 2683671552.0, + "12": 2683671552.0, + "13": 2683671552.0, + "14": 2683671552.0, + "15": 2683671552.0, + "16": 2683671552.0, + "17": 2683671552.0, + "18": 2683671552.0, + "19": 2684194304.0, + "20": 2684194304.0, + "21": 2684194304.0, + "22": 2684194304.0, + "23": 2684194304.0, + "24": 2684194304.0, + "25": 2684194304.0, + "26": 2684194304.0, + "27": 2684194304.0, + "28": 2684194304.0, + "29": 2684194304.0, + "30": 2684194304.0, + "31": 2684194304.0, + "32": 2684194304.0, + "33": 2684194304.0, + "34": 2684194304.0, + "35": 2684194304.0, + "36": 2684194304.0, + "37": 2684194304.0, + "38": 2684194304.0, + "39": 2684194304.0, + "40": 2684194304.0, + "41": 2684194304.0, + "42": 2684194304.0, + "43": 2684194304.0, + "44": 2684194304.0, + "45": 2684194304.0, + "46": 2684194304.0, + "47": 2684194304.0, + "48": 2684194304.0, + "49": 2684194304.0, + "50": 2684194304.0, + "51": 2684194304.0, + "52": 2684194304.0, + "53": 2684194304.0, + "54": 2684194304.0, + "55": 2684194304.0, + "56": 2684194304.0, + "57": 2684194304.0, + "58": 2684194304.0, + "59": 2684194304.0, + "60": 2684195840.0, + "61": 2684195840.0, + "62": 2684195840.0, + "63": 2684195840.0, + "64": 2684195840.0, + "65": 2684195840.0, + "66": 2684195840.0, + "67": 2684195840.0, + "68": 2684195840.0, + "69": 2684195840.0, + "70": 2684195840.0, + "71": 2684195840.0, + "72": 2684195840.0, + "73": 2684195840.0, + "74": 2684195840.0, + "75": 2684195840.0, + "76": 2684195840.0, + "77": 2684195840.0, + "78": 2684195840.0, + "79": 2684195840.0, + "80": 2684195840.0, + "81": 2684195840.0, + "82": 2684195840.0, + "83": 2684195840.0, + "84": 2684195840.0, + "85": 2684195840.0, + "86": 2684195840.0, + "87": 2684195840.0, + "88": 2684195840.0, + "89": 2684195840.0, + "90": 2684195840.0, + "91": 2684195840.0, + "92": 2684195840.0, + "93": 2684195840.0, + "94": 2684195840.0, + "95": 2684195840.0, + "96": 2684195840.0, + "97": 2684195840.0, + "98": 2684195840.0, + "99": 2684195840.0, + "100": 2684195840.0 + } + }, + "iteration-time": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": "nan", + "2": 6.15156, + "3": 1.86234, + "4": 1.74565, + "5": 1.44521, + "6": 1.54802, + "7": 1.27037, + "8": 1.57856, + "9": 1.94239, + "10": 1.72764, + "11": 1.93656, + "12": 1.49016, + "13": 1.54007, + "14": 1.89382, + "15": 1.20402, + "16": 1.69582, + "17": 2.07383, + "18": 1.64828, + "19": 1.65051, + "20": 1.388, + "21": 2.07341, + "22": 1.60595, + "23": 1.70295, + "24": 1.51047, + "25": 2.12996, + "26": 2.26369, + "27": 1.54887, + "28": 1.70674, + "29": 1.47573, + "30": 1.88073, + "31": 1.42806, + "32": 2.49977, + "33": 1.33717, + "34": 1.34399, + "35": 1.98003, + "36": 1.91919, + "37": 1.50567, + "38": 1.02054, + "39": 1.92264, + "40": 1.58269, + "41": 1.81946, + "42": 2.12454, + "43": 1.49655, + "44": 1.46182, + "45": 1.87527, + "46": 1.72537, + "47": 1.49901, + "48": 1.76275, + "49": 1.34074, + "50": 1.69646, + "51": 0.99687, + "52": 1.74019, + "53": 1.70182, + "54": 1.92817, + "55": 1.73006, + "56": 2.21077, + "57": 1.18615, + "58": 1.88348, + "59": 2.3094, + "60": 1.99351, + "61": 1.95814, + "62": 1.86831, + "63": 1.46649, + "64": 1.49152, + "65": 3.02732, + "66": 1.81567, + "67": 1.7731, + "68": 1.13975, + "69": 1.83008, + "70": 1.66395, + "71": 1.75984, + "72": 1.38929, + "73": 1.49735, + "74": 1.53139, + "75": 1.45926, + "76": 1.50068, + "77": 1.32291, + "78": 1.53519, + "79": 2.1402, + "80": 1.66458, + "81": 1.78083, + "82": 2.20276, + "83": 1.78402, + "84": 1.69056, + "85": 1.64974, + "86": 1.88908, + "87": 2.08554, + "88": 1.806, + "89": 1.92606, + "90": 1.75161, + "91": 1.66772, + "92": 2.46731, + "93": 1.58667, + "94": 1.50601, + "95": 1.69343, + "96": 1.68078, + "97": 1.60625, + "98": 1.73359, + "99": 2.28566, + "100": 1.47655 + } + } +} \ No newline at end of file diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist_calculate_per_token_loss_1node/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist_calculate_per_token_loss_1node/model_config.yaml new file mode 100644 index 00000000000..6d77c3df361 --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist_calculate_per_token_loss_1node/model_config.yaml @@ -0,0 +1,57 @@ +ENV_VARS: + CUDA_DEVICE_MAX_CONNECTIONS: 1 + NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0 + NCCL_ALGO: Ring + CUBLAS_WORKSPACE_CONFIG: :4096:8 +MODEL_ARGS: + --num-layers: 12 + --hidden-size: 512 + --num-attention-heads: 8 + --log-params-norm: true + --log-num-zeros-in-grad: true + --log-validation-ppl-to-tensorboard: true + --log-timers-to-tensorboard: true + --tensorboard-dir: ${TENSORBOARD_PATH} + --micro-batch-size: 4 + --global-batch-size: 32 + --seq-length: 1024 + --max-position-embeddings: 1024 + --train-iters: 100 + --timing-log-level: 0 + --lr-decay-iters: 320000 + --save: ${CHECKPOINT_SAVE_PATH} + --load: ${CHECKPOINT_LOAD_PATH} + --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document + --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json + --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt + --split: 949,50,1 + --distributed-backend: nccl + --lr: 0.00015 + --lr-decay-style: cosine + --min-lr: 1.0e-5 + --weight-decay: 1e-2 + --clip-grad: 1.0 + --lr-warmup-fraction: .01 + --log-interval: 1 + --save-interval: 50 + --eval-interval: 1000 + --eval-iters: 10 + --transformer-impl: transformer_engine + --tensor-model-parallel-size: 1 + --pipeline-model-parallel-size: 4 + --num-layers-per-virtual-pipeline-stage: 1 + --calculate-per-token-loss: true + --deterministic-mode: true + --no-gradient-accumulation-fusion: true + --attention-softmax-in-fp32: true + --use-checkpoint-opt_param-scheduler: true + --use-mcore-models: true + --ckpt-format: torch_dist + --dist-ckpt-optim-fully-reshardable: true + --dist-ckpt-strictness: log_all # backward compatibility for TE changes + --data-cache-path: ${DATA_CACHE_PATH} + --bf16: true + --log-memory-to-tensorboard: true + --async-save: true + --use-persistent-ckpt-worker: true +TEST_TYPE: ckpt-resume diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_1node/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_1node/golden_values_dev_dgx_gb200.json new file mode 100644 index 00000000000..6a2f2e57cb4 --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_1node/golden_values_dev_dgx_gb200.json @@ -0,0 +1,537 @@ +{ + "lm loss": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 10.93832, + "2": 10.93769, + "3": 10.93625, + "4": 10.9423, + "5": 10.93352, + "6": 10.93273, + "7": 10.93793, + "8": 10.93357, + "9": 10.93553, + "10": 10.93697, + "11": 10.92587, + "12": 10.92415, + "13": 10.92509, + "14": 10.9126, + "15": 10.89336, + "16": 10.88824, + "17": 10.89973, + "18": 10.87823, + "19": 10.87587, + "20": 10.79123, + "21": 10.78019, + "22": 10.77476, + "23": 10.77499, + "24": 10.73642, + "25": 10.74387, + "26": 10.72466, + "27": 10.6963, + "28": 10.62351, + "29": 10.6038, + "30": 10.58045, + "31": 10.56963, + "32": 10.54858, + "33": 10.52048, + "34": 10.48554, + "35": 10.49228, + "36": 10.4709, + "37": 10.43637, + "38": 10.44107, + "39": 10.40279, + "40": 10.39284, + "41": 10.36864, + "42": 10.34611, + "43": 10.32631, + "44": 10.28964, + "45": 10.30829, + "46": 10.26822, + "47": 10.25249, + "48": 10.2037, + "49": 10.20119, + "50": 10.20925, + "51": 10.2053, + "52": 10.15734, + "53": 10.16687, + "54": 10.13043, + "55": 10.10057, + "56": 10.13011, + "57": 10.11464, + "58": 10.13007, + "59": 10.07596, + "60": 10.08697, + "61": 10.04223, + "62": 10.00953, + "63": 10.08368, + "64": 10.03719, + "65": 10.00997, + "66": 10.04022, + "67": 10.00689, + "68": 9.97342, + "69": 9.99439, + "70": 9.98223, + "71": 10.00185, + "72": 9.97365, + "73": 9.96699, + "74": 9.95725, + "75": 9.92477, + "76": 9.96624, + "77": 9.95575, + "78": 9.90173, + "79": 9.90898, + "80": 9.92106, + "81": 9.94319, + "82": 9.88581, + "83": 9.84494, + "84": 9.78312, + "85": 9.77444, + "86": 9.87683, + "87": 9.91005, + "88": 9.882, + "89": 9.81987, + "90": 9.80797, + "91": 9.81902, + "92": 9.80781, + "93": 9.74749, + "94": 9.82029, + "95": 9.8157, + "96": 9.80355, + "97": 9.7406, + "98": 9.76895, + "99": 9.8159, + "100": 9.70554 + } + }, + "num-zeros": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 1792.0, + "2": 1809.0, + "3": 1721.0, + "4": 1751.0, + "5": 1756.0, + "6": 1766.0, + "7": 2096.0, + "8": 1713.0, + "9": 1804.0, + "10": 1718.0, + "11": 1736.0, + "12": 1724.0, + "13": 1775.0, + "14": 1926.0, + "15": 1658.0, + "16": 1757.0, + "17": 1812.0, + "18": 1879.0, + "19": 1710.0, + "20": 1679.0, + "21": 1723.0, + "22": 1690.0, + "23": 1753.0, + "24": 1892.0, + "25": 1767.0, + "26": 1869.0, + "27": 1862.0, + "28": 1902.0, + "29": 1826.0, + "30": 1880.0, + "31": 1998.0, + "32": 1982.0, + "33": 1996.0, + "34": 2175.0, + "35": 2051.0, + "36": 1974.0, + "37": 2256.0, + "38": 2159.0, + "39": 2063.0, + "40": 2239.0, + "41": 2379.0, + "42": 1944.0, + "43": 2319.0, + "44": 2172.0, + "45": 2550.0, + "46": 2381.0, + "47": 2453.0, + "48": 2591.0, + "49": 2636.0, + "50": 2590.0, + "51": 2477.0, + "52": 2793.0, + "53": 2561.0, + "54": 2717.0, + "55": 2472.0, + "56": 2677.0, + "57": 2183.0, + "58": 3495.0, + "59": 2935.0, + "60": 2895.0, + "61": 2757.0, + "62": 3041.0, + "63": 3137.0, + "64": 3332.0, + "65": 2609.0, + "66": 3019.0, + "67": 3698.0, + "68": 3370.0, + "69": 2933.0, + "70": 3278.0, + "71": 3092.0, + "72": 3047.0, + "73": 3310.0, + "74": 3031.0, + "75": 3167.0, + "76": 3175.0, + "77": 3706.0, + "78": 3237.0, + "79": 3195.0, + "80": 3060.0, + "81": 3281.0, + "82": 3156.0, + "83": 3213.0, + "84": 2976.0, + "85": 2832.0, + "86": 3080.0, + "87": 2995.0, + "88": 3282.0, + "89": 2781.0, + "90": 3476.0, + "91": 2998.0, + "92": 2978.0, + "93": 3222.0, + "94": 3236.0, + "95": 3461.0, + "96": 3503.0, + "97": 3660.0, + "98": 3618.0, + "99": 3139.0, + "100": 3325.0 + } + }, + "mem-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 762681856.0, + "2": 762681856.0, + "3": 762681856.0, + "4": 762681856.0, + "5": 762681856.0, + "6": 762681856.0, + "7": 762681856.0, + "8": 762681856.0, + "9": 762681856.0, + "10": 762681856.0, + "11": 762681856.0, + "12": 762681856.0, + "13": 762681856.0, + "14": 762681856.0, + "15": 762681856.0, + "16": 762681856.0, + "17": 762681856.0, + "18": 762681856.0, + "19": 762681856.0, + "20": 762681856.0, + "21": 762681856.0, + "22": 762681856.0, + "23": 762681856.0, + "24": 762681856.0, + "25": 762681856.0, + "26": 762681856.0, + "27": 762681856.0, + "28": 762681856.0, + "29": 762681856.0, + "30": 762681856.0, + "31": 762681856.0, + "32": 762681856.0, + "33": 762681856.0, + "34": 762681856.0, + "35": 762681856.0, + "36": 762681856.0, + "37": 762681856.0, + "38": 762681856.0, + "39": 762681856.0, + "40": 762681856.0, + "41": 762681856.0, + "42": 762681856.0, + "43": 762681856.0, + "44": 762681856.0, + "45": 762681856.0, + "46": 762681856.0, + "47": 762681856.0, + "48": 762681856.0, + "49": 762681856.0, + "50": 762681856.0, + "51": 762681856.0, + "52": 762681856.0, + "53": 762681856.0, + "54": 762681856.0, + "55": 762681856.0, + "56": 762681856.0, + "57": 762681856.0, + "58": 762681856.0, + "59": 762681856.0, + "60": 762681856.0, + "61": 762681856.0, + "62": 762681856.0, + "63": 762681856.0, + "64": 762681856.0, + "65": 762681856.0, + "66": 762681856.0, + "67": 762681856.0, + "68": 762681856.0, + "69": 762681856.0, + "70": 762681856.0, + "71": 762681856.0, + "72": 762681856.0, + "73": 762681856.0, + "74": 762681856.0, + "75": 762681856.0, + "76": 762681856.0, + "77": 762681856.0, + "78": 762681856.0, + "79": 762681856.0, + "80": 762681856.0, + "81": 762681856.0, + "82": 762681856.0, + "83": 762681856.0, + "84": 762681856.0, + "85": 762681856.0, + "86": 762681856.0, + "87": 762681856.0, + "88": 762681856.0, + "89": 762681856.0, + "90": 762681856.0, + "91": 762681856.0, + "92": 762681856.0, + "93": 762681856.0, + "94": 762681856.0, + "95": 762681856.0, + "96": 762681856.0, + "97": 762681856.0, + "98": 762681856.0, + "99": 762681856.0, + "100": 762681856.0 + } + }, + "mem-max-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 2398797312.0, + "2": 2681558528.0, + "3": 2681558528.0, + "4": 2681558528.0, + "5": 2681558528.0, + "6": 2682082816.0, + "7": 2682082816.0, + "8": 2682082816.0, + "9": 2682082816.0, + "10": 2682082816.0, + "11": 2682082816.0, + "12": 2682082816.0, + "13": 2682084352.0, + "14": 2682084352.0, + "15": 2682084352.0, + "16": 2682084352.0, + "17": 2682084352.0, + "18": 2682084352.0, + "19": 2682084352.0, + "20": 2682084352.0, + "21": 2682084352.0, + "22": 2682084352.0, + "23": 2682084352.0, + "24": 2682084352.0, + "25": 2682608640.0, + "26": 2682608640.0, + "27": 2682608640.0, + "28": 2682608640.0, + "29": 2682608640.0, + "30": 2682608640.0, + "31": 2682608640.0, + "32": 2682608640.0, + "33": 2682608640.0, + "34": 2682608640.0, + "35": 2682608640.0, + "36": 2682608640.0, + "37": 2682608640.0, + "38": 2682608640.0, + "39": 2682608640.0, + "40": 2682608640.0, + "41": 2682608640.0, + "42": 2682608640.0, + "43": 2682608640.0, + "44": 2682608640.0, + "45": 2682608640.0, + "46": 2682608640.0, + "47": 2682608640.0, + "48": 2682608640.0, + "49": 2682608640.0, + "50": 2682608640.0, + "51": 2682608640.0, + "52": 2682608640.0, + "53": 2682608640.0, + "54": 2682608640.0, + "55": 2682608640.0, + "56": 2682608640.0, + "57": 2682608640.0, + "58": 2682608640.0, + "59": 2682608640.0, + "60": 2682608640.0, + "61": 2682608640.0, + "62": 2682608640.0, + "63": 2682608640.0, + "64": 2682608640.0, + "65": 2682608640.0, + "66": 2682608640.0, + "67": 2682608640.0, + "68": 2683131392.0, + "69": 2683131392.0, + "70": 2683131392.0, + "71": 2683131392.0, + "72": 2683131392.0, + "73": 2683131392.0, + "74": 2683131392.0, + "75": 2683131392.0, + "76": 2683131392.0, + "77": 2683131392.0, + "78": 2683131392.0, + "79": 2683131392.0, + "80": 2683131392.0, + "81": 2683131392.0, + "82": 2683131392.0, + "83": 2683131392.0, + "84": 2683131392.0, + "85": 2683131392.0, + "86": 2683131392.0, + "87": 2683131392.0, + "88": 2683131392.0, + "89": 2683131392.0, + "90": 2683131392.0, + "91": 2683131392.0, + "92": 2683131392.0, + "93": 2683131392.0, + "94": 2683131392.0, + "95": 2683131392.0, + "96": 2683131392.0, + "97": 2683131392.0, + "98": 2683131392.0, + "99": 2683131392.0, + "100": 2683131392.0 + } + }, + "iteration-time": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": "nan", + "2": 6.48857, + "3": 1.89418, + "4": 1.53691, + "5": 1.15516, + "6": 1.64315, + "7": 1.22145, + "8": 1.65177, + "9": 1.75204, + "10": 1.47171, + "11": 1.33951, + "12": 1.30387, + "13": 1.75992, + "14": 1.78168, + "15": 1.3185, + "16": 1.70239, + "17": 2.02865, + "18": 1.94054, + "19": 1.87203, + "20": 1.34121, + "21": 1.50995, + "22": 1.55715, + "23": 1.93197, + "24": 1.44159, + "25": 1.66356, + "26": 1.90806, + "27": 1.41765, + "28": 1.52994, + "29": 1.21929, + "30": 1.31161, + "31": 1.25439, + "32": 2.35802, + "33": 1.00648, + "34": 1.25844, + "35": 1.77927, + "36": 1.38216, + "37": 1.3944, + "38": 1.10439, + "39": 1.64635, + "40": 1.53005, + "41": 1.42554, + "42": 1.91492, + "43": 1.37824, + "44": 1.45273, + "45": 1.87076, + "46": 1.42798, + "47": 1.21565, + "48": 1.81053, + "49": 1.54495, + "50": 1.73788, + "51": 1.04315, + "52": 1.67002, + "53": 1.49653, + "54": 1.77176, + "55": 1.46788, + "56": 1.65928, + "57": 0.92023, + "58": 1.64773, + "59": 1.41717, + "60": 1.07688, + "61": 1.83408, + "62": 1.77747, + "63": 1.23518, + "64": 1.6607, + "65": 1.48787, + "66": 1.60066, + "67": 1.61047, + "68": 1.01261, + "69": 1.66735, + "70": 1.54395, + "71": 1.73855, + "72": 1.44629, + "73": 1.38554, + "74": 1.47909, + "75": 2.06308, + "76": 1.45855, + "77": 1.45876, + "78": 1.37273, + "79": 1.88268, + "80": 1.4147, + "81": 1.58799, + "82": 1.66045, + "83": 1.42833, + "84": 1.21598, + "85": 1.25409, + "86": 1.24572, + "87": 1.86805, + "88": 1.46625, + "89": 1.70062, + "90": 1.77392, + "91": 1.6723, + "92": 2.28204, + "93": 1.50609, + "94": 1.86821, + "95": 1.74028, + "96": 1.95599, + "97": 1.65212, + "98": 1.93274, + "99": 2.3596, + "100": 1.5986 + } + } +} \ No newline at end of file diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_1node/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_1node/model_config.yaml new file mode 100644 index 00000000000..0a1f510fc8f --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_1node/model_config.yaml @@ -0,0 +1,57 @@ +ENV_VARS: + CUDA_DEVICE_MAX_CONNECTIONS: 1 + NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0 + NCCL_ALGO: Ring + CUBLAS_WORKSPACE_CONFIG: :4096:8 +MODEL_ARGS: + --num-layers: 12 + --hidden-size: 512 + --num-attention-heads: 8 + --log-params-norm: true + --log-num-zeros-in-grad: true + --log-validation-ppl-to-tensorboard: true + --log-timers-to-tensorboard: true + --tensorboard-dir: ${TENSORBOARD_PATH} + --micro-batch-size: 4 + --global-batch-size: 32 + --seq-length: 1024 + --max-position-embeddings: 1024 + --train-iters: 100 + --timing-log-level: 0 + --lr-decay-iters: 320000 + --save: ${CHECKPOINT_SAVE_PATH} + --load: ${CHECKPOINT_LOAD_PATH} + --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document + --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json + --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt + --split: 949,50,1 + --distributed-backend: nccl + --lr: 0.00015 + --lr-decay-style: cosine + --min-lr: 1.0e-5 + --weight-decay: 1e-2 + --clip-grad: 1.0 + --lr-warmup-fraction: .01 + --log-interval: 1 + --save-interval: 50 + --eval-interval: 1000 + --eval-iters: 10 + --transformer-impl: transformer_engine + --tensor-model-parallel-size: 1 + --pipeline-model-parallel-size: 4 + --num-layers-per-virtual-pipeline-stage: 1 + --use-distributed-optimizer: true + --overlap-grad-reduce: true + --deterministic-mode: true + --no-gradient-accumulation-fusion: true + --attention-softmax-in-fp32: true + --use-checkpoint-opt_param-scheduler: true + --use-mcore-models: true + --ckpt-format: torch_dist + --dist-ckpt-strictness: log_all # backward compatibility for TE changes + --data-cache-path: ${DATA_CACHE_PATH} + --bf16: true + --log-memory-to-tensorboard: true + --async-save: true + --use-persistent-ckpt-worker: true +TEST_TYPE: ckpt-resume diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_tunable_overlap_1node/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_tunable_overlap_1node/model_config.yaml new file mode 100644 index 00000000000..7dcdd335e8c --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_tunable_overlap_1node/model_config.yaml @@ -0,0 +1,59 @@ +ENV_VARS: + CUDA_DEVICE_MAX_CONNECTIONS: 1 + NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0 + NCCL_ALGO: Ring + CUBLAS_WORKSPACE_CONFIG: :4096:8 + PYTORCH_CUDA_ALLOC_CONF: expandable_segments:True +MODEL_ARGS: + --num-layers: 12 + --hidden-size: 512 + --num-attention-heads: 8 + --log-params-norm: true + --log-num-zeros-in-grad: true + --log-validation-ppl-to-tensorboard: true + --log-timers-to-tensorboard: true + --tensorboard-dir: ${TENSORBOARD_PATH} + --micro-batch-size: 4 + --global-batch-size: 40 + --seq-length: 1024 + --max-position-embeddings: 1024 + --train-iters: 50 + --timing-log-level: 0 + --lr-decay-iters: 320000 + --save: ${CHECKPOINT_SAVE_PATH} + --load: ${CHECKPOINT_LOAD_PATH} + --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document + --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json + --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt + --split: 949,50,1 + --distributed-backend: nccl + --lr: 0.00015 + --lr-decay-style: cosine + --min-lr: 1.0e-5 + --weight-decay: 1e-2 + --clip-grad: 1.0 + --lr-warmup-fraction: .01 + --log-interval: 1 + --save-interval: 10000 + --eval-interval: 1000 + --eval-iters: 10 + --transformer-impl: transformer_engine + --tensor-model-parallel-size: 1 + --pipeline-model-parallel-size: 4 + --num-layers-per-virtual-pipeline-stage: 1 + --overlap-p2p-communication-warmup-flush: true + --microbatch-group-size-per-virtual-pipeline-stage: 5 + --deterministic-mode: true + --no-gradient-accumulation-fusion: true + --attention-softmax-in-fp32: true + --use-mcore-models: true + --ckpt-format: torch_dist + --dist-ckpt-optim-fully-reshardable: true + --dist-ckpt-strictness: log_all # backward compatibility for TE changes + --data-cache-path: ${DATA_CACHE_PATH} + --bf16: true + --attention-backend: unfused + --log-memory-to-tensorboard: true + --async-save: true + --use-persistent-ckpt-worker: true +TEST_TYPE: regular diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_uneven_pipeline_1node/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_uneven_pipeline_1node/golden_values_dev_dgx_gb200.json new file mode 100644 index 00000000000..2e145054d9c --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_uneven_pipeline_1node/golden_values_dev_dgx_gb200.json @@ -0,0 +1,287 @@ +{ + "lm loss": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 10.91778, + "2": 10.91602, + "3": 10.9198, + "4": 10.91999, + "5": 10.91201, + "6": 10.91954, + "7": 10.92034, + "8": 10.91471, + "9": 10.90787, + "10": 10.90661, + "11": 10.90292, + "12": 10.90754, + "13": 10.90318, + "14": 10.8975, + "15": 10.86511, + "16": 10.86752, + "17": 10.8629, + "18": 10.85425, + "19": 10.85849, + "20": 10.77184, + "21": 10.76543, + "22": 10.74101, + "23": 10.73028, + "24": 10.71854, + "25": 10.69262, + "26": 10.68889, + "27": 10.63789, + "28": 10.57948, + "29": 10.54148, + "30": 10.50449, + "31": 10.51059, + "32": 10.4944, + "33": 10.4464, + "34": 10.4268, + "35": 10.42715, + "36": 10.39378, + "37": 10.37391, + "38": 10.36201, + "39": 10.33359, + "40": 10.30551, + "41": 10.29099, + "42": 10.25817, + "43": 10.24905, + "44": 10.21682, + "45": 10.227, + "46": 10.17061, + "47": 10.17437, + "48": 10.13475, + "49": 10.13473, + "50": 10.11788 + } + }, + "num-zeros": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 22961698.0, + "2": 22850648.0, + "3": 22708944.0, + "4": 22777238.0, + "5": 22782670.0, + "6": 22745544.0, + "7": 22871068.0, + "8": 22611328.0, + "9": 22758860.0, + "10": 22488844.0, + "11": 22752972.0, + "12": 22641496.0, + "13": 23322216.0, + "14": 22991336.0, + "15": 22717318.0, + "16": 22822488.0, + "17": 22929414.0, + "18": 22994556.0, + "19": 23084612.0, + "20": 22722388.0, + "21": 22918198.0, + "22": 22944376.0, + "23": 22628422.0, + "24": 22862476.0, + "25": 22636572.0, + "26": 23004492.0, + "27": 22801540.0, + "28": 22998012.0, + "29": 22979304.0, + "30": 22944624.0, + "31": 22907846.0, + "32": 22665480.0, + "33": 22739828.0, + "34": 23075384.0, + "35": 22750508.0, + "36": 22694996.0, + "37": 23103238.0, + "38": 22963552.0, + "39": 22984982.0, + "40": 22749680.0, + "41": 23065176.0, + "42": 22688580.0, + "43": 22987652.0, + "44": 22704676.0, + "45": 22847124.0, + "46": 22734064.0, + "47": 22849372.0, + "48": 22832772.0, + "49": 22888554.0, + "50": 22644322.0 + } + }, + "mem-allocated-bytes": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 688128512.0, + "2": 688128512.0, + "3": 688128512.0, + "4": 688128512.0, + "5": 688128512.0, + "6": 688128512.0, + "7": 688128512.0, + "8": 688128512.0, + "9": 688128512.0, + "10": 688128512.0, + "11": 688128512.0, + "12": 688128512.0, + "13": 688128512.0, + "14": 688128512.0, + "15": 688128512.0, + "16": 688128512.0, + "17": 688128512.0, + "18": 688128512.0, + "19": 688128512.0, + "20": 688128512.0, + "21": 688128512.0, + "22": 688128512.0, + "23": 688128512.0, + "24": 688128512.0, + "25": 688128512.0, + "26": 688128512.0, + "27": 688128512.0, + "28": 688128512.0, + "29": 688128512.0, + "30": 688128512.0, + "31": 688128512.0, + "32": 688128512.0, + "33": 688128512.0, + "34": 688128512.0, + "35": 688128512.0, + "36": 688128512.0, + "37": 688128512.0, + "38": 688128512.0, + "39": 688128512.0, + "40": 688128512.0, + "41": 688128512.0, + "42": 688128512.0, + "43": 688128512.0, + "44": 688128512.0, + "45": 688128512.0, + "46": 688128512.0, + "47": 688128512.0, + "48": 688128512.0, + "49": 688128512.0, + "50": 688128512.0 + } + }, + "mem-max-allocated-bytes": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 2158027264.0, + "2": 2415568384.0, + "3": 2415568384.0, + "4": 2415568384.0, + "5": 2415568384.0, + "6": 2415568384.0, + "7": 2415568384.0, + "8": 2415568384.0, + "9": 2415568384.0, + "10": 2415568384.0, + "11": 2415568384.0, + "12": 2415568384.0, + "13": 2415568384.0, + "14": 2415568384.0, + "15": 2415568384.0, + "16": 2415568384.0, + "17": 2415568384.0, + "18": 2415568384.0, + "19": 2415568384.0, + "20": 2415568384.0, + "21": 2415568384.0, + "22": 2415568384.0, + "23": 2415568384.0, + "24": 2415568384.0, + "25": 2415568384.0, + "26": 2415568384.0, + "27": 2415568384.0, + "28": 2415568384.0, + "29": 2415568384.0, + "30": 2415568384.0, + "31": 2415568384.0, + "32": 2415568384.0, + "33": 2415568384.0, + "34": 2415568384.0, + "35": 2415568384.0, + "36": 2415568384.0, + "37": 2415568384.0, + "38": 2415568384.0, + "39": 2415568384.0, + "40": 2415568384.0, + "41": 2415568384.0, + "42": 2415568384.0, + "43": 2415568384.0, + "44": 2415568384.0, + "45": 2415568384.0, + "46": 2415568384.0, + "47": 2415568384.0, + "48": 2415568384.0, + "49": 2415568384.0, + "50": 2415568384.0 + } + }, + "iteration-time": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": "nan", + "2": 5.81561, + "3": 1.61423, + "4": 1.7743, + "5": 1.58849, + "6": 1.77427, + "7": 1.44405, + "8": 1.89746, + "9": 1.87642, + "10": 1.85317, + "11": 1.98502, + "12": 1.53301, + "13": 2.11746, + "14": 2.06467, + "15": 1.30939, + "16": 1.55104, + "17": 1.9754, + "18": 1.72668, + "19": 1.7842, + "20": 1.37505, + "21": 1.75157, + "22": 1.76143, + "23": 2.02366, + "24": 1.34304, + "25": 2.04863, + "26": 1.65142, + "27": 1.54773, + "28": 1.11394, + "29": 1.48097, + "30": 1.70539, + "31": 1.25705, + "32": 2.10407, + "33": 1.00854, + "34": 1.11545, + "35": 1.92515, + "36": 1.86816, + "37": 1.81912, + "38": 1.21316, + "39": 2.205, + "40": 1.99058, + "41": 1.92051, + "42": 1.86634, + "43": 1.4904, + "44": 1.28791, + "45": 1.65553, + "46": 1.11328, + "47": 1.5837, + "48": 1.73323, + "49": 1.57162, + "50": 1.59122 + } + } +} \ No newline at end of file diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_uneven_pipeline_1node/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_uneven_pipeline_1node/model_config.yaml new file mode 100644 index 00000000000..e068c864d10 --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp1_uneven_pipeline_1node/model_config.yaml @@ -0,0 +1,57 @@ +ENV_VARS: + CUDA_DEVICE_MAX_CONNECTIONS: 1 + NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0 + NCCL_ALGO: Ring + CUBLAS_WORKSPACE_CONFIG: :4096:8 +MODEL_ARGS: + --num-layers: 12 + --hidden-size: 512 + --num-attention-heads: 8 + --log-params-norm: true + --log-num-zeros-in-grad: true + --log-validation-ppl-to-tensorboard: true + --log-timers-to-tensorboard: true + --tensorboard-dir: ${TENSORBOARD_PATH} + --micro-batch-size: 4 + --global-batch-size: 32 + --seq-length: 1024 + --max-position-embeddings: 1024 + --train-iters: 50 + --timing-log-level: 0 + --lr-decay-iters: 320000 + --save: ${CHECKPOINT_SAVE_PATH} + --load: ${CHECKPOINT_LOAD_PATH} + --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document + --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json + --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt + --split: 949,50,1 + --distributed-backend: nccl + --lr: 0.00015 + --lr-decay-style: cosine + --min-lr: 1.0e-5 + --weight-decay: 1e-2 + --clip-grad: 1.0 + --lr-warmup-fraction: .01 + --log-interval: 1 + --save-interval: 10000 + --eval-interval: 1000 + --eval-iters: 10 + --transformer-impl: transformer_engine + --tensor-model-parallel-size: 1 + --pipeline-model-parallel-size: 4 + --untie-embeddings-and-output-weights: true + --deterministic-mode: true + --no-gradient-accumulation-fusion: true + --attention-softmax-in-fp32: true + --use-mcore-models: true + --ckpt-format: torch_dist + --dist-ckpt-strictness: log_all # backward compatibility for TE changes + --data-cache-path: ${DATA_CACHE_PATH} + --bf16: true + --decoder-first-pipeline-num-layers: 2 + --decoder-last-pipeline-num-layers: 2 + --attention-backend: unfused + --log-memory-to-tensorboard: true + --async-save: true + --use-persistent-ckpt-worker: true +TEST_TYPE: regular diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp2_account_for_embedding_loss_in_pipeline_split_1node/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp2_account_for_embedding_loss_in_pipeline_split_1node/golden_values_dev_dgx_gb200.json new file mode 100644 index 00000000000..c4a4f44df1d --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp2_account_for_embedding_loss_in_pipeline_split_1node/golden_values_dev_dgx_gb200.json @@ -0,0 +1,287 @@ +{ + "lm loss": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 10.90673, + "2": 10.91263, + "3": 10.91376, + "4": 10.90507, + "5": 10.90666, + "6": 10.9031, + "7": 10.9059, + "8": 10.91387, + "9": 10.90878, + "10": 10.9237, + "11": 10.90423, + "12": 10.90022, + "13": 10.91206, + "14": 10.91487, + "15": 10.89545, + "16": 10.88862, + "17": 10.89879, + "18": 10.88901, + "19": 10.89342, + "20": 10.85175, + "21": 10.85495, + "22": 10.85415, + "23": 10.84295, + "24": 10.82607, + "25": 10.82944, + "26": 10.83505, + "27": 10.81605, + "28": 10.76187, + "29": 10.73276, + "30": 10.71482, + "31": 10.70796, + "32": 10.71541, + "33": 10.683, + "34": 10.6638, + "35": 10.66095, + "36": 10.64805, + "37": 10.62756, + "38": 10.60871, + "39": 10.55474, + "40": 10.54268, + "41": 10.52944, + "42": 10.51104, + "43": 10.50087, + "44": 10.46797, + "45": 10.47461, + "46": 10.4319, + "47": 10.42659, + "48": 10.39161, + "49": 10.3871, + "50": 10.36849 + } + }, + "num-zeros": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 22962016.0, + "2": 22848584.0, + "3": 22708364.0, + "4": 22777148.0, + "5": 22782800.0, + "6": 22745854.0, + "7": 22870548.0, + "8": 22610756.0, + "9": 22757820.0, + "10": 22489088.0, + "11": 22753460.0, + "12": 22640528.0, + "13": 23321632.0, + "14": 22990940.0, + "15": 22717336.0, + "16": 22821172.0, + "17": 22928714.0, + "18": 22994028.0, + "19": 23084446.0, + "20": 22722252.0, + "21": 22917710.0, + "22": 22945000.0, + "23": 22627552.0, + "24": 22861514.0, + "25": 22635880.0, + "26": 23004104.0, + "27": 22801224.0, + "28": 22998068.0, + "29": 22978356.0, + "30": 22943064.0, + "31": 22906302.0, + "32": 22664320.0, + "33": 22739984.0, + "34": 23075192.0, + "35": 22749556.0, + "36": 22693736.0, + "37": 23103292.0, + "38": 22964034.0, + "39": 22983724.0, + "40": 22749088.0, + "41": 23063492.0, + "42": 22687832.0, + "43": 22986674.0, + "44": 22703486.0, + "45": 22846044.0, + "46": 22732872.0, + "47": 22848216.0, + "48": 22832092.0, + "49": 22887284.0, + "50": 22642954.0 + } + }, + "mem-allocated-bytes": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 640822784.0, + "2": 640822784.0, + "3": 640822784.0, + "4": 640822784.0, + "5": 640822784.0, + "6": 640822784.0, + "7": 640822784.0, + "8": 640822784.0, + "9": 640822784.0, + "10": 640822784.0, + "11": 640822784.0, + "12": 640822784.0, + "13": 640822784.0, + "14": 640822784.0, + "15": 640822784.0, + "16": 640822784.0, + "17": 640822784.0, + "18": 640822784.0, + "19": 640822784.0, + "20": 640822784.0, + "21": 640822784.0, + "22": 640822784.0, + "23": 640822784.0, + "24": 640822784.0, + "25": 640822784.0, + "26": 640822784.0, + "27": 640822784.0, + "28": 640822784.0, + "29": 640822784.0, + "30": 640822784.0, + "31": 640822784.0, + "32": 640822784.0, + "33": 640822784.0, + "34": 640822784.0, + "35": 640822784.0, + "36": 640822784.0, + "37": 640822784.0, + "38": 640822784.0, + "39": 640822784.0, + "40": 640822784.0, + "41": 640822784.0, + "42": 640822784.0, + "43": 640822784.0, + "44": 640822784.0, + "45": 640822784.0, + "46": 640822784.0, + "47": 640822784.0, + "48": 640822784.0, + "49": 640822784.0, + "50": 640822784.0 + } + }, + "mem-max-allocated-bytes": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 2610027008.0, + "2": 2843266560.0, + "3": 2843266560.0, + "4": 2843268608.0, + "5": 2843268608.0, + "6": 2843268608.0, + "7": 2843268608.0, + "8": 2843268608.0, + "9": 2843268608.0, + "10": 2843268608.0, + "11": 2843268608.0, + "12": 2843268608.0, + "13": 2843268608.0, + "14": 2843268608.0, + "15": 2843268608.0, + "16": 2843268608.0, + "17": 2843268608.0, + "18": 2843268608.0, + "19": 2843268608.0, + "20": 2843268608.0, + "21": 2843268608.0, + "22": 2843268608.0, + "23": 2843268608.0, + "24": 2843268608.0, + "25": 2843268608.0, + "26": 2843268608.0, + "27": 2843268608.0, + "28": 2843268608.0, + "29": 2843268608.0, + "30": 2843268608.0, + "31": 2843268608.0, + "32": 2843268608.0, + "33": 2843268608.0, + "34": 2843268608.0, + "35": 2843268608.0, + "36": 2843268608.0, + "37": 2843268608.0, + "38": 2843268608.0, + "39": 2843268608.0, + "40": 2843268608.0, + "41": 2843268608.0, + "42": 2843268608.0, + "43": 2843268608.0, + "44": 2843268608.0, + "45": 2843268608.0, + "46": 2843268608.0, + "47": 2843268608.0, + "48": 2843268608.0, + "49": 2843268608.0, + "50": 2843268608.0 + } + }, + "iteration-time": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": "nan", + "2": 5.67773, + "3": 1.95977, + "4": 1.93376, + "5": 1.44653, + "6": 1.54869, + "7": 1.36796, + "8": 1.82344, + "9": 1.63208, + "10": 1.60682, + "11": 1.49636, + "12": 1.37219, + "13": 1.56632, + "14": 1.9658, + "15": 1.18584, + "16": 1.33671, + "17": 2.05916, + "18": 1.52076, + "19": 2.08932, + "20": 1.36969, + "21": 1.21968, + "22": 1.37833, + "23": 1.62, + "24": 1.06272, + "25": 1.80834, + "26": 1.55266, + "27": 1.34042, + "28": 1.48622, + "29": 1.1836, + "30": 1.42884, + "31": 1.02616, + "32": 2.22716, + "33": 1.17295, + "34": 1.09922, + "35": 1.6344, + "36": 1.53254, + "37": 1.49098, + "38": 1.42991, + "39": 1.69379, + "40": 1.43006, + "41": 1.705, + "42": 2.12016, + "43": 1.42916, + "44": 1.24415, + "45": 1.75688, + "46": 1.40748, + "47": 1.44547, + "48": 1.44888, + "49": 1.19857, + "50": 1.19151 + } + } +} \ No newline at end of file diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp2_account_for_embedding_loss_in_pipeline_split_1node/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp2_account_for_embedding_loss_in_pipeline_split_1node/model_config.yaml new file mode 100644 index 00000000000..b9e66c55bff --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp1_pp4_vp2_account_for_embedding_loss_in_pipeline_split_1node/model_config.yaml @@ -0,0 +1,58 @@ +ENV_VARS: + CUDA_DEVICE_MAX_CONNECTIONS: 1 + NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0 + NCCL_ALGO: Ring + CUBLAS_WORKSPACE_CONFIG: :4096:8 +MODEL_ARGS: + --num-layers: 6 + --hidden-size: 512 + --num-attention-heads: 8 + --log-params-norm: true + --log-num-zeros-in-grad: true + --log-validation-ppl-to-tensorboard: true + --log-timers-to-tensorboard: true + --tensorboard-dir: ${TENSORBOARD_PATH} + --micro-batch-size: 4 + --global-batch-size: 32 + --seq-length: 1024 + --max-position-embeddings: 1024 + --train-iters: 50 + --timing-log-level: 0 + --lr-decay-iters: 320000 + --save: ${CHECKPOINT_SAVE_PATH} + --load: ${CHECKPOINT_LOAD_PATH} + --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document + --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json + --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt + --split: 949,50,1 + --distributed-backend: nccl + --lr: 0.00015 + --lr-decay-style: cosine + --min-lr: 1.0e-5 + --weight-decay: 1e-2 + --clip-grad: 1.0 + --lr-warmup-fraction: .01 + --log-interval: 1 + --save-interval: 10000 + --eval-interval: 1000 + --eval-iters: 10 + --transformer-impl: transformer_engine + --tensor-model-parallel-size: 1 + --pipeline-model-parallel-size: 4 + --num-virtual-stages-per-pipeline-rank: 2 + --untie-embeddings-and-output-weights: true + --deterministic-mode: true + --no-gradient-accumulation-fusion: true + --attention-softmax-in-fp32: true + --use-mcore-models: true + --ckpt-format: torch_dist + --dist-ckpt-strictness: log_all # backward compatibility for TE changes + --data-cache-path: ${DATA_CACHE_PATH} + --bf16: true + --account-for-embedding-in-pipeline-split: true + --account-for-loss-in-pipeline-split: true + --attention-backend: unfused + --log-memory-to-tensorboard: true + --async-save: true + --use-persistent-ckpt-worker: true +TEST_TYPE: regular diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp1_gdn_1node/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp1_gdn_1node/model_config.yaml new file mode 100644 index 00000000000..cc7211c9967 --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp1_gdn_1node/model_config.yaml @@ -0,0 +1,82 @@ +ENV_VARS: + CUDA_DEVICE_MAX_CONNECTIONS: 1 + NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0 + NCCL_ALGO: Ring + CUBLAS_WORKSPACE_CONFIG: :4096:8 + ENABLE_LIGHTWEIGHT_MODE: true +MODEL_ARGS: + # Add network size args + --untie-embeddings-and-output-weights: true + --num-layers: 6 + --hidden-size: 512 + --num-attention-heads: 8 + --group-query-attention: true + --num-query-groups: 2 + --swiglu: true + --position-embedding-type: rope + --rotary-percent: 0.5 + --no-rope-fusion: true #TODO: We can remove this once upgrading to the DEV container + --apply-layernorm-1p: true + --apply-wd-to-qk-layernorm: true + --attention-output-gate: true + --experimental-attention-variant: gated_delta_net + --linear-attention-freq: 3 + --linear-conv-kernel-dim: 4 + --linear-key-head-dim: 64 + --linear-value-head-dim: 64 + --linear-num-key-heads: 4 + --linear-num-value-heads: 8 + # Add MoE args + --num-experts: 32 + --moe-ffn-hidden-size: 64 + --moe-shared-expert-intermediate-size: 64 + --moe-shared-expert-gate: true + --moe-router-load-balancing-type: aux_loss + --moe-router-topk: 8 + --disable-bias-linear: true + --moe-router-dtype: fp32 + # Add logging args + --log-params-norm: true + --log-num-zeros-in-grad: true + --log-validation-ppl-to-tensorboard: true + --log-timers-to-tensorboard: true + --tensorboard-dir: ${TENSORBOARD_PATH} + --micro-batch-size: 4 + --global-batch-size: 32 + --seq-length: 1024 + --max-position-embeddings: 1024 + --train-iters: 50 + --timing-log-level: 0 + --lr-decay-iters: 320000 + --save: ${CHECKPOINT_SAVE_PATH} + --load: ${CHECKPOINT_LOAD_PATH} + --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document + --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json + --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt + --split: 949,50,1 + --distributed-backend: nccl + --lr: 0.00015 + --lr-decay-style: cosine + --min-lr: 1.0e-5 + --weight-decay: 1e-2 + --clip-grad: 1.0 + --lr-warmup-fraction: .01 + --log-interval: 1 + --save-interval: 25 + --eval-interval: 1000 + --eval-iters: 10 + --transformer-impl: transformer_engine + --tensor-model-parallel-size: 2 + --pipeline-model-parallel-size: 1 + --sequence-parallel: true + --untie-embeddings-and-output-weights: true + --deterministic-mode: true + --no-gradient-accumulation-fusion: true + --attention-softmax-in-fp32: true + --use-mcore-models: true + --ckpt-format: torch_dist + --data-cache-path: ${DATA_CACHE_PATH} + --bf16: true + --attention-backend: unfused + --log-memory-to-tensorboard: true +TEST_TYPE: ckpt-resume diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp1_modelopt_distill_resume_1node/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp1_modelopt_distill_resume_1node/model_config.yaml new file mode 100644 index 00000000000..c75a5a81414 --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp1_modelopt_distill_resume_1node/model_config.yaml @@ -0,0 +1,71 @@ +ENV_VARS: + SKIP_PYTEST: 1 + CUDA_DEVICE_MAX_CONNECTIONS: 1 + NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0 + NCCL_ALGO: Ring + CUBLAS_WORKSPACE_CONFIG: :4096:8 + ARTIFACTS_ROOT: /workspace/checkpoints + DISTILL_CONFIG: '{intermediate_layer_pairs: [["decoder.final_layernorm", "decoder.final_layernorm"]], logit_layers: ["output_layer", "output_layer"], skip_lm_loss: true, kd_loss_scale: 10.0}' +BEFORE_SCRIPT: | + mkdir -p ${DATA_CACHE_PATH}/distill && echo $DISTILL_CONFIG | yq -P > ${DATA_CACHE_PATH}/distill/distill_config.yaml +MODEL_ARGS: + --export-te-mcore-model: true + --export-kd-teacher-load: ${DATA_PATH}/model/gpt_dummy_pyt/ckpt/24.10.0_bf16_teacher + --export-kd-cfg: ${DATA_CACHE_PATH}/distill/distill_config.yaml + --auto-detect-ckpt-format: true + --num-layers: 12 + --hidden-size: 512 + --num-attention-heads: 8 + --normalization: RMSNorm + --log-params-norm: true + --log-num-zeros-in-grad: true + --log-validation-ppl-to-tensorboard: true + --log-timers-to-tensorboard: true + --tensorboard-dir: ${TENSORBOARD_PATH} + --micro-batch-size: 2 + --global-batch-size: 16 + --seq-length: 1024 + --max-position-embeddings: 1024 + --position-embedding-type: rope + --no-rope-fusion: true #TODO: We can remove this once upgrading to the DEV container + --rotary-percent: 0.5 + --swiglu: true + --untie-embeddings-and-output-weights: true + --disable-bias-linear: true + --train-iters: 100 + --timing-log-level: 0 + --lr-decay-iters: 320000 + --save: ${CHECKPOINT_SAVE_PATH} + --load: ${CHECKPOINT_LOAD_PATH} + --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document + --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json + --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt + --split: 949,50,1 + --distributed-backend: nccl + --lr: 0.00015 + --lr-decay-style: cosine + --min-lr: 1.0e-5 + --weight-decay: 1e-2 + --clip-grad: 1.0 + --lr-warmup-fraction: .01 + --use-distributed-optimizer: true + --log-interval: 1 + --save-interval: 50 + --eval-interval: 1000 + --eval-iters: 10 + --transformer-impl: transformer_engine + --tensor-model-parallel-size: 2 + --pipeline-model-parallel-size: 1 + --sequence-parallel: true + --deterministic-mode: true + --no-gradient-accumulation-fusion: true + --use-checkpoint-opt_param-scheduler: true + --ckpt-format: torch_dist + --dist-ckpt-strictness: log_all # backward compatibility for TE changes + --data-cache-path: ${DATA_CACHE_PATH} + --bf16: true + --attention-backend: unfused + --log-memory-to-tensorboard: true + --async-save: true + --use-persistent-ckpt-worker: true +TEST_TYPE: ckpt-resume diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp1_resume_torch_dist_cp2_nondeterministic_1node/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp1_resume_torch_dist_cp2_nondeterministic_1node/golden_values_dev_dgx_gb200.json new file mode 100644 index 00000000000..698afb0e549 --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp1_resume_torch_dist_cp2_nondeterministic_1node/golden_values_dev_dgx_gb200.json @@ -0,0 +1,537 @@ +{ + "lm loss": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 10.95727, + "2": 10.94891, + "3": 10.95179, + "4": 10.93702, + "5": 10.9434, + "6": 10.93813, + "7": 10.94908, + "8": 10.92912, + "9": 10.94297, + "10": 10.93263, + "11": 10.93261, + "12": 10.92193, + "13": 10.91841, + "14": 10.91221, + "15": 10.89101, + "16": 10.87487, + "17": 10.88663, + "18": 10.86795, + "19": 10.87323, + "20": 10.77512, + "21": 10.76178, + "22": 10.75198, + "23": 10.73998, + "24": 10.70111, + "25": 10.71374, + "26": 10.6855, + "27": 10.64306, + "28": 10.56422, + "29": 10.53587, + "30": 10.52934, + "31": 10.51592, + "32": 10.49935, + "33": 10.4655, + "34": 10.42361, + "35": 10.42253, + "36": 10.41395, + "37": 10.3677, + "38": 10.37996, + "39": 10.34012, + "40": 10.3256, + "41": 10.3051, + "42": 10.29477, + "43": 10.25702, + "44": 10.23375, + "45": 10.24531, + "46": 10.21281, + "47": 10.19589, + "48": 10.15323, + "49": 10.14468, + "50": 10.1515, + "51": 10.15987, + "52": 10.10839, + "53": 10.10794, + "54": 10.07446, + "55": 10.04843, + "56": 10.07749, + "57": 10.05892, + "58": 10.07687, + "59": 10.02269, + "60": 10.03903, + "61": 9.99479, + "62": 9.96373, + "63": 10.03991, + "64": 9.99852, + "65": 9.96345, + "66": 9.99446, + "67": 9.96877, + "68": 9.92808, + "69": 9.94599, + "70": 9.93082, + "71": 9.96385, + "72": 9.92605, + "73": 9.91707, + "74": 9.90177, + "75": 9.86957, + "76": 9.91504, + "77": 9.90642, + "78": 9.84565, + "79": 9.85519, + "80": 9.87465, + "81": 9.90488, + "82": 9.84705, + "83": 9.79935, + "84": 9.72983, + "85": 9.72434, + "86": 9.82916, + "87": 9.86552, + "88": 9.83644, + "89": 9.76533, + "90": 9.75736, + "91": 9.77544, + "92": 9.76552, + "93": 9.68785, + "94": 9.77515, + "95": 9.77587, + "96": 9.75058, + "97": 9.67839, + "98": 9.71972, + "99": 9.77325, + "100": 9.65792 + } + }, + "num-zeros": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 634.0, + "2": 628.0, + "3": 608.0, + "4": 678.0, + "5": 634.0, + "6": 632.0, + "7": 629.0, + "8": 679.0, + "9": 660.0, + "10": 698.0, + "11": 637.0, + "12": 622.0, + "13": 652.0, + "14": 606.0, + "15": 628.0, + "16": 671.0, + "17": 675.0, + "18": 609.0, + "19": 605.0, + "20": 669.0, + "21": 641.0, + "22": 660.0, + "23": 639.0, + "24": 680.0, + "25": 641.0, + "26": 683.0, + "27": 657.0, + "28": 642.0, + "29": 732.0, + "30": 700.0, + "31": 769.0, + "32": 783.0, + "33": 734.0, + "34": 812.0, + "35": 763.0, + "36": 808.0, + "37": 840.0, + "38": 809.0, + "39": 850.0, + "40": 848.0, + "41": 844.0, + "42": 819.0, + "43": 817.0, + "44": 848.0, + "45": 953.0, + "46": 926.0, + "47": 922.0, + "48": 951.0, + "49": 969.0, + "50": 1001.0, + "51": 904.0, + "52": 996.0, + "53": 1000.0, + "54": 1038.0, + "55": 999.0, + "56": 1131.0, + "57": 814.0, + "58": 1291.0, + "59": 1056.0, + "60": 1116.0, + "61": 1086.0, + "62": 1186.0, + "63": 1175.0, + "64": 1181.0, + "65": 962.0, + "66": 1140.0, + "67": 1282.0, + "68": 1180.0, + "69": 1023.0, + "70": 1100.0, + "71": 1105.0, + "72": 1060.0, + "73": 1106.0, + "74": 1164.0, + "75": 1013.0, + "76": 1184.0, + "77": 1278.0, + "78": 1140.0, + "79": 1151.0, + "80": 1092.0, + "81": 1177.0, + "82": 1198.0, + "83": 1068.0, + "84": 1109.0, + "85": 1042.0, + "86": 1087.0, + "87": 1095.0, + "88": 1160.0, + "89": 1044.0, + "90": 1300.0, + "91": 1026.0, + "92": 1051.0, + "93": 1139.0, + "94": 1226.0, + "95": 1030.0, + "96": 1235.0, + "97": 1146.0, + "98": 1187.0, + "99": 1116.0, + "100": 1100.0 + } + }, + "mem-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 638631424.0, + "2": 638631424.0, + "3": 638631424.0, + "4": 638631424.0, + "5": 638631424.0, + "6": 638631424.0, + "7": 638631424.0, + "8": 638631424.0, + "9": 638631424.0, + "10": 638631424.0, + "11": 638631424.0, + "12": 638631424.0, + "13": 638631424.0, + "14": 638631424.0, + "15": 638631424.0, + "16": 638631424.0, + "17": 638631424.0, + "18": 638631424.0, + "19": 638631424.0, + "20": 638631424.0, + "21": 638631424.0, + "22": 638631424.0, + "23": 638631424.0, + "24": 638631424.0, + "25": 638631424.0, + "26": 638631424.0, + "27": 638631424.0, + "28": 638631424.0, + "29": 638631424.0, + "30": 638631424.0, + "31": 638631424.0, + "32": 638631424.0, + "33": 638631424.0, + "34": 638631424.0, + "35": 638631424.0, + "36": 638631424.0, + "37": 638631424.0, + "38": 638631424.0, + "39": 638631424.0, + "40": 638631424.0, + "41": 638631424.0, + "42": 638631424.0, + "43": 638631424.0, + "44": 638631424.0, + "45": 638631424.0, + "46": 638631424.0, + "47": 638631424.0, + "48": 638631424.0, + "49": 638631424.0, + "50": 638631424.0, + "51": 638631424.0, + "52": 638631424.0, + "53": 638631424.0, + "54": 638631424.0, + "55": 638631424.0, + "56": 638631424.0, + "57": 638631424.0, + "58": 638631424.0, + "59": 638631424.0, + "60": 638631424.0, + "61": 638631424.0, + "62": 638631424.0, + "63": 638631424.0, + "64": 638631424.0, + "65": 638631424.0, + "66": 638631424.0, + "67": 638631424.0, + "68": 638631424.0, + "69": 638631424.0, + "70": 638631424.0, + "71": 638631424.0, + "72": 638631424.0, + "73": 638631424.0, + "74": 638631424.0, + "75": 638631424.0, + "76": 638631424.0, + "77": 638631424.0, + "78": 638631424.0, + "79": 638631424.0, + "80": 638631424.0, + "81": 638631424.0, + "82": 638631424.0, + "83": 638631424.0, + "84": 638631424.0, + "85": 638631424.0, + "86": 638631424.0, + "87": 638631424.0, + "88": 638631424.0, + "89": 638631424.0, + "90": 638631424.0, + "91": 638631424.0, + "92": 638631424.0, + "93": 638631424.0, + "94": 638631424.0, + "95": 638631424.0, + "96": 638631424.0, + "97": 638631424.0, + "98": 638631424.0, + "99": 638631424.0, + "100": 638631424.0 + } + }, + "mem-max-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 910635520.0, + "2": 1171549184.0, + "3": 1171550720.0, + "4": 1171550720.0, + "5": 1173122048.0, + "6": 1173122048.0, + "7": 1173122560.0, + "8": 1173122560.0, + "9": 1173122560.0, + "10": 1173122560.0, + "11": 1173122560.0, + "12": 1173122560.0, + "13": 1173122560.0, + "14": 1173122560.0, + "15": 1173122560.0, + "16": 1173123072.0, + "17": 1173123072.0, + "18": 1173123072.0, + "19": 1173123072.0, + "20": 1173123072.0, + "21": 1173123072.0, + "22": 1173123072.0, + "23": 1173123072.0, + "24": 1173123072.0, + "25": 1173123584.0, + "26": 1173123584.0, + "27": 1173123584.0, + "28": 1173123584.0, + "29": 1173123584.0, + "30": 1173123584.0, + "31": 1173123584.0, + "32": 1173123584.0, + "33": 1173123584.0, + "34": 1179413504.0, + "35": 1179413504.0, + "36": 1179413504.0, + "37": 1179413504.0, + "38": 1179413504.0, + "39": 1179413504.0, + "40": 1179413504.0, + "41": 1179413504.0, + "42": 1179413504.0, + "43": 1179413504.0, + "44": 1179413504.0, + "45": 1179413504.0, + "46": 1179413504.0, + "47": 1179413504.0, + "48": 1179413504.0, + "49": 1179413504.0, + "50": 1179413504.0, + "51": 1179413504.0, + "52": 1179413504.0, + "53": 1179413504.0, + "54": 1179413504.0, + "55": 1179413504.0, + "56": 1179413504.0, + "57": 1179413504.0, + "58": 1179413504.0, + "59": 1179413504.0, + "60": 1179413504.0, + "61": 1179413504.0, + "62": 1179413504.0, + "63": 1179413504.0, + "64": 1179413504.0, + "65": 1179413504.0, + "66": 1179413504.0, + "67": 1179413504.0, + "68": 1179413504.0, + "69": 1179413504.0, + "70": 1179413504.0, + "71": 1179413504.0, + "72": 1179413504.0, + "73": 1179413504.0, + "74": 1179413504.0, + "75": 1179413504.0, + "76": 1179413504.0, + "77": 1179413504.0, + "78": 1179937792.0, + "79": 1179937792.0, + "80": 1179937792.0, + "81": 1179937792.0, + "82": 1179937792.0, + "83": 1179937792.0, + "84": 1179937792.0, + "85": 1179937792.0, + "86": 1179937792.0, + "87": 1179937792.0, + "88": 1179937792.0, + "89": 1179937792.0, + "90": 1179937792.0, + "91": 1179937792.0, + "92": 1179937792.0, + "93": 1179937792.0, + "94": 1179937792.0, + "95": 1179937792.0, + "96": 1179937792.0, + "97": 1179937792.0, + "98": 1179937792.0, + "99": 1179937792.0, + "100": 1179937792.0 + } + }, + "iteration-time": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": "nan", + "2": 5.94784, + "3": 2.08989, + "4": 1.62854, + "5": 1.49412, + "6": 2.06668, + "7": 1.44995, + "8": 2.12202, + "9": 2.25984, + "10": 1.89714, + "11": 1.96681, + "12": 1.8059, + "13": 1.8177, + "14": 1.89502, + "15": 1.5108, + "16": 1.69645, + "17": 2.25448, + "18": 1.71528, + "19": 1.44119, + "20": 1.58651, + "21": 1.76674, + "22": 1.41152, + "23": 1.76179, + "24": 1.68752, + "25": 1.97738, + "26": 2.10964, + "27": 1.66462, + "28": 1.73044, + "29": 1.44311, + "30": 1.65568, + "31": 1.27011, + "32": 2.68701, + "33": 1.3054, + "34": 1.26904, + "35": 2.04492, + "36": 1.62545, + "37": 1.42593, + "38": 1.61933, + "39": 1.98633, + "40": 1.76601, + "41": 1.88481, + "42": 2.19483, + "43": 1.6052, + "44": 1.82284, + "45": 2.03776, + "46": 1.42544, + "47": 1.85168, + "48": 1.9093, + "49": 1.21813, + "50": 1.74689, + "51": 1.74951, + "52": 1.84263, + "53": 2.0954, + "54": 2.40718, + "55": 1.81303, + "56": 1.96736, + "57": 1.11692, + "58": 2.01276, + "59": 1.63557, + "60": 1.60161, + "61": 1.66175, + "62": 1.82999, + "63": 1.37127, + "64": 1.27688, + "65": 1.33288, + "66": 1.65871, + "67": 1.73508, + "68": 1.25249, + "69": 1.43039, + "70": 1.57786, + "71": 2.09041, + "72": 1.61361, + "73": 1.64157, + "74": 1.30333, + "75": 1.52759, + "76": 1.69149, + "77": 1.70182, + "78": 1.81036, + "79": 2.23472, + "80": 1.86222, + "81": 1.94715, + "82": 1.90718, + "83": 1.91791, + "84": 1.69936, + "85": 1.82649, + "86": 1.70956, + "87": 2.16816, + "88": 2.15716, + "89": 1.61203, + "90": 2.29268, + "91": 1.12311, + "92": 2.26283, + "93": 1.47039, + "94": 2.23172, + "95": 1.90627, + "96": 1.93021, + "97": 1.73481, + "98": 1.68955, + "99": 2.39816, + "100": 1.62725 + } + } +} \ No newline at end of file diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp1_resume_torch_dist_cp2_nondeterministic_1node/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp1_resume_torch_dist_cp2_nondeterministic_1node/model_config.yaml new file mode 100644 index 00000000000..bd303906bd6 --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp1_resume_torch_dist_cp2_nondeterministic_1node/model_config.yaml @@ -0,0 +1,57 @@ +ENV_VARS: + CUDA_DEVICE_MAX_CONNECTIONS: 1 + NVTE_ALLOW_NONDETERMINISTIC_ALGO: 1 +MODEL_ARGS: + --num-layers: 12 + --hidden-size: 512 + --num-attention-heads: 8 + --log-params-norm: true + --log-num-zeros-in-grad: true + --log-validation-ppl-to-tensorboard: true + --log-timers-to-tensorboard: true + --tensorboard-dir: ${TENSORBOARD_PATH} + --micro-batch-size: 4 + --global-batch-size: 32 + --seq-length: 1024 + --max-position-embeddings: 1024 + --train-iters: 100 + --timing-log-level: 0 + --lr-decay-iters: 320000 + --save: ${CHECKPOINT_SAVE_PATH} + --load: ${CHECKPOINT_LOAD_PATH} + --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document + --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json + --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt + --split: 949,50,1 + --distributed-backend: nccl + --lr: 0.00015 + --lr-decay-style: cosine + --min-lr: 1.0e-5 + --weight-decay: 1e-2 + --clip-grad: 1.0 + --lr-warmup-fraction: .01 + --log-interval: 1 + --save-interval: 50 + --eval-interval: 1000 + --eval-iters: 10 + --transformer-impl: transformer_engine + --tensor-model-parallel-size: 2 + --pipeline-model-parallel-size: 1 + --context-parallel-size: 2 + --sequence-parallel: true + --hidden-dropout: 0.0 + --attention-dropout: 0.0 + --no-gradient-accumulation-fusion: true + --attention-softmax-in-fp32: true + --use-checkpoint-opt_param-scheduler: true + --use-mcore-models: true + --ckpt-format: torch_dist + --dist-ckpt-optim-fully-reshardable: true + --dist-ckpt-strictness: log_all # backward compatibility for TE changes + --data-cache-path: ${DATA_CACHE_PATH} + --bf16: true + --attention-backend: flash + --log-memory-to-tensorboard: true + --async-save: true + --use-persistent-ckpt-worker: true +TEST_TYPE: ckpt-resume diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp1_resume_torch_dist_multi_dist_optimizer_instances_1node/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp1_resume_torch_dist_multi_dist_optimizer_instances_1node/golden_values_dev_dgx_gb200.json new file mode 100644 index 00000000000..e4c41e6a17d --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp1_resume_torch_dist_multi_dist_optimizer_instances_1node/golden_values_dev_dgx_gb200.json @@ -0,0 +1,537 @@ +{ + "lm loss": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 10.95267, + "2": 10.9487, + "3": 10.95574, + "4": 10.94431, + "5": 10.94704, + "6": 10.94549, + "7": 10.95237, + "8": 10.93777, + "9": 10.94948, + "10": 10.93746, + "11": 10.93723, + "12": 10.93039, + "13": 10.9262, + "14": 10.92334, + "15": 10.90308, + "16": 10.8917, + "17": 10.90057, + "18": 10.88579, + "19": 10.88639, + "20": 10.8026, + "21": 10.79321, + "22": 10.78322, + "23": 10.77587, + "24": 10.73896, + "25": 10.75417, + "26": 10.72426, + "27": 10.68676, + "28": 10.60946, + "29": 10.58265, + "30": 10.57715, + "31": 10.56009, + "32": 10.54731, + "33": 10.51567, + "34": 10.47312, + "35": 10.47454, + "36": 10.46305, + "37": 10.41894, + "38": 10.43058, + "39": 10.39286, + "40": 10.37587, + "41": 10.35957, + "42": 10.34601, + "43": 10.30948, + "44": 10.28618, + "45": 10.29673, + "46": 10.26334, + "47": 10.24621, + "48": 10.20438, + "49": 10.19586, + "50": 10.1985, + "51": 10.20538, + "52": 10.15528, + "53": 10.15622, + "54": 10.12185, + "55": 10.09957, + "56": 10.12109, + "57": 10.10642, + "58": 10.11963, + "59": 10.06902, + "60": 10.0833, + "61": 10.03794, + "62": 10.01121, + "63": 10.07814, + "64": 10.04133, + "65": 10.01344, + "66": 10.03614, + "67": 10.01253, + "68": 9.97682, + "69": 9.99821, + "70": 9.98055, + "71": 10.00729, + "72": 9.98249, + "73": 9.97937, + "74": 9.96109, + "75": 9.93579, + "76": 9.96744, + "77": 9.96184, + "78": 9.90936, + "79": 9.9194, + "80": 9.9384, + "81": 9.96274, + "82": 9.89505, + "83": 9.85891, + "84": 9.7885, + "85": 9.7864, + "86": 9.88552, + "87": 9.91121, + "88": 9.88699, + "89": 9.8184, + "90": 9.81194, + "91": 9.83103, + "92": 9.81784, + "93": 9.74871, + "94": 9.83439, + "95": 9.82929, + "96": 9.80802, + "97": 9.74284, + "98": 9.78138, + "99": 9.82283, + "100": 9.71499 + } + }, + "num-zeros": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 1769.0, + "2": 1730.0, + "3": 1728.0, + "4": 1612.0, + "5": 1692.0, + "6": 1719.0, + "7": 1872.0, + "8": 1657.0, + "9": 1770.0, + "10": 1777.0, + "11": 1697.0, + "12": 1721.0, + "13": 1783.0, + "14": 1784.0, + "15": 1583.0, + "16": 1711.0, + "17": 1743.0, + "18": 1766.0, + "19": 1755.0, + "20": 1676.0, + "21": 1838.0, + "22": 1726.0, + "23": 1650.0, + "24": 1801.0, + "25": 1822.0, + "26": 1818.0, + "27": 1857.0, + "28": 1830.0, + "29": 2024.0, + "30": 1772.0, + "31": 2004.0, + "32": 1984.0, + "33": 1939.0, + "34": 2014.0, + "35": 2190.0, + "36": 2029.0, + "37": 2171.0, + "38": 2076.0, + "39": 2307.0, + "40": 2283.0, + "41": 2352.0, + "42": 2038.0, + "43": 2462.0, + "44": 2201.0, + "45": 2496.0, + "46": 2332.0, + "47": 2431.0, + "48": 2657.0, + "49": 2706.0, + "50": 2523.0, + "51": 2422.0, + "52": 2630.0, + "53": 2633.0, + "54": 2831.0, + "55": 2584.0, + "56": 2781.0, + "57": 2244.0, + "58": 3551.0, + "59": 3006.0, + "60": 2928.0, + "61": 2894.0, + "62": 3200.0, + "63": 3409.0, + "64": 3588.0, + "65": 2745.0, + "66": 3199.0, + "67": 3933.0, + "68": 3489.0, + "69": 3109.0, + "70": 3355.0, + "71": 3067.0, + "72": 2965.0, + "73": 3434.0, + "74": 3343.0, + "75": 3203.0, + "76": 3315.0, + "77": 3776.0, + "78": 3323.0, + "79": 3260.0, + "80": 3268.0, + "81": 3533.0, + "82": 2993.0, + "83": 3035.0, + "84": 2907.0, + "85": 2822.0, + "86": 2891.0, + "87": 2942.0, + "88": 3135.0, + "89": 3215.0, + "90": 3838.0, + "91": 2968.0, + "92": 2759.0, + "93": 3024.0, + "94": 2973.0, + "95": 3242.0, + "96": 3295.0, + "97": 3308.0, + "98": 3192.0, + "99": 3267.0, + "100": 3362.0 + } + }, + "mem-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 687009280.0, + "2": 687009280.0, + "3": 687009280.0, + "4": 687009280.0, + "5": 687009280.0, + "6": 687009280.0, + "7": 687009280.0, + "8": 687009280.0, + "9": 687009280.0, + "10": 687009280.0, + "11": 687009280.0, + "12": 687009280.0, + "13": 687009280.0, + "14": 687009280.0, + "15": 687009280.0, + "16": 687009280.0, + "17": 687009280.0, + "18": 687009280.0, + "19": 687009280.0, + "20": 687009280.0, + "21": 687009280.0, + "22": 687009280.0, + "23": 687009280.0, + "24": 687009280.0, + "25": 687009280.0, + "26": 687009280.0, + "27": 687009280.0, + "28": 687009280.0, + "29": 687009280.0, + "30": 687009280.0, + "31": 687009280.0, + "32": 687009280.0, + "33": 687009280.0, + "34": 687009280.0, + "35": 687009280.0, + "36": 687009280.0, + "37": 687009280.0, + "38": 687009280.0, + "39": 687009280.0, + "40": 687009280.0, + "41": 687009280.0, + "42": 687009280.0, + "43": 687009280.0, + "44": 687009280.0, + "45": 687009280.0, + "46": 687009280.0, + "47": 687009280.0, + "48": 687009280.0, + "49": 687009280.0, + "50": 687009280.0, + "51": 687009280.0, + "52": 687009280.0, + "53": 687009280.0, + "54": 687009280.0, + "55": 687009280.0, + "56": 687009280.0, + "57": 687009280.0, + "58": 687009280.0, + "59": 687009280.0, + "60": 687009280.0, + "61": 687009280.0, + "62": 687009280.0, + "63": 687009280.0, + "64": 687009280.0, + "65": 687009280.0, + "66": 687009280.0, + "67": 687009280.0, + "68": 687009280.0, + "69": 687009280.0, + "70": 687009280.0, + "71": 687009280.0, + "72": 687009280.0, + "73": 687009280.0, + "74": 687009280.0, + "75": 687009280.0, + "76": 687009280.0, + "77": 687009280.0, + "78": 687009280.0, + "79": 687009280.0, + "80": 687009280.0, + "81": 687009280.0, + "82": 687009280.0, + "83": 687009280.0, + "84": 687009280.0, + "85": 687009280.0, + "86": 687009280.0, + "87": 687009280.0, + "88": 687009280.0, + "89": 687009280.0, + "90": 687009280.0, + "91": 687009280.0, + "92": 687009280.0, + "93": 687009280.0, + "94": 687009280.0, + "95": 687009280.0, + "96": 687009280.0, + "97": 687009280.0, + "98": 687009280.0, + "99": 687009280.0, + "100": 687009280.0 + } + }, + "mem-max-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 1642131968.0, + "2": 1902129664.0, + "3": 1902129664.0, + "4": 1902129664.0, + "5": 1902129664.0, + "6": 1902129664.0, + "7": 1902129664.0, + "8": 1902129664.0, + "9": 1902129664.0, + "10": 1902129664.0, + "11": 1902129664.0, + "12": 1902129664.0, + "13": 1902129664.0, + "14": 1902129664.0, + "15": 1902129664.0, + "16": 1902129664.0, + "17": 1902129664.0, + "18": 1902129664.0, + "19": 1902129664.0, + "20": 1902129664.0, + "21": 1902129664.0, + "22": 1902129664.0, + "23": 1902129664.0, + "24": 1902129664.0, + "25": 1902129664.0, + "26": 1902129664.0, + "27": 1902129664.0, + "28": 1902129664.0, + "29": 1902129664.0, + "30": 1902129664.0, + "31": 1902129664.0, + "32": 1902129664.0, + "33": 1902129664.0, + "34": 1902129664.0, + "35": 1902129664.0, + "36": 1902129664.0, + "37": 1902129664.0, + "38": 1902129664.0, + "39": 1902129664.0, + "40": 1902129664.0, + "41": 1902129664.0, + "42": 1902129664.0, + "43": 1902129664.0, + "44": 1902129664.0, + "45": 1902129664.0, + "46": 1902129664.0, + "47": 1902129664.0, + "48": 1902129664.0, + "49": 1902129664.0, + "50": 1902129664.0, + "51": 1903702528.0, + "52": 1903702528.0, + "53": 1903702528.0, + "54": 1903702528.0, + "55": 1903702528.0, + "56": 1903702528.0, + "57": 1903702528.0, + "58": 1903702528.0, + "59": 1903702528.0, + "60": 1903702528.0, + "61": 1903702528.0, + "62": 1903702528.0, + "63": 1903702528.0, + "64": 1903702528.0, + "65": 1903702528.0, + "66": 1903702528.0, + "67": 1903702528.0, + "68": 1903702528.0, + "69": 1903702528.0, + "70": 1903702528.0, + "71": 1903702528.0, + "72": 1903702528.0, + "73": 1903702528.0, + "74": 1903702528.0, + "75": 1903702528.0, + "76": 1903702528.0, + "77": 1903702528.0, + "78": 1903702528.0, + "79": 1903702528.0, + "80": 1903702528.0, + "81": 1903702528.0, + "82": 1903702528.0, + "83": 1903702528.0, + "84": 1903702528.0, + "85": 1903702528.0, + "86": 1903702528.0, + "87": 1903702528.0, + "88": 1903702528.0, + "89": 1903702528.0, + "90": 1903702528.0, + "91": 1903702528.0, + "92": 1903702528.0, + "93": 1903702528.0, + "94": 1903702528.0, + "95": 1903702528.0, + "96": 1903702528.0, + "97": 1903702528.0, + "98": 1903702528.0, + "99": 1903702528.0, + "100": 1903702528.0 + } + }, + "iteration-time": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": "nan", + "2": 4.24044, + "3": 1.20774, + "4": 0.94125, + "5": 0.79955, + "6": 1.09586, + "7": 1.10757, + "8": 1.36003, + "9": 1.46472, + "10": 0.74202, + "11": 1.00679, + "12": 0.98819, + "13": 1.03964, + "14": 1.19787, + "15": 0.7958, + "16": 0.83744, + "17": 1.16784, + "18": 0.97498, + "19": 0.94387, + "20": 0.96183, + "21": 0.92579, + "22": 0.98049, + "23": 1.03962, + "24": 1.41726, + "25": 0.70146, + "26": 0.71921, + "27": 0.93146, + "28": 1.02839, + "29": 0.91762, + "30": 0.72943, + "31": 0.87004, + "32": 1.55458, + "33": 1.13357, + "34": 0.38001, + "35": 1.31151, + "36": 1.30424, + "37": 0.92451, + "38": 0.68198, + "39": 1.14724, + "40": 1.37501, + "41": 1.00481, + "42": 1.161, + "43": 0.74683, + "44": 1.30066, + "45": 1.06816, + "46": 0.90499, + "47": 1.11186, + "48": 0.90036, + "49": 0.78506, + "50": 1.13534, + "51": 0.40522, + "52": 0.99452, + "53": 1.1552, + "54": 1.28012, + "55": 1.18767, + "56": 0.99463, + "57": 0.7799, + "58": 1.01165, + "59": 1.01438, + "60": 0.86387, + "61": 0.88368, + "62": 1.40302, + "63": 0.90476, + "64": 1.24618, + "65": 0.81385, + "66": 0.90104, + "67": 1.02832, + "68": 0.89884, + "69": 1.33259, + "70": 0.9515, + "71": 1.0536, + "72": 1.0425, + "73": 0.93706, + "74": 0.66844, + "75": 0.98577, + "76": 1.20057, + "77": 1.20013, + "78": 1.16248, + "79": 1.56964, + "80": 1.00278, + "81": 0.97408, + "82": 1.3113, + "83": 1.14101, + "84": 0.85463, + "85": 0.71484, + "86": 0.84515, + "87": 1.71943, + "88": 0.97869, + "89": 0.72003, + "90": 1.11243, + "91": 0.8385, + "92": 1.31116, + "93": 0.97175, + "94": 1.39879, + "95": 0.79823, + "96": 1.68543, + "97": 1.19839, + "98": 1.15626, + "99": 1.42586, + "100": 0.98493 + } + } +} \ No newline at end of file diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp1_resume_torch_dist_multi_dist_optimizer_instances_1node/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp1_resume_torch_dist_multi_dist_optimizer_instances_1node/model_config.yaml new file mode 100644 index 00000000000..d226ebb89c5 --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp1_resume_torch_dist_multi_dist_optimizer_instances_1node/model_config.yaml @@ -0,0 +1,60 @@ +ENV_VARS: + CUDA_DEVICE_MAX_CONNECTIONS: 1 + NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0 + NCCL_ALGO: Ring + CUBLAS_WORKSPACE_CONFIG: :4096:8 +MODEL_ARGS: + --num-layers: 12 + --hidden-size: 512 + --num-attention-heads: 8 + --log-params-norm: true + --log-num-zeros-in-grad: true + --log-validation-ppl-to-tensorboard: true + --log-timers-to-tensorboard: true + --tensorboard-dir: ${TENSORBOARD_PATH} + --micro-batch-size: 4 + --global-batch-size: 32 + --seq-length: 1024 + --max-position-embeddings: 1024 + --train-iters: 100 + --timing-log-level: 0 + --lr-decay-iters: 320000 + --save: ${CHECKPOINT_SAVE_PATH} + --load: ${CHECKPOINT_LOAD_PATH} + --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document + --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json + --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt + --split: 949,50,1 + --distributed-backend: nccl + --lr: 0.00015 + --lr-decay-style: cosine + --min-lr: 1.0e-5 + --weight-decay: 1e-2 + --clip-grad: 1.0 + --lr-warmup-fraction: .01 + --log-interval: 1 + --save-interval: 50 + --eval-interval: 1000 + --eval-iters: 10 + --transformer-impl: transformer_engine + --tensor-model-parallel-size: 2 + --pipeline-model-parallel-size: 1 + --use-distributed-optimizer: true + --num-distributed-optimizer-instances: 2 + --overlap-grad-reduce: true + --overlap-param-gather: true + --deterministic-mode: true + --no-gradient-accumulation-fusion: true + --attention-softmax-in-fp32: true + --use-checkpoint-opt_param-scheduler: true + --use-mcore-models: true + --ckpt-format: torch_dist + --dist-ckpt-optim-fully-reshardable: true + --dist-ckpt-strictness: log_all # backward compatibility for TE changes + --data-cache-path: ${DATA_CACHE_PATH} + --bf16: true + --log-memory-to-tensorboard: true + --async-save: true + --use-persistent-ckpt-worker: true +TEST_TYPE: ckpt-resume +LAUNCHER: ft_launcher diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_1node/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_1node/golden_values_dev_dgx_gb200.json new file mode 100644 index 00000000000..ea82be620bf --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_1node/golden_values_dev_dgx_gb200.json @@ -0,0 +1,287 @@ +{ + "lm loss": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 10.95727, + "2": 10.94891, + "3": 10.95181, + "4": 10.93701, + "5": 10.94341, + "6": 10.9381, + "7": 10.94906, + "8": 10.92913, + "9": 10.94291, + "10": 10.93268, + "11": 10.9326, + "12": 10.92193, + "13": 10.91841, + "14": 10.91226, + "15": 10.891, + "16": 10.87484, + "17": 10.88661, + "18": 10.86792, + "19": 10.87319, + "20": 10.77508, + "21": 10.76177, + "22": 10.75197, + "23": 10.73997, + "24": 10.70111, + "25": 10.71372, + "26": 10.68547, + "27": 10.64309, + "28": 10.56423, + "29": 10.53588, + "30": 10.52933, + "31": 10.51593, + "32": 10.49935, + "33": 10.46546, + "34": 10.42362, + "35": 10.42255, + "36": 10.41397, + "37": 10.36771, + "38": 10.37996, + "39": 10.34012, + "40": 10.32558, + "41": 10.30513, + "42": 10.29477, + "43": 10.25699, + "44": 10.23372, + "45": 10.2453, + "46": 10.21282, + "47": 10.19585, + "48": 10.15323, + "49": 10.14471, + "50": 10.15149 + } + }, + "num-zeros": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 652.0, + "2": 628.0, + "3": 651.0, + "4": 603.0, + "5": 648.0, + "6": 672.0, + "7": 646.0, + "8": 637.0, + "9": 647.0, + "10": 666.0, + "11": 630.0, + "12": 627.0, + "13": 648.0, + "14": 598.0, + "15": 603.0, + "16": 597.0, + "17": 647.0, + "18": 570.0, + "19": 581.0, + "20": 612.0, + "21": 628.0, + "22": 681.0, + "23": 689.0, + "24": 651.0, + "25": 653.0, + "26": 660.0, + "27": 668.0, + "28": 670.0, + "29": 787.0, + "30": 654.0, + "31": 739.0, + "32": 721.0, + "33": 751.0, + "34": 729.0, + "35": 794.0, + "36": 819.0, + "37": 818.0, + "38": 785.0, + "39": 803.0, + "40": 874.0, + "41": 860.0, + "42": 726.0, + "43": 861.0, + "44": 774.0, + "45": 953.0, + "46": 942.0, + "47": 867.0, + "48": 920.0, + "49": 986.0, + "50": 993.0 + } + }, + "mem-allocated-bytes": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 640073216.0, + "2": 640073216.0, + "3": 640073216.0, + "4": 640073216.0, + "5": 640073216.0, + "6": 640073216.0, + "7": 640073216.0, + "8": 640073216.0, + "9": 640073216.0, + "10": 640073216.0, + "11": 640073216.0, + "12": 640073216.0, + "13": 640073216.0, + "14": 640073216.0, + "15": 640073216.0, + "16": 640073216.0, + "17": 640073216.0, + "18": 640073216.0, + "19": 640073216.0, + "20": 640073216.0, + "21": 640073216.0, + "22": 640073216.0, + "23": 640073216.0, + "24": 640073216.0, + "25": 640073216.0, + "26": 640073216.0, + "27": 640073216.0, + "28": 640073216.0, + "29": 640073216.0, + "30": 640073216.0, + "31": 640073216.0, + "32": 640073216.0, + "33": 640073216.0, + "34": 640073216.0, + "35": 640073216.0, + "36": 640073216.0, + "37": 640073216.0, + "38": 640073216.0, + "39": 640073216.0, + "40": 640073216.0, + "41": 640073216.0, + "42": 640073216.0, + "43": 640073216.0, + "44": 640073216.0, + "45": 640073216.0, + "46": 640073216.0, + "47": 640073216.0, + "48": 640073216.0, + "49": 640073216.0, + "50": 640073216.0 + } + }, + "mem-max-allocated-bytes": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 911419392.0, + "2": 1178235392.0, + "3": 1182822912.0, + "4": 1183213056.0, + "5": 1183213056.0, + "6": 1183213056.0, + "7": 1183213568.0, + "8": 1183213568.0, + "9": 1183213568.0, + "10": 1183213568.0, + "11": 1183213568.0, + "12": 1183213568.0, + "13": 1183214592.0, + "14": 1183214592.0, + "15": 1183214592.0, + "16": 1183214592.0, + "17": 1183214592.0, + "18": 1183216128.0, + "19": 1183216128.0, + "20": 1183216128.0, + "21": 1183216128.0, + "22": 1183216128.0, + "23": 1183216128.0, + "24": 1183216128.0, + "25": 1183216128.0, + "26": 1183216128.0, + "27": 1183216128.0, + "28": 1183216128.0, + "29": 1183216128.0, + "30": 1183216128.0, + "31": 1183216128.0, + "32": 1183216128.0, + "33": 1183216128.0, + "34": 1183216128.0, + "35": 1183216128.0, + "36": 1183216128.0, + "37": 1183216128.0, + "38": 1183216128.0, + "39": 1183216128.0, + "40": 1183216128.0, + "41": 1183216128.0, + "42": 1183216128.0, + "43": 1183216128.0, + "44": 1183216128.0, + "45": 1183216128.0, + "46": 1183216128.0, + "47": 1183216128.0, + "48": 1183216128.0, + "49": 1183216128.0, + "50": 1183216128.0 + } + }, + "iteration-time": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": "nan", + "2": 5.60587, + "3": 1.78086, + "4": 1.81166, + "5": 1.46675, + "6": 1.97916, + "7": 1.41332, + "8": 1.88449, + "9": 2.22605, + "10": 1.8832, + "11": 1.66721, + "12": 1.6728, + "13": 1.65567, + "14": 1.93406, + "15": 1.64166, + "16": 1.63509, + "17": 2.1247, + "18": 1.66225, + "19": 1.81546, + "20": 1.69112, + "21": 2.07268, + "22": 1.58574, + "23": 1.88968, + "24": 1.53894, + "25": 1.99143, + "26": 2.082, + "27": 1.56796, + "28": 1.73914, + "29": 1.53288, + "30": 1.50939, + "31": 1.54291, + "32": 2.54507, + "33": 1.53583, + "34": 1.47496, + "35": 2.22401, + "36": 2.03324, + "37": 1.77357, + "38": 1.36324, + "39": 2.12685, + "40": 1.80811, + "41": 1.78468, + "42": 2.2478, + "43": 1.31016, + "44": 1.80069, + "45": 2.07249, + "46": 1.30933, + "47": 1.85259, + "48": 1.8144, + "49": 1.49511, + "50": 1.69966 + } + } +} \ No newline at end of file diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_1node/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_1node/model_config.yaml new file mode 100644 index 00000000000..7be86a80447 --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_1node/model_config.yaml @@ -0,0 +1,55 @@ +ENV_VARS: + CUDA_DEVICE_MAX_CONNECTIONS: 1 + NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0 +MODEL_ARGS: + --num-layers: 12 + --hidden-size: 512 + --num-attention-heads: 8 + --log-params-norm: true + --log-num-zeros-in-grad: true + --log-validation-ppl-to-tensorboard: true + --log-timers-to-tensorboard: true + --tensorboard-dir: ${TENSORBOARD_PATH} + --micro-batch-size: 4 + --global-batch-size: 32 + --seq-length: 1024 + --max-position-embeddings: 1024 + --train-iters: 50 + --timing-log-level: 0 + --lr-decay-iters: 320000 + --save: ${CHECKPOINT_SAVE_PATH} + --load: ${CHECKPOINT_LOAD_PATH} + --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document + --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json + --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt + --split: 949,50,1 + --distributed-backend: nccl + --lr: 0.00015 + --lr-decay-style: cosine + --min-lr: 1.0e-5 + --weight-decay: 1e-2 + --clip-grad: 1.0 + --lr-warmup-fraction: .01 + --log-interval: 1 + --save-interval: 10000 + --eval-interval: 1000 + --eval-iters: 10 + --transformer-impl: transformer_engine + --tensor-model-parallel-size: 2 + --pipeline-model-parallel-size: 1 + --context-parallel-size: 2 + --sequence-parallel: true + --hidden-dropout: 0.0 + --attention-dropout: 0.0 + --no-gradient-accumulation-fusion: true + --attention-softmax-in-fp32: true + --use-mcore-models: true + --ckpt-format: torch_dist + --dist-ckpt-strictness: log_all # backward compatibility for TE changes + --data-cache-path: ${DATA_CACHE_PATH} + --bf16: true + --attention-backend: flash + --log-memory-to-tensorboard: true + --async-save: true + --use-persistent-ckpt-worker: true +TEST_TYPE: regular diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_calculate_per_token_loss_1node/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_calculate_per_token_loss_1node/golden_values_dev_dgx_gb200.json new file mode 100644 index 00000000000..cc303f41c1b --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_calculate_per_token_loss_1node/golden_values_dev_dgx_gb200.json @@ -0,0 +1,287 @@ +{ + "lm loss": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 10.95727, + "2": 10.94891, + "3": 10.95181, + "4": 10.93701, + "5": 10.94341, + "6": 10.9381, + "7": 10.94906, + "8": 10.92913, + "9": 10.94291, + "10": 10.93268, + "11": 10.9326, + "12": 10.92193, + "13": 10.91841, + "14": 10.91226, + "15": 10.891, + "16": 10.87484, + "17": 10.88661, + "18": 10.86792, + "19": 10.87319, + "20": 10.77508, + "21": 10.76177, + "22": 10.75197, + "23": 10.73997, + "24": 10.70111, + "25": 10.71372, + "26": 10.68547, + "27": 10.64309, + "28": 10.56423, + "29": 10.53588, + "30": 10.52933, + "31": 10.51593, + "32": 10.49935, + "33": 10.46546, + "34": 10.42362, + "35": 10.42255, + "36": 10.41397, + "37": 10.36771, + "38": 10.37996, + "39": 10.34012, + "40": 10.32558, + "41": 10.30513, + "42": 10.29477, + "43": 10.25699, + "44": 10.23372, + "45": 10.2453, + "46": 10.21282, + "47": 10.19585, + "48": 10.15323, + "49": 10.14471, + "50": 10.15149 + } + }, + "num-zeros": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 652.0, + "2": 628.0, + "3": 651.0, + "4": 603.0, + "5": 648.0, + "6": 672.0, + "7": 646.0, + "8": 637.0, + "9": 647.0, + "10": 666.0, + "11": 630.0, + "12": 627.0, + "13": 648.0, + "14": 598.0, + "15": 603.0, + "16": 597.0, + "17": 647.0, + "18": 570.0, + "19": 581.0, + "20": 612.0, + "21": 628.0, + "22": 681.0, + "23": 689.0, + "24": 651.0, + "25": 653.0, + "26": 660.0, + "27": 668.0, + "28": 670.0, + "29": 787.0, + "30": 654.0, + "31": 739.0, + "32": 721.0, + "33": 751.0, + "34": 729.0, + "35": 794.0, + "36": 819.0, + "37": 818.0, + "38": 785.0, + "39": 803.0, + "40": 874.0, + "41": 860.0, + "42": 726.0, + "43": 861.0, + "44": 774.0, + "45": 953.0, + "46": 942.0, + "47": 867.0, + "48": 920.0, + "49": 986.0, + "50": 993.0 + } + }, + "mem-allocated-bytes": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 638631424.0, + "2": 638631424.0, + "3": 638631424.0, + "4": 638631424.0, + "5": 638631424.0, + "6": 638631424.0, + "7": 638631424.0, + "8": 638631424.0, + "9": 638631424.0, + "10": 638631424.0, + "11": 638631424.0, + "12": 638631424.0, + "13": 638631424.0, + "14": 638631424.0, + "15": 638631424.0, + "16": 638631424.0, + "17": 638631424.0, + "18": 638631424.0, + "19": 638631424.0, + "20": 638631424.0, + "21": 638631424.0, + "22": 638631424.0, + "23": 638631424.0, + "24": 638631424.0, + "25": 638631424.0, + "26": 638631424.0, + "27": 638631424.0, + "28": 638631424.0, + "29": 638631424.0, + "30": 638631424.0, + "31": 638631424.0, + "32": 638631424.0, + "33": 638631424.0, + "34": 638631424.0, + "35": 638631424.0, + "36": 638631424.0, + "37": 638631424.0, + "38": 638631424.0, + "39": 638631424.0, + "40": 638631424.0, + "41": 638631424.0, + "42": 638631424.0, + "43": 638631424.0, + "44": 638631424.0, + "45": 638631424.0, + "46": 638631424.0, + "47": 638631424.0, + "48": 638631424.0, + "49": 638631424.0, + "50": 638631424.0 + } + }, + "mem-max-allocated-bytes": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 910635008.0, + "2": 1172072960.0, + "3": 1172072960.0, + "4": 1173121024.0, + "5": 1173121024.0, + "6": 1173121024.0, + "7": 1173121024.0, + "8": 1173121024.0, + "9": 1173121024.0, + "10": 1173121024.0, + "11": 1173121024.0, + "12": 1173121024.0, + "13": 1173121024.0, + "14": 1175741952.0, + "15": 1179936256.0, + "16": 1179936256.0, + "17": 1179936256.0, + "18": 1179936256.0, + "19": 1179936256.0, + "20": 1179936256.0, + "21": 1179936256.0, + "22": 1179936256.0, + "23": 1179936256.0, + "24": 1179936256.0, + "25": 1179936256.0, + "26": 1179936256.0, + "27": 1179936256.0, + "28": 1179936256.0, + "29": 1180986880.0, + "30": 1180986880.0, + "31": 1180986880.0, + "32": 1180986880.0, + "33": 1180986880.0, + "34": 1180986880.0, + "35": 1180986880.0, + "36": 1180986880.0, + "37": 1180986880.0, + "38": 1180986880.0, + "39": 1180986880.0, + "40": 1180986880.0, + "41": 1180986880.0, + "42": 1180986880.0, + "43": 1180986880.0, + "44": 1180986880.0, + "45": 1180986880.0, + "46": 1180986880.0, + "47": 1180986880.0, + "48": 1180986880.0, + "49": 1180986880.0, + "50": 1180986880.0 + } + }, + "iteration-time": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": "nan", + "2": 5.89687, + "3": 1.8132, + "4": 1.45156, + "5": 1.52182, + "6": 1.66759, + "7": 1.1228, + "8": 1.74845, + "9": 1.98205, + "10": 1.86854, + "11": 1.67637, + "12": 1.42277, + "13": 1.47847, + "14": 1.68429, + "15": 1.39073, + "16": 1.45687, + "17": 1.83046, + "18": 1.51151, + "19": 1.44769, + "20": 1.48115, + "21": 1.63929, + "22": 1.40921, + "23": 1.5631, + "24": 1.57265, + "25": 2.00311, + "26": 2.0412, + "27": 1.54499, + "28": 1.50724, + "29": 1.45671, + "30": 1.4227, + "31": 1.44662, + "32": 2.37318, + "33": 1.24696, + "34": 1.33917, + "35": 1.89352, + "36": 1.97068, + "37": 1.78024, + "38": 1.50716, + "39": 2.17551, + "40": 1.79008, + "41": 1.89868, + "42": 2.30152, + "43": 1.54042, + "44": 1.71917, + "45": 2.16959, + "46": 1.59849, + "47": 1.72987, + "48": 1.91501, + "49": 1.79104, + "50": 1.6482 + } + } +} \ No newline at end of file diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_calculate_per_token_loss_1node/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_calculate_per_token_loss_1node/model_config.yaml new file mode 100644 index 00000000000..27fd45b5701 --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_calculate_per_token_loss_1node/model_config.yaml @@ -0,0 +1,55 @@ +ENV_VARS: + CUDA_DEVICE_MAX_CONNECTIONS: 1 + NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0 +MODEL_ARGS: + --num-layers: 12 + --hidden-size: 512 + --num-attention-heads: 8 + --log-params-norm: true + --log-num-zeros-in-grad: true + --log-validation-ppl-to-tensorboard: true + --log-timers-to-tensorboard: true + --tensorboard-dir: ${TENSORBOARD_PATH} + --micro-batch-size: 4 + --global-batch-size: 32 + --seq-length: 1024 + --max-position-embeddings: 1024 + --train-iters: 50 + --timing-log-level: 0 + --lr-decay-iters: 320000 + --save: ${CHECKPOINT_SAVE_PATH} + --load: ${CHECKPOINT_LOAD_PATH} + --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document + --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json + --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt + --split: 949,50,1 + --distributed-backend: nccl + --lr: 0.00015 + --lr-decay-style: cosine + --min-lr: 1.0e-5 + --weight-decay: 1e-2 + --clip-grad: 1.0 + --lr-warmup-fraction: .01 + --log-interval: 1 + --save-interval: 10000 + --eval-interval: 1000 + --eval-iters: 10 + --transformer-impl: transformer_engine + --tensor-model-parallel-size: 2 + --pipeline-model-parallel-size: 1 + --context-parallel-size: 2 + --sequence-parallel: true + --hidden-dropout: 0.0 + --attention-dropout: 0.0 + --no-gradient-accumulation-fusion: true + --attention-softmax-in-fp32: true + --use-mcore-models: true + --ckpt-format: torch_dist + --data-cache-path: ${DATA_CACHE_PATH} + --bf16: true + --attention-backend: flash + --log-memory-to-tensorboard: true + --calculate-per-token-loss: true + --async-save: true + --use-persistent-ckpt-worker: true +TEST_TYPE: regular diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_calculate_per_token_loss_nondeterministic_1node/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_calculate_per_token_loss_nondeterministic_1node/golden_values_dev_dgx_gb200.json new file mode 100644 index 00000000000..298678e05a1 --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_calculate_per_token_loss_nondeterministic_1node/golden_values_dev_dgx_gb200.json @@ -0,0 +1,287 @@ +{ + "lm loss": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 10.95727, + "2": 10.94891, + "3": 10.95179, + "4": 10.93702, + "5": 10.94339, + "6": 10.93813, + "7": 10.9491, + "8": 10.92914, + "9": 10.94288, + "10": 10.9327, + "11": 10.93261, + "12": 10.92195, + "13": 10.91843, + "14": 10.91224, + "15": 10.89099, + "16": 10.87486, + "17": 10.88663, + "18": 10.86792, + "19": 10.87316, + "20": 10.77511, + "21": 10.76175, + "22": 10.75195, + "23": 10.73997, + "24": 10.70102, + "25": 10.7137, + "26": 10.68548, + "27": 10.64306, + "28": 10.56429, + "29": 10.53587, + "30": 10.52935, + "31": 10.51591, + "32": 10.49933, + "33": 10.46546, + "34": 10.4236, + "35": 10.42255, + "36": 10.41393, + "37": 10.36773, + "38": 10.37995, + "39": 10.34016, + "40": 10.32559, + "41": 10.30514, + "42": 10.29481, + "43": 10.25698, + "44": 10.23375, + "45": 10.24531, + "46": 10.2128, + "47": 10.19582, + "48": 10.15323, + "49": 10.14467, + "50": 10.15151 + } + }, + "num-zeros": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 615.0, + "2": 629.0, + "3": 633.0, + "4": 653.0, + "5": 599.0, + "6": 679.0, + "7": 685.0, + "8": 620.0, + "9": 655.0, + "10": 652.0, + "11": 645.0, + "12": 599.0, + "13": 646.0, + "14": 622.0, + "15": 590.0, + "16": 625.0, + "17": 627.0, + "18": 621.0, + "19": 572.0, + "20": 631.0, + "21": 667.0, + "22": 660.0, + "23": 669.0, + "24": 658.0, + "25": 613.0, + "26": 701.0, + "27": 678.0, + "28": 682.0, + "29": 769.0, + "30": 719.0, + "31": 764.0, + "32": 757.0, + "33": 739.0, + "34": 807.0, + "35": 742.0, + "36": 791.0, + "37": 784.0, + "38": 794.0, + "39": 782.0, + "40": 843.0, + "41": 897.0, + "42": 716.0, + "43": 858.0, + "44": 841.0, + "45": 918.0, + "46": 896.0, + "47": 897.0, + "48": 942.0, + "49": 1019.0, + "50": 975.0 + } + }, + "mem-allocated-bytes": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 640073216.0, + "2": 640073216.0, + "3": 640073216.0, + "4": 640073216.0, + "5": 640073216.0, + "6": 640073216.0, + "7": 640073216.0, + "8": 640073216.0, + "9": 640073216.0, + "10": 640073216.0, + "11": 640073216.0, + "12": 640073216.0, + "13": 640073216.0, + "14": 640073216.0, + "15": 640073216.0, + "16": 640073216.0, + "17": 640073216.0, + "18": 640073216.0, + "19": 640073216.0, + "20": 640073216.0, + "21": 640073216.0, + "22": 640073216.0, + "23": 640073216.0, + "24": 640073216.0, + "25": 640073216.0, + "26": 640073216.0, + "27": 640073216.0, + "28": 640073216.0, + "29": 640073216.0, + "30": 640073216.0, + "31": 640073216.0, + "32": 640073216.0, + "33": 640073216.0, + "34": 640073216.0, + "35": 640073216.0, + "36": 640073216.0, + "37": 640073216.0, + "38": 640073216.0, + "39": 640073216.0, + "40": 640073216.0, + "41": 640073216.0, + "42": 640073216.0, + "43": 640073216.0, + "44": 640073216.0, + "45": 640073216.0, + "46": 640073216.0, + "47": 640073216.0, + "48": 640073216.0, + "49": 640073216.0, + "50": 640073216.0 + } + }, + "mem-max-allocated-bytes": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 910372864.0, + "2": 1180329472.0, + "3": 1183738880.0, + "4": 1183738880.0, + "5": 1183739904.0, + "6": 1183739904.0, + "7": 1183739904.0, + "8": 1183739904.0, + "9": 1183739904.0, + "10": 1183739904.0, + "11": 1183739904.0, + "12": 1183739904.0, + "13": 1183739904.0, + "14": 1183739904.0, + "15": 1183739904.0, + "16": 1183739904.0, + "17": 1184394752.0, + "18": 1184394752.0, + "19": 1184394752.0, + "20": 1184394752.0, + "21": 1184394752.0, + "22": 1184394752.0, + "23": 1184394752.0, + "24": 1184394752.0, + "25": 1184394752.0, + "26": 1184394752.0, + "27": 1184394752.0, + "28": 1184394752.0, + "29": 1184394752.0, + "30": 1184394752.0, + "31": 1184394752.0, + "32": 1184394752.0, + "33": 1184394752.0, + "34": 1184394752.0, + "35": 1184394752.0, + "36": 1184394752.0, + "37": 1184394752.0, + "38": 1184394752.0, + "39": 1184394752.0, + "40": 1184394752.0, + "41": 1184394752.0, + "42": 1184394752.0, + "43": 1184394752.0, + "44": 1184394752.0, + "45": 1184394752.0, + "46": 1184394752.0, + "47": 1184394752.0, + "48": 1184394752.0, + "49": 1184394752.0, + "50": 1184394752.0 + } + }, + "iteration-time": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": "nan", + "2": 5.8018, + "3": 1.99965, + "4": 1.92573, + "5": 1.54001, + "6": 2.19746, + "7": 1.43906, + "8": 2.04863, + "9": 2.00883, + "10": 2.07086, + "11": 2.13186, + "12": 1.7911, + "13": 1.79769, + "14": 1.91661, + "15": 1.63494, + "16": 1.674, + "17": 2.19385, + "18": 1.48561, + "19": 1.85783, + "20": 1.75399, + "21": 1.91777, + "22": 1.39804, + "23": 6.7165, + "24": 1.55767, + "25": 1.93668, + "26": 2.09832, + "27": 1.58232, + "28": 1.71268, + "29": 1.51705, + "30": 1.44289, + "31": 1.46202, + "32": 2.28951, + "33": 1.26999, + "34": 1.44668, + "35": 2.10374, + "36": 1.77327, + "37": 1.69113, + "38": 1.38242, + "39": 2.16164, + "40": 1.75766, + "41": 1.82132, + "42": 2.26711, + "43": 1.76921, + "44": 1.77682, + "45": 1.91594, + "46": 1.69269, + "47": 1.79872, + "48": 1.93603, + "49": 1.50668, + "50": 1.75984 + } + } +} \ No newline at end of file diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_calculate_per_token_loss_nondeterministic_1node/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_calculate_per_token_loss_nondeterministic_1node/model_config.yaml new file mode 100644 index 00000000000..27ff315894a --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_calculate_per_token_loss_nondeterministic_1node/model_config.yaml @@ -0,0 +1,55 @@ +ENV_VARS: + CUDA_DEVICE_MAX_CONNECTIONS: 1 + NVTE_ALLOW_NONDETERMINISTIC_ALGO: 1 +MODEL_ARGS: + --num-layers: 12 + --hidden-size: 512 + --num-attention-heads: 8 + --log-params-norm: true + --log-num-zeros-in-grad: true + --log-validation-ppl-to-tensorboard: true + --log-timers-to-tensorboard: true + --tensorboard-dir: ${TENSORBOARD_PATH} + --micro-batch-size: 4 + --global-batch-size: 32 + --seq-length: 1024 + --max-position-embeddings: 1024 + --train-iters: 50 + --timing-log-level: 0 + --lr-decay-iters: 320000 + --save: ${CHECKPOINT_SAVE_PATH} + --load: ${CHECKPOINT_LOAD_PATH} + --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document + --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json + --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt + --split: 949,50,1 + --distributed-backend: nccl + --lr: 0.00015 + --lr-decay-style: cosine + --min-lr: 1.0e-5 + --weight-decay: 1e-2 + --clip-grad: 1.0 + --lr-warmup-fraction: .01 + --log-interval: 1 + --save-interval: 10000 + --eval-interval: 1000 + --eval-iters: 10 + --transformer-impl: transformer_engine + --tensor-model-parallel-size: 2 + --pipeline-model-parallel-size: 1 + --context-parallel-size: 2 + --sequence-parallel: true + --hidden-dropout: 0.0 + --attention-dropout: 0.0 + --no-gradient-accumulation-fusion: true + --attention-softmax-in-fp32: true + --use-mcore-models: true + --ckpt-format: torch_dist + --data-cache-path: ${DATA_CACHE_PATH} + --bf16: true + --attention-backend: flash + --log-memory-to-tensorboard: true + --calculate-per-token-loss: true + --async-save: true + --use-persistent-ckpt-worker: true +TEST_TYPE: regular diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_etp4_calculate_per_token_loss_dp_last_1node/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_etp4_calculate_per_token_loss_dp_last_1node/golden_values_dev_dgx_gb200.json new file mode 100644 index 00000000000..d93eb6028ee --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_etp4_calculate_per_token_loss_dp_last_1node/golden_values_dev_dgx_gb200.json @@ -0,0 +1,287 @@ +{ + "lm loss": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 10.95727, + "2": 10.94891, + "3": 10.95181, + "4": 10.93701, + "5": 10.94341, + "6": 10.9381, + "7": 10.94906, + "8": 10.92913, + "9": 10.94291, + "10": 10.93268, + "11": 10.9326, + "12": 10.92193, + "13": 10.91841, + "14": 10.91226, + "15": 10.891, + "16": 10.87484, + "17": 10.88661, + "18": 10.86792, + "19": 10.87319, + "20": 10.77508, + "21": 10.76177, + "22": 10.75197, + "23": 10.73997, + "24": 10.70111, + "25": 10.71372, + "26": 10.68547, + "27": 10.64309, + "28": 10.56423, + "29": 10.53588, + "30": 10.52933, + "31": 10.51593, + "32": 10.49935, + "33": 10.46546, + "34": 10.42362, + "35": 10.42255, + "36": 10.41397, + "37": 10.36771, + "38": 10.37996, + "39": 10.34012, + "40": 10.32558, + "41": 10.30513, + "42": 10.29477, + "43": 10.25699, + "44": 10.23372, + "45": 10.2453, + "46": 10.21282, + "47": 10.19585, + "48": 10.15323, + "49": 10.14471, + "50": 10.15149 + } + }, + "num-zeros": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 652.0, + "2": 628.0, + "3": 651.0, + "4": 603.0, + "5": 648.0, + "6": 672.0, + "7": 646.0, + "8": 637.0, + "9": 647.0, + "10": 666.0, + "11": 630.0, + "12": 627.0, + "13": 648.0, + "14": 598.0, + "15": 603.0, + "16": 597.0, + "17": 647.0, + "18": 570.0, + "19": 581.0, + "20": 612.0, + "21": 628.0, + "22": 681.0, + "23": 689.0, + "24": 651.0, + "25": 653.0, + "26": 660.0, + "27": 668.0, + "28": 670.0, + "29": 787.0, + "30": 654.0, + "31": 739.0, + "32": 721.0, + "33": 751.0, + "34": 729.0, + "35": 794.0, + "36": 819.0, + "37": 818.0, + "38": 785.0, + "39": 803.0, + "40": 874.0, + "41": 860.0, + "42": 726.0, + "43": 861.0, + "44": 774.0, + "45": 953.0, + "46": 942.0, + "47": 867.0, + "48": 920.0, + "49": 986.0, + "50": 993.0 + } + }, + "mem-allocated-bytes": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 638631424.0, + "2": 638631424.0, + "3": 638631424.0, + "4": 638631424.0, + "5": 638631424.0, + "6": 638631424.0, + "7": 638631424.0, + "8": 638631424.0, + "9": 638631424.0, + "10": 638631424.0, + "11": 638631424.0, + "12": 638631424.0, + "13": 638631424.0, + "14": 638631424.0, + "15": 638631424.0, + "16": 638631424.0, + "17": 638631424.0, + "18": 638631424.0, + "19": 638631424.0, + "20": 638631424.0, + "21": 638631424.0, + "22": 638631424.0, + "23": 638631424.0, + "24": 638631424.0, + "25": 638631424.0, + "26": 638631424.0, + "27": 638631424.0, + "28": 638631424.0, + "29": 638631424.0, + "30": 638631424.0, + "31": 638631424.0, + "32": 638631424.0, + "33": 638631424.0, + "34": 638631424.0, + "35": 638631424.0, + "36": 638631424.0, + "37": 638631424.0, + "38": 638631424.0, + "39": 638631424.0, + "40": 638631424.0, + "41": 638631424.0, + "42": 638631424.0, + "43": 638631424.0, + "44": 638631424.0, + "45": 638631424.0, + "46": 638631424.0, + "47": 638631424.0, + "48": 638631424.0, + "49": 638631424.0, + "50": 638631424.0 + } + }, + "mem-max-allocated-bytes": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 910635520.0, + "2": 1169977344.0, + "3": 1169977344.0, + "4": 1169977344.0, + "5": 1169977344.0, + "6": 1169977344.0, + "7": 1175744512.0, + "8": 1175744512.0, + "9": 1175744512.0, + "10": 1175744512.0, + "11": 1175744512.0, + "12": 1175744512.0, + "13": 1175744512.0, + "14": 1175744512.0, + "15": 1175744512.0, + "16": 1175744512.0, + "17": 1175744512.0, + "18": 1175744512.0, + "19": 1175744512.0, + "20": 1175744512.0, + "21": 1175744512.0, + "22": 1175744512.0, + "23": 1175744512.0, + "24": 1175744512.0, + "25": 1175744512.0, + "26": 1175744512.0, + "27": 1175744512.0, + "28": 1175744512.0, + "29": 1175744512.0, + "30": 1175744512.0, + "31": 1175744512.0, + "32": 1175744512.0, + "33": 1175744512.0, + "34": 1175744512.0, + "35": 1175744512.0, + "36": 1175744512.0, + "37": 1175744512.0, + "38": 1175744512.0, + "39": 1175744512.0, + "40": 1175744512.0, + "41": 1175744512.0, + "42": 1175744512.0, + "43": 1175744512.0, + "44": 1175744512.0, + "45": 1175744512.0, + "46": 1175744512.0, + "47": 1175744512.0, + "48": 1175744512.0, + "49": 1175744512.0, + "50": 1175744512.0 + } + }, + "iteration-time": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": "nan", + "2": 6.41138, + "3": 1.5898, + "4": 1.41004, + "5": 1.12467, + "6": 1.38063, + "7": 1.22586, + "8": 1.69308, + "9": 2.15818, + "10": 1.86364, + "11": 1.92819, + "12": 1.49576, + "13": 1.87719, + "14": 1.88982, + "15": 1.53839, + "16": 1.72437, + "17": 2.1365, + "18": 1.57223, + "19": 1.60366, + "20": 1.70455, + "21": 2.01358, + "22": 1.53222, + "23": 1.95425, + "24": 1.58234, + "25": 2.02428, + "26": 2.20633, + "27": 1.57798, + "28": 1.79601, + "29": 1.55598, + "30": 1.59685, + "31": 1.38619, + "32": 2.50506, + "33": 1.37307, + "34": 1.37461, + "35": 1.97449, + "36": 1.9462, + "37": 1.63442, + "38": 1.28528, + "39": 1.80714, + "40": 1.49405, + "41": 1.58269, + "42": 2.2949, + "43": 1.22817, + "44": 1.65637, + "45": 1.90472, + "46": 1.38963, + "47": 1.56295, + "48": 1.96312, + "49": 1.53437, + "50": 1.78818 + } + } +} \ No newline at end of file diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_etp4_calculate_per_token_loss_dp_last_1node/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_etp4_calculate_per_token_loss_dp_last_1node/model_config.yaml new file mode 100644 index 00000000000..caf364efb28 --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_etp4_calculate_per_token_loss_dp_last_1node/model_config.yaml @@ -0,0 +1,58 @@ +ENV_VARS: + CUDA_DEVICE_MAX_CONNECTIONS: 1 + NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0 +MODEL_ARGS: + --num-layers: 12 + --hidden-size: 512 + --num-attention-heads: 8 + --log-params-norm: true + --log-num-zeros-in-grad: true + --log-validation-ppl-to-tensorboard: true + --log-timers-to-tensorboard: true + --tensorboard-dir: ${TENSORBOARD_PATH} + --micro-batch-size: 4 + --global-batch-size: 32 + --seq-length: 1024 + --max-position-embeddings: 1024 + --train-iters: 50 + --timing-log-level: 0 + --lr-decay-iters: 320000 + --save: ${CHECKPOINT_SAVE_PATH} + --load: ${CHECKPOINT_LOAD_PATH} + --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document + --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json + --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt + --split: 949,50,1 + --distributed-backend: nccl + --lr: 0.00015 + --lr-decay-style: cosine + --min-lr: 1.0e-5 + --weight-decay: 1e-2 + --clip-grad: 1.0 + --lr-warmup-fraction: .01 + --log-interval: 1 + --save-interval: 10000 + --eval-interval: 1000 + --eval-iters: 10 + --transformer-impl: transformer_engine + --tensor-model-parallel-size: 2 + --pipeline-model-parallel-size: 1 + --context-parallel-size: 2 + --sequence-parallel: true + --hidden-dropout: 0.0 + --attention-dropout: 0.0 + --no-gradient-accumulation-fusion: true + --attention-softmax-in-fp32: true + --use-mcore-models: true + --ckpt-format: torch_dist + --dist-ckpt-optim-fully-reshardable: true + --data-cache-path: ${DATA_CACHE_PATH} + --bf16: true + --attention-backend: flash + --log-memory-to-tensorboard: true + --use-tp-pp-dp-mapping: true + --expert-tensor-parallel-size: 4 + --calculate-per-token-loss: true + --async-save: true + --use-persistent-ckpt-worker: true +TEST_TYPE: regular diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_etp4_calculate_per_token_loss_nondeterministic_dp_last_1node/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_etp4_calculate_per_token_loss_nondeterministic_dp_last_1node/golden_values_dev_dgx_gb200.json new file mode 100644 index 00000000000..a981e98b820 --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_etp4_calculate_per_token_loss_nondeterministic_dp_last_1node/golden_values_dev_dgx_gb200.json @@ -0,0 +1,287 @@ +{ + "lm loss": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 10.95727, + "2": 10.94891, + "3": 10.95183, + "4": 10.93701, + "5": 10.94343, + "6": 10.93812, + "7": 10.94908, + "8": 10.92911, + "9": 10.94294, + "10": 10.9327, + "11": 10.93263, + "12": 10.92198, + "13": 10.91843, + "14": 10.91225, + "15": 10.89099, + "16": 10.87483, + "17": 10.88665, + "18": 10.86793, + "19": 10.87316, + "20": 10.77507, + "21": 10.76179, + "22": 10.75193, + "23": 10.73998, + "24": 10.7011, + "25": 10.71371, + "26": 10.68545, + "27": 10.64309, + "28": 10.56426, + "29": 10.53592, + "30": 10.52937, + "31": 10.51596, + "32": 10.49932, + "33": 10.46549, + "34": 10.42361, + "35": 10.42257, + "36": 10.41395, + "37": 10.36774, + "38": 10.37998, + "39": 10.34012, + "40": 10.32558, + "41": 10.30511, + "42": 10.29478, + "43": 10.25698, + "44": 10.23372, + "45": 10.24529, + "46": 10.21284, + "47": 10.19583, + "48": 10.15324, + "49": 10.14467, + "50": 10.15149 + } + }, + "num-zeros": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 640.0, + "2": 628.0, + "3": 656.0, + "4": 621.0, + "5": 636.0, + "6": 623.0, + "7": 696.0, + "8": 639.0, + "9": 676.0, + "10": 675.0, + "11": 666.0, + "12": 623.0, + "13": 605.0, + "14": 623.0, + "15": 619.0, + "16": 616.0, + "17": 627.0, + "18": 645.0, + "19": 647.0, + "20": 650.0, + "21": 668.0, + "22": 680.0, + "23": 637.0, + "24": 726.0, + "25": 654.0, + "26": 614.0, + "27": 715.0, + "28": 683.0, + "29": 749.0, + "30": 684.0, + "31": 765.0, + "32": 711.0, + "33": 752.0, + "34": 784.0, + "35": 814.0, + "36": 751.0, + "37": 832.0, + "38": 751.0, + "39": 802.0, + "40": 884.0, + "41": 862.0, + "42": 779.0, + "43": 904.0, + "44": 845.0, + "45": 1007.0, + "46": 888.0, + "47": 889.0, + "48": 917.0, + "49": 995.0, + "50": 927.0 + } + }, + "mem-allocated-bytes": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 640073216.0, + "2": 640073216.0, + "3": 640073216.0, + "4": 640073216.0, + "5": 640073216.0, + "6": 640073216.0, + "7": 640073216.0, + "8": 640073216.0, + "9": 640073216.0, + "10": 640073216.0, + "11": 640073216.0, + "12": 640073216.0, + "13": 640073216.0, + "14": 640073216.0, + "15": 640073216.0, + "16": 640073216.0, + "17": 640073216.0, + "18": 640073216.0, + "19": 640073216.0, + "20": 640073216.0, + "21": 640073216.0, + "22": 640073216.0, + "23": 640073216.0, + "24": 640073216.0, + "25": 640073216.0, + "26": 640073216.0, + "27": 640073216.0, + "28": 640073216.0, + "29": 640073216.0, + "30": 640073216.0, + "31": 640073216.0, + "32": 640073216.0, + "33": 640073216.0, + "34": 640073216.0, + "35": 640073216.0, + "36": 640073216.0, + "37": 640073216.0, + "38": 640073216.0, + "39": 640073216.0, + "40": 640073216.0, + "41": 640073216.0, + "42": 640073216.0, + "43": 640073216.0, + "44": 640073216.0, + "45": 640073216.0, + "46": 640073216.0, + "47": 640073216.0, + "48": 640073216.0, + "49": 640073216.0, + "50": 640073216.0 + } + }, + "mem-max-allocated-bytes": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 909324800.0, + "2": 1183478272.0, + "3": 1183739904.0, + "4": 1183739904.0, + "5": 1183739904.0, + "6": 1183739904.0, + "7": 1183739904.0, + "8": 1183739904.0, + "9": 1183739904.0, + "10": 1183739904.0, + "11": 1183739904.0, + "12": 1183739904.0, + "13": 1183739904.0, + "14": 1183739904.0, + "15": 1183739904.0, + "16": 1183739904.0, + "17": 1183740416.0, + "18": 1183740416.0, + "19": 1183740416.0, + "20": 1183740416.0, + "21": 1183740416.0, + "22": 1183740416.0, + "23": 1183740416.0, + "24": 1183740416.0, + "25": 1183740416.0, + "26": 1183740416.0, + "27": 1183740416.0, + "28": 1183740416.0, + "29": 1183740416.0, + "30": 1183740416.0, + "31": 1183740416.0, + "32": 1183740416.0, + "33": 1183740416.0, + "34": 1183740416.0, + "35": 1183740416.0, + "36": 1183740416.0, + "37": 1183740416.0, + "38": 1183740416.0, + "39": 1183740416.0, + "40": 1183740416.0, + "41": 1183740416.0, + "42": 1183740416.0, + "43": 1183740416.0, + "44": 1183740416.0, + "45": 1183740416.0, + "46": 1183740416.0, + "47": 1183740416.0, + "48": 1183740416.0, + "49": 1183740416.0, + "50": 1183740416.0 + } + }, + "iteration-time": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": "nan", + "2": 6.44656, + "3": 2.16908, + "4": 1.58097, + "5": 1.48576, + "6": 1.83861, + "7": 1.5433, + "8": 2.14623, + "9": 2.25759, + "10": 1.76365, + "11": 1.97749, + "12": 1.85147, + "13": 1.94881, + "14": 2.01241, + "15": 1.43002, + "16": 1.84223, + "17": 2.33806, + "18": 1.58684, + "19": 1.99328, + "20": 1.67722, + "21": 2.0037, + "22": 1.44884, + "23": 2.22322, + "24": 1.30707, + "25": 2.08299, + "26": 2.09183, + "27": 1.54922, + "28": 1.81834, + "29": 1.63723, + "30": 1.49456, + "31": 1.48932, + "32": 2.53742, + "33": 1.41869, + "34": 1.37959, + "35": 2.14227, + "36": 2.07134, + "37": 1.4489, + "38": 1.52831, + "39": 2.03558, + "40": 1.56484, + "41": 1.73136, + "42": 2.24828, + "43": 1.44329, + "44": 1.70175, + "45": 2.06244, + "46": 1.64855, + "47": 1.73936, + "48": 1.95382, + "49": 1.46305, + "50": 1.54965 + } + } +} \ No newline at end of file diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_etp4_calculate_per_token_loss_nondeterministic_dp_last_1node/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_etp4_calculate_per_token_loss_nondeterministic_dp_last_1node/model_config.yaml new file mode 100644 index 00000000000..c1087788693 --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_etp4_calculate_per_token_loss_nondeterministic_dp_last_1node/model_config.yaml @@ -0,0 +1,57 @@ +ENV_VARS: + CUDA_DEVICE_MAX_CONNECTIONS: 1 + NVTE_ALLOW_NONDETERMINISTIC_ALGO: 1 +MODEL_ARGS: + --num-layers: 12 + --hidden-size: 512 + --num-attention-heads: 8 + --log-params-norm: true + --log-num-zeros-in-grad: true + --log-validation-ppl-to-tensorboard: true + --log-timers-to-tensorboard: true + --tensorboard-dir: ${TENSORBOARD_PATH} + --micro-batch-size: 4 + --global-batch-size: 32 + --seq-length: 1024 + --max-position-embeddings: 1024 + --train-iters: 50 + --timing-log-level: 0 + --lr-decay-iters: 320000 + --save: ${CHECKPOINT_SAVE_PATH} + --load: ${CHECKPOINT_LOAD_PATH} + --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document + --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json + --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt + --split: 949,50,1 + --distributed-backend: nccl + --lr: 0.00015 + --lr-decay-style: cosine + --min-lr: 1.0e-5 + --weight-decay: 1e-2 + --clip-grad: 1.0 + --lr-warmup-fraction: .01 + --log-interval: 1 + --save-interval: 10000 + --eval-interval: 1000 + --eval-iters: 10 + --transformer-impl: transformer_engine + --tensor-model-parallel-size: 2 + --pipeline-model-parallel-size: 1 + --context-parallel-size: 2 + --sequence-parallel: true + --hidden-dropout: 0.0 + --attention-dropout: 0.0 + --no-gradient-accumulation-fusion: true + --attention-softmax-in-fp32: true + --use-mcore-models: true + --ckpt-format: torch_dist + --data-cache-path: ${DATA_CACHE_PATH} + --bf16: true + --attention-backend: flash + --log-memory-to-tensorboard: true + --use-tp-pp-dp-mapping: true + --expert-tensor-parallel-size: 4 + --calculate-per-token-loss: true + --async-save: true + --use-persistent-ckpt-worker: true +TEST_TYPE: regular diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_etp4_dp_last_1node/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_etp4_dp_last_1node/golden_values_dev_dgx_gb200.json new file mode 100644 index 00000000000..d47c2287f05 --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_etp4_dp_last_1node/golden_values_dev_dgx_gb200.json @@ -0,0 +1,287 @@ +{ + "lm loss": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 10.95727, + "2": 10.94891, + "3": 10.95181, + "4": 10.93701, + "5": 10.94341, + "6": 10.9381, + "7": 10.94906, + "8": 10.92913, + "9": 10.94291, + "10": 10.93268, + "11": 10.9326, + "12": 10.92193, + "13": 10.91841, + "14": 10.91226, + "15": 10.891, + "16": 10.87484, + "17": 10.88661, + "18": 10.86792, + "19": 10.87319, + "20": 10.77508, + "21": 10.76177, + "22": 10.75197, + "23": 10.73997, + "24": 10.70111, + "25": 10.71372, + "26": 10.68547, + "27": 10.64309, + "28": 10.56423, + "29": 10.53588, + "30": 10.52933, + "31": 10.51593, + "32": 10.49935, + "33": 10.46546, + "34": 10.42362, + "35": 10.42255, + "36": 10.41397, + "37": 10.36771, + "38": 10.37996, + "39": 10.34012, + "40": 10.32558, + "41": 10.30513, + "42": 10.29477, + "43": 10.25699, + "44": 10.23372, + "45": 10.2453, + "46": 10.21282, + "47": 10.19585, + "48": 10.15323, + "49": 10.14471, + "50": 10.15149 + } + }, + "num-zeros": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 652.0, + "2": 628.0, + "3": 651.0, + "4": 603.0, + "5": 648.0, + "6": 672.0, + "7": 646.0, + "8": 637.0, + "9": 647.0, + "10": 666.0, + "11": 630.0, + "12": 627.0, + "13": 648.0, + "14": 598.0, + "15": 603.0, + "16": 597.0, + "17": 647.0, + "18": 570.0, + "19": 581.0, + "20": 612.0, + "21": 628.0, + "22": 681.0, + "23": 689.0, + "24": 651.0, + "25": 653.0, + "26": 660.0, + "27": 668.0, + "28": 670.0, + "29": 787.0, + "30": 654.0, + "31": 739.0, + "32": 721.0, + "33": 751.0, + "34": 729.0, + "35": 794.0, + "36": 819.0, + "37": 818.0, + "38": 785.0, + "39": 803.0, + "40": 874.0, + "41": 860.0, + "42": 726.0, + "43": 861.0, + "44": 774.0, + "45": 953.0, + "46": 942.0, + "47": 867.0, + "48": 920.0, + "49": 986.0, + "50": 993.0 + } + }, + "mem-allocated-bytes": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 638631424.0, + "2": 638631424.0, + "3": 638631424.0, + "4": 638631424.0, + "5": 638631424.0, + "6": 638631424.0, + "7": 638631424.0, + "8": 638631424.0, + "9": 638631424.0, + "10": 638631424.0, + "11": 638631424.0, + "12": 638631424.0, + "13": 638631424.0, + "14": 638631424.0, + "15": 638631424.0, + "16": 638631424.0, + "17": 638631424.0, + "18": 638631424.0, + "19": 638631424.0, + "20": 638631424.0, + "21": 638631424.0, + "22": 638631424.0, + "23": 638631424.0, + "24": 638631424.0, + "25": 638631424.0, + "26": 638631424.0, + "27": 638631424.0, + "28": 638631424.0, + "29": 638631424.0, + "30": 638631424.0, + "31": 638631424.0, + "32": 638631424.0, + "33": 638631424.0, + "34": 638631424.0, + "35": 638631424.0, + "36": 638631424.0, + "37": 638631424.0, + "38": 638631424.0, + "39": 638631424.0, + "40": 638631424.0, + "41": 638631424.0, + "42": 638631424.0, + "43": 638631424.0, + "44": 638631424.0, + "45": 638631424.0, + "46": 638631424.0, + "47": 638631424.0, + "48": 638631424.0, + "49": 638631424.0, + "50": 638631424.0 + } + }, + "mem-max-allocated-bytes": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 911681024.0, + "2": 1171025408.0, + "3": 1171025408.0, + "4": 1171025408.0, + "5": 1171547648.0, + "6": 1171547648.0, + "7": 1171547648.0, + "8": 1172072960.0, + "9": 1172072960.0, + "10": 1172072960.0, + "11": 1172072960.0, + "12": 1172072960.0, + "13": 1172072960.0, + "14": 1172072960.0, + "15": 1178366464.0, + "16": 1178366464.0, + "17": 1178366464.0, + "18": 1178366464.0, + "19": 1178366464.0, + "20": 1178366464.0, + "21": 1178366464.0, + "22": 1178366464.0, + "23": 1178366464.0, + "24": 1178366464.0, + "25": 1178366464.0, + "26": 1178366464.0, + "27": 1178366464.0, + "28": 1179414528.0, + "29": 1179414528.0, + "30": 1179414528.0, + "31": 1179414528.0, + "32": 1179414528.0, + "33": 1179414528.0, + "34": 1179414528.0, + "35": 1179414528.0, + "36": 1179414528.0, + "37": 1179414528.0, + "38": 1180985856.0, + "39": 1180985856.0, + "40": 1180985856.0, + "41": 1180985856.0, + "42": 1180985856.0, + "43": 1180985856.0, + "44": 1180985856.0, + "45": 1180985856.0, + "46": 1180987904.0, + "47": 1180987904.0, + "48": 1180987904.0, + "49": 1180987904.0, + "50": 1180987904.0 + } + }, + "iteration-time": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": "nan", + "2": 5.85273, + "3": 1.754, + "4": 1.524, + "5": 1.17706, + "6": 1.77467, + "7": 1.15062, + "8": 1.73608, + "9": 2.05454, + "10": 1.69295, + "11": 1.73324, + "12": 1.72982, + "13": 1.41918, + "14": 1.5191, + "15": 1.52286, + "16": 1.48935, + "17": 1.75452, + "18": 1.18806, + "19": 1.33464, + "20": 1.46888, + "21": 1.75231, + "22": 1.26387, + "23": 1.62263, + "24": 1.25956, + "25": 1.61872, + "26": 1.57574, + "27": 1.43191, + "28": 1.41469, + "29": 1.10952, + "30": 1.24628, + "31": 1.49183, + "32": 1.86176, + "33": 1.11786, + "34": 1.20029, + "35": 1.43349, + "36": 1.44269, + "37": 1.37221, + "38": 1.763, + "39": 1.62956, + "40": 1.56541, + "41": 1.7466, + "42": 1.90789, + "43": 1.40692, + "44": 1.56585, + "45": 1.55207, + "46": 1.40605, + "47": 1.44817, + "48": 1.59176, + "49": 1.14705, + "50": 1.69603 + } + } +} \ No newline at end of file diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_etp4_dp_last_1node/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_etp4_dp_last_1node/model_config.yaml new file mode 100644 index 00000000000..67831941bf5 --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_etp4_dp_last_1node/model_config.yaml @@ -0,0 +1,57 @@ +ENV_VARS: + CUDA_DEVICE_MAX_CONNECTIONS: 1 + NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0 +MODEL_ARGS: + --num-layers: 12 + --hidden-size: 512 + --num-attention-heads: 8 + --log-params-norm: true + --log-num-zeros-in-grad: true + --log-validation-ppl-to-tensorboard: true + --log-timers-to-tensorboard: true + --tensorboard-dir: ${TENSORBOARD_PATH} + --micro-batch-size: 4 + --global-batch-size: 32 + --seq-length: 1024 + --max-position-embeddings: 1024 + --train-iters: 50 + --timing-log-level: 0 + --lr-decay-iters: 320000 + --save: ${CHECKPOINT_SAVE_PATH} + --load: ${CHECKPOINT_LOAD_PATH} + --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document + --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json + --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt + --split: 949,50,1 + --distributed-backend: nccl + --lr: 0.00015 + --lr-decay-style: cosine + --min-lr: 1.0e-5 + --weight-decay: 1e-2 + --clip-grad: 1.0 + --lr-warmup-fraction: .01 + --log-interval: 1 + --save-interval: 10000 + --eval-interval: 1000 + --eval-iters: 10 + --transformer-impl: transformer_engine + --tensor-model-parallel-size: 2 + --pipeline-model-parallel-size: 1 + --context-parallel-size: 2 + --sequence-parallel: true + --hidden-dropout: 0.0 + --attention-dropout: 0.0 + --no-gradient-accumulation-fusion: true + --attention-softmax-in-fp32: true + --use-mcore-models: true + --ckpt-format: torch_dist + --dist-ckpt-strictness: log_all # backward compatibility for TE changes + --data-cache-path: ${DATA_CACHE_PATH} + --bf16: true + --attention-backend: flash + --log-memory-to-tensorboard: true + --use-tp-pp-dp-mapping: true + --expert-tensor-parallel-size: 4 + --async-save: true + --use-persistent-ckpt-worker: true +TEST_TYPE: regular diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_etp4_nondeterministic_dp_last_1node/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_etp4_nondeterministic_dp_last_1node/golden_values_dev_dgx_gb200.json new file mode 100644 index 00000000000..db21b12ea1a --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_etp4_nondeterministic_dp_last_1node/golden_values_dev_dgx_gb200.json @@ -0,0 +1,287 @@ +{ + "lm loss": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 10.95727, + "2": 10.94891, + "3": 10.95177, + "4": 10.93704, + "5": 10.94344, + "6": 10.93813, + "7": 10.94906, + "8": 10.92916, + "9": 10.94292, + "10": 10.93268, + "11": 10.93263, + "12": 10.92191, + "13": 10.91842, + "14": 10.91226, + "15": 10.89099, + "16": 10.87486, + "17": 10.88661, + "18": 10.86794, + "19": 10.87322, + "20": 10.77507, + "21": 10.76177, + "22": 10.75195, + "23": 10.73999, + "24": 10.7011, + "25": 10.71372, + "26": 10.68547, + "27": 10.64305, + "28": 10.56426, + "29": 10.53587, + "30": 10.52933, + "31": 10.51592, + "32": 10.49933, + "33": 10.46544, + "34": 10.42365, + "35": 10.42255, + "36": 10.4139, + "37": 10.36771, + "38": 10.37996, + "39": 10.34013, + "40": 10.32559, + "41": 10.30514, + "42": 10.29478, + "43": 10.25701, + "44": 10.23374, + "45": 10.24529, + "46": 10.2128, + "47": 10.19583, + "48": 10.15323, + "49": 10.14471, + "50": 10.15149 + } + }, + "num-zeros": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 638.0, + "2": 625.0, + "3": 654.0, + "4": 638.0, + "5": 647.0, + "6": 650.0, + "7": 664.0, + "8": 617.0, + "9": 699.0, + "10": 654.0, + "11": 658.0, + "12": 634.0, + "13": 658.0, + "14": 610.0, + "15": 635.0, + "16": 642.0, + "17": 639.0, + "18": 658.0, + "19": 612.0, + "20": 642.0, + "21": 631.0, + "22": 631.0, + "23": 681.0, + "24": 668.0, + "25": 628.0, + "26": 679.0, + "27": 728.0, + "28": 650.0, + "29": 779.0, + "30": 658.0, + "31": 727.0, + "32": 745.0, + "33": 736.0, + "34": 787.0, + "35": 764.0, + "36": 736.0, + "37": 812.0, + "38": 783.0, + "39": 807.0, + "40": 917.0, + "41": 831.0, + "42": 789.0, + "43": 901.0, + "44": 807.0, + "45": 911.0, + "46": 935.0, + "47": 925.0, + "48": 933.0, + "49": 965.0, + "50": 941.0 + } + }, + "mem-allocated-bytes": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 640073216.0, + "2": 640073216.0, + "3": 640073216.0, + "4": 640073216.0, + "5": 640073216.0, + "6": 640073216.0, + "7": 640073216.0, + "8": 640073216.0, + "9": 640073216.0, + "10": 640073216.0, + "11": 640073216.0, + "12": 640073216.0, + "13": 640073216.0, + "14": 640073216.0, + "15": 640073216.0, + "16": 640073216.0, + "17": 640073216.0, + "18": 640073216.0, + "19": 640073216.0, + "20": 640073216.0, + "21": 640073216.0, + "22": 640073216.0, + "23": 640073216.0, + "24": 640073216.0, + "25": 640073216.0, + "26": 640073216.0, + "27": 640073216.0, + "28": 640073216.0, + "29": 640073216.0, + "30": 640073216.0, + "31": 640073216.0, + "32": 640073216.0, + "33": 640073216.0, + "34": 640073216.0, + "35": 640073216.0, + "36": 640073216.0, + "37": 640073216.0, + "38": 640073216.0, + "39": 640073216.0, + "40": 640073216.0, + "41": 640073216.0, + "42": 640073216.0, + "43": 640073216.0, + "44": 640073216.0, + "45": 640073216.0, + "46": 640073216.0, + "47": 640073216.0, + "48": 640073216.0, + "49": 640073216.0, + "50": 640073216.0 + } + }, + "mem-max-allocated-bytes": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 912991744.0, + "2": 1176137216.0, + "3": 1179675136.0, + "4": 1182426624.0, + "5": 1182429184.0, + "6": 1182429184.0, + "7": 1182429184.0, + "8": 1182429184.0, + "9": 1183213056.0, + "10": 1183213056.0, + "11": 1183213056.0, + "12": 1183213056.0, + "13": 1183215616.0, + "14": 1183215616.0, + "15": 1183215616.0, + "16": 1183216128.0, + "17": 1183216128.0, + "18": 1183216128.0, + "19": 1183216128.0, + "20": 1183216128.0, + "21": 1183216128.0, + "22": 1183216128.0, + "23": 1183216128.0, + "24": 1183216128.0, + "25": 1183216128.0, + "26": 1183216128.0, + "27": 1183216128.0, + "28": 1183216128.0, + "29": 1183216128.0, + "30": 1183216128.0, + "31": 1183216128.0, + "32": 1183216128.0, + "33": 1183216128.0, + "34": 1183216128.0, + "35": 1183216128.0, + "36": 1183216128.0, + "37": 1183216128.0, + "38": 1183216128.0, + "39": 1183216128.0, + "40": 1183216128.0, + "41": 1183216128.0, + "42": 1183216128.0, + "43": 1183216128.0, + "44": 1183216128.0, + "45": 1183216128.0, + "46": 1183216128.0, + "47": 1183216128.0, + "48": 1183216128.0, + "49": 1183216128.0, + "50": 1183216128.0 + } + }, + "iteration-time": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": "nan", + "2": 6.49618, + "3": 2.09818, + "4": 1.72325, + "5": 1.42862, + "6": 2.28034, + "7": 1.50726, + "8": 2.23266, + "9": 2.03436, + "10": 1.82099, + "11": 2.14935, + "12": 1.56478, + "13": 1.799, + "14": 1.92293, + "15": 1.45784, + "16": 1.49109, + "17": 1.72284, + "18": 1.58572, + "19": 1.7527, + "20": 1.70321, + "21": 1.80934, + "22": 1.38826, + "23": 1.78832, + "24": 1.57859, + "25": 2.33851, + "26": 2.50204, + "27": 1.73973, + "28": 1.78459, + "29": 1.4745, + "30": 1.6494, + "31": 1.58083, + "32": 2.53005, + "33": 1.76136, + "34": 1.59369, + "35": 2.44097, + "36": 2.08472, + "37": 1.86224, + "38": 1.69656, + "39": 2.44513, + "40": 1.66926, + "41": 1.81833, + "42": 2.29526, + "43": 1.64123, + "44": 1.88236, + "45": 2.1215, + "46": 1.70564, + "47": 1.95959, + "48": 2.17988, + "49": 1.62037, + "50": 1.88092 + } + } +} \ No newline at end of file diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_etp4_nondeterministic_dp_last_1node/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_etp4_nondeterministic_dp_last_1node/model_config.yaml new file mode 100644 index 00000000000..72faa6ac2da --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_etp4_nondeterministic_dp_last_1node/model_config.yaml @@ -0,0 +1,57 @@ +ENV_VARS: + CUDA_DEVICE_MAX_CONNECTIONS: 1 + NVTE_ALLOW_NONDETERMINISTIC_ALGO: 1 +MODEL_ARGS: + --num-layers: 12 + --hidden-size: 512 + --num-attention-heads: 8 + --log-params-norm: true + --log-num-zeros-in-grad: true + --log-validation-ppl-to-tensorboard: true + --log-timers-to-tensorboard: true + --tensorboard-dir: ${TENSORBOARD_PATH} + --micro-batch-size: 4 + --global-batch-size: 32 + --seq-length: 1024 + --max-position-embeddings: 1024 + --train-iters: 50 + --timing-log-level: 0 + --lr-decay-iters: 320000 + --save: ${CHECKPOINT_SAVE_PATH} + --load: ${CHECKPOINT_LOAD_PATH} + --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document + --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json + --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt + --split: 949,50,1 + --distributed-backend: nccl + --lr: 0.00015 + --lr-decay-style: cosine + --min-lr: 1.0e-5 + --weight-decay: 1e-2 + --clip-grad: 1.0 + --lr-warmup-fraction: .01 + --log-interval: 1 + --save-interval: 10000 + --eval-interval: 1000 + --eval-iters: 10 + --transformer-impl: transformer_engine + --tensor-model-parallel-size: 2 + --pipeline-model-parallel-size: 1 + --context-parallel-size: 2 + --sequence-parallel: true + --hidden-dropout: 0.0 + --attention-dropout: 0.0 + --no-gradient-accumulation-fusion: true + --attention-softmax-in-fp32: true + --use-mcore-models: true + --ckpt-format: torch_dist + --dist-ckpt-strictness: log_all # backward compatibility for TE changes + --data-cache-path: ${DATA_CACHE_PATH} + --bf16: true + --attention-backend: flash + --log-memory-to-tensorboard: true + --use-tp-pp-dp-mapping: true + --expert-tensor-parallel-size: 4 + --async-save: true + --use-persistent-ckpt-worker: true +TEST_TYPE: regular diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_nondeterministic_1node/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_nondeterministic_1node/golden_values_dev_dgx_gb200.json new file mode 100644 index 00000000000..8e1404f1d9e --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_nondeterministic_1node/golden_values_dev_dgx_gb200.json @@ -0,0 +1,287 @@ +{ + "lm loss": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 10.95727, + "2": 10.94891, + "3": 10.95182, + "4": 10.93703, + "5": 10.9434, + "6": 10.93812, + "7": 10.94907, + "8": 10.92916, + "9": 10.94293, + "10": 10.93267, + "11": 10.93258, + "12": 10.92195, + "13": 10.9184, + "14": 10.91222, + "15": 10.89099, + "16": 10.87484, + "17": 10.88662, + "18": 10.8679, + "19": 10.87319, + "20": 10.7751, + "21": 10.7618, + "22": 10.75194, + "23": 10.73997, + "24": 10.70111, + "25": 10.71372, + "26": 10.68547, + "27": 10.64307, + "28": 10.56425, + "29": 10.53591, + "30": 10.52936, + "31": 10.51594, + "32": 10.49931, + "33": 10.46545, + "34": 10.42362, + "35": 10.42256, + "36": 10.41392, + "37": 10.36772, + "38": 10.37998, + "39": 10.34011, + "40": 10.3256, + "41": 10.30511, + "42": 10.2948, + "43": 10.25695, + "44": 10.23373, + "45": 10.2453, + "46": 10.21281, + "47": 10.19583, + "48": 10.15323, + "49": 10.14469, + "50": 10.15149 + } + }, + "num-zeros": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 662.0, + "2": 661.0, + "3": 638.0, + "4": 615.0, + "5": 643.0, + "6": 643.0, + "7": 678.0, + "8": 638.0, + "9": 672.0, + "10": 656.0, + "11": 606.0, + "12": 628.0, + "13": 624.0, + "14": 611.0, + "15": 648.0, + "16": 588.0, + "17": 629.0, + "18": 633.0, + "19": 623.0, + "20": 610.0, + "21": 674.0, + "22": 656.0, + "23": 660.0, + "24": 645.0, + "25": 648.0, + "26": 656.0, + "27": 663.0, + "28": 675.0, + "29": 781.0, + "30": 696.0, + "31": 771.0, + "32": 748.0, + "33": 781.0, + "34": 762.0, + "35": 757.0, + "36": 744.0, + "37": 896.0, + "38": 761.0, + "39": 796.0, + "40": 838.0, + "41": 852.0, + "42": 721.0, + "43": 891.0, + "44": 849.0, + "45": 924.0, + "46": 945.0, + "47": 913.0, + "48": 975.0, + "49": 1046.0, + "50": 1002.0 + } + }, + "mem-allocated-bytes": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 638631424.0, + "2": 638631424.0, + "3": 638631424.0, + "4": 638631424.0, + "5": 638631424.0, + "6": 638631424.0, + "7": 638631424.0, + "8": 638631424.0, + "9": 638631424.0, + "10": 638631424.0, + "11": 638631424.0, + "12": 638631424.0, + "13": 638631424.0, + "14": 638631424.0, + "15": 638631424.0, + "16": 638631424.0, + "17": 638631424.0, + "18": 638631424.0, + "19": 638631424.0, + "20": 638631424.0, + "21": 638631424.0, + "22": 638631424.0, + "23": 638631424.0, + "24": 638631424.0, + "25": 638631424.0, + "26": 638631424.0, + "27": 638631424.0, + "28": 638631424.0, + "29": 638631424.0, + "30": 638631424.0, + "31": 638631424.0, + "32": 638631424.0, + "33": 638631424.0, + "34": 638631424.0, + "35": 638631424.0, + "36": 638631424.0, + "37": 638631424.0, + "38": 638631424.0, + "39": 638631424.0, + "40": 638631424.0, + "41": 638631424.0, + "42": 638631424.0, + "43": 638631424.0, + "44": 638631424.0, + "45": 638631424.0, + "46": 638631424.0, + "47": 638631424.0, + "48": 638631424.0, + "49": 638631424.0, + "50": 638631424.0 + } + }, + "mem-max-allocated-bytes": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 910635520.0, + "2": 1171025920.0, + "3": 1171025920.0, + "4": 1171550720.0, + "5": 1171550720.0, + "6": 1171550720.0, + "7": 1171550720.0, + "8": 1171550720.0, + "9": 1171550720.0, + "10": 1171550720.0, + "11": 1171550720.0, + "12": 1171550720.0, + "13": 1171550720.0, + "14": 1171550720.0, + "15": 1171550720.0, + "16": 1171550720.0, + "17": 1180985856.0, + "18": 1180985856.0, + "19": 1180985856.0, + "20": 1180985856.0, + "21": 1180985856.0, + "22": 1180985856.0, + "23": 1180985856.0, + "24": 1180985856.0, + "25": 1180985856.0, + "26": 1180985856.0, + "27": 1180985856.0, + "28": 1180985856.0, + "29": 1180985856.0, + "30": 1180985856.0, + "31": 1180985856.0, + "32": 1180985856.0, + "33": 1180985856.0, + "34": 1180985856.0, + "35": 1180985856.0, + "36": 1180985856.0, + "37": 1180985856.0, + "38": 1180985856.0, + "39": 1180985856.0, + "40": 1180985856.0, + "41": 1180985856.0, + "42": 1180985856.0, + "43": 1180985856.0, + "44": 1180985856.0, + "45": 1180985856.0, + "46": 1180985856.0, + "47": 1180985856.0, + "48": 1180985856.0, + "49": 1180985856.0, + "50": 1180985856.0 + } + }, + "iteration-time": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": "nan", + "2": 5.47288, + "3": 1.93654, + "4": 1.9264, + "5": 1.82496, + "6": 1.93694, + "7": 1.50761, + "8": 2.03593, + "9": 2.25272, + "10": 2.04371, + "11": 2.13532, + "12": 1.73101, + "13": 1.79569, + "14": 2.16897, + "15": 1.74896, + "16": 1.66587, + "17": 2.27932, + "18": 1.8672, + "19": 2.13213, + "20": 1.75914, + "21": 2.22765, + "22": 1.72196, + "23": 2.24042, + "24": 1.42527, + "25": 2.31659, + "26": 2.10826, + "27": 1.94617, + "28": 1.85056, + "29": 1.8432, + "30": 1.68121, + "31": 1.38065, + "32": 2.81083, + "33": 1.89613, + "34": 1.62574, + "35": 2.21002, + "36": 1.86434, + "37": 1.75905, + "38": 1.44844, + "39": 2.17781, + "40": 1.7533, + "41": 1.88064, + "42": 2.34525, + "43": 1.44179, + "44": 2.00883, + "45": 2.31072, + "46": 1.77297, + "47": 1.8201, + "48": 1.796, + "49": 1.55485, + "50": 1.69233 + } + } +} \ No newline at end of file diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_nondeterministic_1node/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_nondeterministic_1node/model_config.yaml new file mode 100644 index 00000000000..2f33dea359d --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cp2_nondeterministic_1node/model_config.yaml @@ -0,0 +1,56 @@ +ENV_VARS: + CUDA_DEVICE_MAX_CONNECTIONS: 1 + NVTE_ALLOW_NONDETERMINISTIC_ALGO: 1 +MODEL_ARGS: + --num-layers: 12 + --hidden-size: 512 + --num-attention-heads: 8 + --log-params-norm: true + --log-num-zeros-in-grad: true + --log-validation-ppl-to-tensorboard: true + --log-timers-to-tensorboard: true + --tensorboard-dir: ${TENSORBOARD_PATH} + --micro-batch-size: 4 + --global-batch-size: 32 + --seq-length: 1024 + --max-position-embeddings: 1024 + --train-iters: 50 + --timing-log-level: 0 + --lr-decay-iters: 320000 + --save: ${CHECKPOINT_SAVE_PATH} + --load: ${CHECKPOINT_LOAD_PATH} + --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document + --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json + --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt + --split: 949,50,1 + --distributed-backend: nccl + --lr: 0.00015 + --lr-decay-style: cosine + --min-lr: 1.0e-5 + --weight-decay: 1e-2 + --clip-grad: 1.0 + --lr-warmup-fraction: .01 + --log-interval: 1 + --save-interval: 10000 + --eval-interval: 1000 + --eval-iters: 10 + --transformer-impl: transformer_engine + --tensor-model-parallel-size: 2 + --pipeline-model-parallel-size: 1 + --context-parallel-size: 2 + --sequence-parallel: true + --hidden-dropout: 0.0 + --attention-dropout: 0.0 + --no-gradient-accumulation-fusion: true + --attention-softmax-in-fp32: true + --use-mcore-models: true + --ckpt-format: torch_dist + --dist-ckpt-optim-fully-reshardable: true + --dist-ckpt-strictness: log_all # backward compatibility for TE changes + --data-cache-path: ${DATA_CACHE_PATH} + --bf16: true + --attention-backend: flash + --log-memory-to-tensorboard: true + --async-save: true + --use-persistent-ckpt-worker: true +TEST_TYPE: regular diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cross_entropy_loss_fusion_1node/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cross_entropy_loss_fusion_1node/golden_values_dev_dgx_gb200.json new file mode 100644 index 00000000000..d5f4d1bf850 --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cross_entropy_loss_fusion_1node/golden_values_dev_dgx_gb200.json @@ -0,0 +1,287 @@ +{ + "lm loss": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 10.93259, + "2": 10.92912, + "3": 10.92985, + "4": 10.92992, + "5": 10.93096, + "6": 10.92837, + "7": 10.9248, + "8": 10.9227, + "9": 10.92097, + "10": 10.92135, + "11": 10.91167, + "12": 10.9192, + "13": 10.91617, + "14": 10.90004, + "15": 10.88222, + "16": 10.86996, + "17": 10.8703, + "18": 10.86541, + "19": 10.86384, + "20": 10.77636, + "21": 10.75923, + "22": 10.75923, + "23": 10.74899, + "24": 10.7154, + "25": 10.71529, + "26": 10.70434, + "27": 10.65689, + "28": 10.58671, + "29": 10.56058, + "30": 10.5533, + "31": 10.52935, + "32": 10.52188, + "33": 10.48602, + "34": 10.4552, + "35": 10.45394, + "36": 10.4396, + "37": 10.39897, + "38": 10.39948, + "39": 10.36401, + "40": 10.36152, + "41": 10.33164, + "42": 10.31047, + "43": 10.28597, + "44": 10.2596, + "45": 10.27365, + "46": 10.2373, + "47": 10.22227, + "48": 10.17321, + "49": 10.17862, + "50": 10.18087 + } + }, + "num-zeros": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 1762.0, + "2": 1670.0, + "3": 1782.0, + "4": 1730.0, + "5": 1670.0, + "6": 1619.0, + "7": 1871.0, + "8": 1620.0, + "9": 1808.0, + "10": 1719.0, + "11": 1705.0, + "12": 1588.0, + "13": 1739.0, + "14": 1768.0, + "15": 1580.0, + "16": 1689.0, + "17": 1734.0, + "18": 1726.0, + "19": 1682.0, + "20": 1673.0, + "21": 1613.0, + "22": 1703.0, + "23": 1640.0, + "24": 1718.0, + "25": 1589.0, + "26": 1760.0, + "27": 1894.0, + "28": 1738.0, + "29": 1829.0, + "30": 1790.0, + "31": 2023.0, + "32": 1822.0, + "33": 1893.0, + "34": 1941.0, + "35": 2078.0, + "36": 2003.0, + "37": 2151.0, + "38": 2094.0, + "39": 2143.0, + "40": 2257.0, + "41": 2336.0, + "42": 1965.0, + "43": 2402.0, + "44": 2228.0, + "45": 2666.0, + "46": 2471.0, + "47": 2510.0, + "48": 2632.0, + "49": 2825.0, + "50": 2498.0 + } + }, + "mem-allocated-bytes": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 464552448.0, + "2": 464552448.0, + "3": 464552448.0, + "4": 464552448.0, + "5": 464552448.0, + "6": 464552448.0, + "7": 464552448.0, + "8": 464552448.0, + "9": 464552448.0, + "10": 464552448.0, + "11": 464552448.0, + "12": 464552448.0, + "13": 464552448.0, + "14": 464552448.0, + "15": 464552448.0, + "16": 464552448.0, + "17": 464552448.0, + "18": 464552448.0, + "19": 464552448.0, + "20": 464552448.0, + "21": 464552448.0, + "22": 464552448.0, + "23": 464552448.0, + "24": 464552448.0, + "25": 464552448.0, + "26": 464552448.0, + "27": 464552448.0, + "28": 464552448.0, + "29": 464552448.0, + "30": 464552448.0, + "31": 464552448.0, + "32": 464552448.0, + "33": 464552448.0, + "34": 464552448.0, + "35": 464552448.0, + "36": 464552448.0, + "37": 464552448.0, + "38": 464552448.0, + "39": 464552448.0, + "40": 464552448.0, + "41": 464552448.0, + "42": 464552448.0, + "43": 464552448.0, + "44": 464552448.0, + "45": 464552448.0, + "46": 464552448.0, + "47": 464552448.0, + "48": 464552448.0, + "49": 464552448.0, + "50": 464552448.0 + } + }, + "mem-max-allocated-bytes": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 1728999424.0, + "2": 1787834880.0, + "3": 1787834880.0, + "4": 1787834880.0, + "5": 1787834880.0, + "6": 1787834880.0, + "7": 1787834880.0, + "8": 1787834880.0, + "9": 1787834880.0, + "10": 1787834880.0, + "11": 1787834880.0, + "12": 1787834880.0, + "13": 1787834880.0, + "14": 1787834880.0, + "15": 1787834880.0, + "16": 1787834880.0, + "17": 1787834880.0, + "18": 1787834880.0, + "19": 1787834880.0, + "20": 1787834880.0, + "21": 1787834880.0, + "22": 1787834880.0, + "23": 1787834880.0, + "24": 1787834880.0, + "25": 1787834880.0, + "26": 1787834880.0, + "27": 1787834880.0, + "28": 1787834880.0, + "29": 1787834880.0, + "30": 1787834880.0, + "31": 1787834880.0, + "32": 1787834880.0, + "33": 1787834880.0, + "34": 1787834880.0, + "35": 1787834880.0, + "36": 1787834880.0, + "37": 1787834880.0, + "38": 1787834880.0, + "39": 1787834880.0, + "40": 1787834880.0, + "41": 1787834880.0, + "42": 1787834880.0, + "43": 1787834880.0, + "44": 1787834880.0, + "45": 1787834880.0, + "46": 1787834880.0, + "47": 1787834880.0, + "48": 1787834880.0, + "49": 1787834880.0, + "50": 1787834880.0 + } + }, + "iteration-time": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": "nan", + "2": 6.24733, + "3": 1.91189, + "4": 1.87045, + "5": 1.62791, + "6": 2.01487, + "7": 1.52699, + "8": 1.94407, + "9": 1.90627, + "10": 1.63362, + "11": 2.00131, + "12": 1.52055, + "13": 1.95086, + "14": 1.99851, + "15": 1.28991, + "16": 1.72904, + "17": 2.18003, + "18": 1.4681, + "19": 1.36379, + "20": 1.46459, + "21": 1.57703, + "22": 1.76317, + "23": 1.82828, + "24": 1.46843, + "25": 1.77733, + "26": 1.74613, + "27": 1.35765, + "28": 1.5617, + "29": 1.40264, + "30": 1.54505, + "31": 1.50211, + "32": 2.41074, + "33": 1.42486, + "34": 1.37479, + "35": 2.02524, + "36": 1.98499, + "37": 1.48976, + "38": 1.3669, + "39": 2.09095, + "40": 1.69308, + "41": 1.8834, + "42": 2.25556, + "43": 1.86841, + "44": 1.86539, + "45": 2.22331, + "46": 1.71541, + "47": 1.85792, + "48": 2.26308, + "49": 1.60283, + "50": 1.9925 + } + } +} \ No newline at end of file diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cross_entropy_loss_fusion_1node/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cross_entropy_loss_fusion_1node/model_config.yaml new file mode 100644 index 00000000000..2f73bf24f01 --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_cross_entropy_loss_fusion_1node/model_config.yaml @@ -0,0 +1,51 @@ +ENV_VARS: + CUDA_DEVICE_MAX_CONNECTIONS: 1 + NVTE_ALLOW_NONDETERMINISTIC_ALGO: 1 +MODEL_ARGS: + --num-layers: 12 + --hidden-size: 512 + --num-attention-heads: 8 + --log-params-norm: true + --log-num-zeros-in-grad: true + --log-validation-ppl-to-tensorboard: true + --log-timers-to-tensorboard: true + --tensorboard-dir: ${TENSORBOARD_PATH} + --micro-batch-size: 4 + --global-batch-size: 32 + --seq-length: 1024 + --max-position-embeddings: 1024 + --train-iters: 50 + --timing-log-level: 0 + --lr-decay-iters: 320000 + --save: ${CHECKPOINT_SAVE_PATH} + --load: ${CHECKPOINT_LOAD_PATH} + --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document + --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json + --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt + --split: 949,50,1 + --distributed-backend: nccl + --lr: 0.00015 + --lr-decay-style: cosine + --min-lr: 1.0e-5 + --weight-decay: 1e-2 + --clip-grad: 1.0 + --lr-warmup-fraction: .01 + --log-interval: 1 + --save-interval: 10000 + --eval-interval: 1000 + --eval-iters: 10 + --transformer-impl: transformer_engine + --tensor-model-parallel-size: 2 + --pipeline-model-parallel-size: 2 + --cross-entropy-loss-fusion: true + --no-gradient-accumulation-fusion: true + --attention-softmax-in-fp32: true + --use-mcore-models: true + --ckpt-format: torch_dist + --dist-ckpt-strictness: log_all # backward compatibility for TE changes + --data-cache-path: ${DATA_CACHE_PATH} + --bf16: true + --log-memory-to-tensorboard: true + --async-save: true + --use-persistent-ckpt-worker: true +TEST_TYPE: regular diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_mla_1node/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_mla_1node/golden_values_dev_dgx_gb200.json new file mode 100644 index 00000000000..0d39785a1ea --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_mla_1node/golden_values_dev_dgx_gb200.json @@ -0,0 +1,287 @@ +{ + "lm loss": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 10.95497, + "2": 10.95336, + "3": 10.95084, + "4": 10.95138, + "5": 10.95095, + "6": 10.95418, + "7": 10.95215, + "8": 10.95329, + "9": 10.9475, + "10": 10.94585, + "11": 10.94813, + "12": 10.96094, + "13": 10.9515, + "14": 10.95222, + "15": 10.94823, + "16": 10.94694, + "17": 10.95226, + "18": 10.94757, + "19": 10.95164, + "20": 10.93972, + "21": 10.93985, + "22": 10.93777, + "23": 10.92832, + "24": 10.92783, + "25": 10.93223, + "26": 10.92799, + "27": 10.9229, + "28": 10.89388, + "29": 10.88953, + "30": 10.8825, + "31": 10.88534, + "32": 10.88053, + "33": 10.87143, + "34": 10.86879, + "35": 10.86909, + "36": 10.86039, + "37": 10.85941, + "38": 10.83806, + "39": 10.80305, + "40": 10.80807, + "41": 10.79481, + "42": 10.77754, + "43": 10.77432, + "44": 10.77182, + "45": 10.77167, + "46": 10.75137, + "47": 10.73716, + "48": 10.71898, + "49": 10.73121, + "50": 10.71218 + } + }, + "num-zeros": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 23026300.0, + "2": 22914086.0, + "3": 22773382.0, + "4": 22840872.0, + "5": 22848084.0, + "6": 22810006.0, + "7": 22935596.0, + "8": 22675152.0, + "9": 22822660.0, + "10": 22552656.0, + "11": 22817710.0, + "12": 22705288.0, + "13": 23386302.0, + "14": 23056148.0, + "15": 22781850.0, + "16": 22886906.0, + "17": 22993636.0, + "18": 23058726.0, + "19": 23148372.0, + "20": 22787076.0, + "21": 22981484.0, + "22": 23010492.0, + "23": 22692514.0, + "24": 22925772.0, + "25": 22699682.0, + "26": 23068768.0, + "27": 22865260.0, + "28": 23063292.0, + "29": 23042696.0, + "30": 23008720.0, + "31": 22972018.0, + "32": 22730204.0, + "33": 22804204.0, + "34": 23139172.0, + "35": 22813746.0, + "36": 22758552.0, + "37": 23167014.0, + "38": 23028354.0, + "39": 23048620.0, + "40": 22812828.0, + "41": 23129236.0, + "42": 22752788.0, + "43": 23050796.0, + "44": 22767704.0, + "45": 22909896.0, + "46": 22797186.0, + "47": 22913150.0, + "48": 22896154.0, + "49": 22952004.0, + "50": 22707804.0 + } + }, + "mem-allocated-bytes": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 387483136.0, + "2": 387483136.0, + "3": 387483136.0, + "4": 387483136.0, + "5": 387483136.0, + "6": 387483136.0, + "7": 387483136.0, + "8": 387483136.0, + "9": 387483136.0, + "10": 387483136.0, + "11": 387483136.0, + "12": 387483136.0, + "13": 387483136.0, + "14": 387483136.0, + "15": 387483136.0, + "16": 387483136.0, + "17": 387483136.0, + "18": 387483136.0, + "19": 387483136.0, + "20": 387483136.0, + "21": 387483136.0, + "22": 387483136.0, + "23": 387483136.0, + "24": 387483136.0, + "25": 387483136.0, + "26": 387483136.0, + "27": 387483136.0, + "28": 387483136.0, + "29": 387483136.0, + "30": 387483136.0, + "31": 387483136.0, + "32": 387483136.0, + "33": 387483136.0, + "34": 387483136.0, + "35": 387483136.0, + "36": 387483136.0, + "37": 387483136.0, + "38": 387483136.0, + "39": 387483136.0, + "40": 387483136.0, + "41": 387483136.0, + "42": 387483136.0, + "43": 387483136.0, + "44": 387483136.0, + "45": 387483136.0, + "46": 387483136.0, + "47": 387483136.0, + "48": 387483136.0, + "49": 387483136.0, + "50": 387483136.0 + } + }, + "mem-max-allocated-bytes": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 1123435520.0, + "2": 1246423552.0, + "3": 1246423552.0, + "4": 1246423552.0, + "5": 1246423552.0, + "6": 1246423552.0, + "7": 1246423552.0, + "8": 1246423552.0, + "9": 1246423552.0, + "10": 1246423552.0, + "11": 1246423552.0, + "12": 1246423552.0, + "13": 1246423552.0, + "14": 1246423552.0, + "15": 1246423552.0, + "16": 1246423552.0, + "17": 1246423552.0, + "18": 1246423552.0, + "19": 1246423552.0, + "20": 1246423552.0, + "21": 1246423552.0, + "22": 1246423552.0, + "23": 1246423552.0, + "24": 1246423552.0, + "25": 1246423552.0, + "26": 1248258048.0, + "27": 1248258560.0, + "28": 1248258560.0, + "29": 1248258560.0, + "30": 1248258560.0, + "31": 1248258560.0, + "32": 1248258560.0, + "33": 1248258560.0, + "34": 1248258560.0, + "35": 1248258560.0, + "36": 1248258560.0, + "37": 1248258560.0, + "38": 1248258560.0, + "39": 1248258560.0, + "40": 1248258560.0, + "41": 1248258560.0, + "42": 1248258560.0, + "43": 1248258560.0, + "44": 1248258560.0, + "45": 1248258560.0, + "46": 1248258560.0, + "47": 1248258560.0, + "48": 1248258560.0, + "49": 1248258560.0, + "50": 1248258560.0 + } + }, + "iteration-time": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": "nan", + "2": 4.83564, + "3": 1.77329, + "4": 1.76732, + "5": 1.14743, + "6": 1.27866, + "7": 1.40471, + "8": 1.52809, + "9": 1.85934, + "10": 1.35422, + "11": 1.66809, + "12": 1.45421, + "13": 1.48852, + "14": 1.61638, + "15": 1.35936, + "16": 1.48157, + "17": 1.58173, + "18": 1.57213, + "19": 1.70714, + "20": 1.63531, + "21": 1.79973, + "22": 1.73956, + "23": 1.81441, + "24": 1.31445, + "25": 2.12573, + "26": 1.66586, + "27": 1.83747, + "28": 1.21653, + "29": 1.25168, + "30": 1.52684, + "31": 1.33831, + "32": 2.40359, + "33": 1.51321, + "34": 1.19186, + "35": 1.74093, + "36": 1.63651, + "37": 1.36169, + "38": 1.16745, + "39": 2.04148, + "40": 1.7202, + "41": 1.84594, + "42": 2.03378, + "43": 1.66299, + "44": 1.60653, + "45": 1.72402, + "46": 1.50615, + "47": 1.85149, + "48": 2.06771, + "49": 1.65665, + "50": 1.82848 + } + } +} \ No newline at end of file diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_mla_1node/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_mla_1node/model_config.yaml new file mode 100644 index 00000000000..d8afdbf0756 --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_mla_1node/model_config.yaml @@ -0,0 +1,61 @@ +ENV_VARS: + CUDA_DEVICE_MAX_CONNECTIONS: 1 + NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0 + NCCL_ALGO: Ring + CUBLAS_WORKSPACE_CONFIG: :4096:8 +MODEL_ARGS: + --num-layers: 4 + --hidden-size: 512 + --num-attention-heads: 8 + --multi-latent-attention: true + --q-lora-rank: 192 + --kv-lora-rank: 64 + --qk-head-dim: 16 + --qk-pos-emb-head-dim: 8 + --v-head-dim: 16 + --log-params-norm: true + --log-num-zeros-in-grad: true + --log-validation-ppl-to-tensorboard: true + --log-timers-to-tensorboard: true + --tensorboard-dir: ${TENSORBOARD_PATH} + --micro-batch-size: 4 + --global-batch-size: 32 + --seq-length: 1024 + --max-position-embeddings: 1024 + --train-iters: 50 + --timing-log-level: 0 + --lr-decay-iters: 320000 + --save: ${CHECKPOINT_SAVE_PATH} + --load: ${CHECKPOINT_LOAD_PATH} + --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document + --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json + --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt + --split: 949,50,1 + --distributed-backend: nccl + --lr: 0.00015 + --lr-decay-style: cosine + --min-lr: 1.0e-5 + --weight-decay: 1e-2 + --clip-grad: 1.0 + --lr-warmup-fraction: .01 + --log-interval: 1 + --save-interval: 25 + --eval-interval: 1000 + --eval-iters: 10 + --transformer-impl: transformer_engine + --tensor-model-parallel-size: 2 + --pipeline-model-parallel-size: 2 + --sequence-parallel: true + --untie-embeddings-and-output-weights: true + --deterministic-mode: true + --no-gradient-accumulation-fusion: true + --attention-softmax-in-fp32: true + --use-mcore-models: true + --ckpt-format: torch_dist + --data-cache-path: ${DATA_CACHE_PATH} + --bf16: true + --attention-backend: unfused + --log-memory-to-tensorboard: true + --async-save: true + --use-persistent-ckpt-worker: true +TEST_TYPE: ckpt-resume diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_1node/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_1node/golden_values_dev_dgx_gb200.json new file mode 100644 index 00000000000..236b42d4a3d --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_1node/golden_values_dev_dgx_gb200.json @@ -0,0 +1,537 @@ +{ + "lm loss": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 10.93086, + "2": 10.92833, + "3": 10.92866, + "4": 10.9288, + "5": 10.93088, + "6": 10.92771, + "7": 10.92336, + "8": 10.92159, + "9": 10.92031, + "10": 10.91964, + "11": 10.91027, + "12": 10.91814, + "13": 10.91523, + "14": 10.89825, + "15": 10.88042, + "16": 10.86841, + "17": 10.86882, + "18": 10.86258, + "19": 10.86185, + "20": 10.77348, + "21": 10.75516, + "22": 10.75521, + "23": 10.74508, + "24": 10.71106, + "25": 10.71082, + "26": 10.69968, + "27": 10.65192, + "28": 10.58249, + "29": 10.55636, + "30": 10.54863, + "31": 10.52499, + "32": 10.51761, + "33": 10.48144, + "34": 10.451, + "35": 10.44994, + "36": 10.43486, + "37": 10.39411, + "38": 10.39546, + "39": 10.36016, + "40": 10.35798, + "41": 10.32782, + "42": 10.30704, + "43": 10.28206, + "44": 10.25604, + "45": 10.27039, + "46": 10.23404, + "47": 10.21882, + "48": 10.17023, + "49": 10.1759, + "50": 10.17855, + "51": 10.19228, + "52": 10.13723, + "53": 10.13552, + "54": 10.10521, + "55": 10.08055, + "56": 10.11116, + "57": 10.09599, + "58": 10.10931, + "59": 10.05623, + "60": 10.07399, + "61": 10.02775, + "62": 10.0003, + "63": 10.07322, + "64": 10.03695, + "65": 10.00341, + "66": 10.03038, + "67": 10.0072, + "68": 9.97019, + "69": 9.9915, + "70": 9.97453, + "71": 10.00065, + "72": 9.97779, + "73": 9.97166, + "74": 9.95604, + "75": 9.93149, + "76": 9.96344, + "77": 9.95639, + "78": 9.90552, + "79": 9.91218, + "80": 9.92898, + "81": 9.95655, + "82": 9.89202, + "83": 9.85589, + "84": 9.78695, + "85": 9.78572, + "86": 9.88392, + "87": 9.91028, + "88": 9.88719, + "89": 9.81819, + "90": 9.81249, + "91": 9.82779, + "92": 9.81659, + "93": 9.74998, + "94": 9.8309, + "95": 9.82648, + "96": 9.80702, + "97": 9.74393, + "98": 9.77873, + "99": 9.82058, + "100": 9.71398 + } + }, + "num-zeros": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 1731.0, + "2": 1691.0, + "3": 1669.0, + "4": 1581.0, + "5": 1644.0, + "6": 1685.0, + "7": 1815.0, + "8": 1709.0, + "9": 1781.0, + "10": 1748.0, + "11": 1630.0, + "12": 1732.0, + "13": 1742.0, + "14": 1815.0, + "15": 1549.0, + "16": 1654.0, + "17": 1749.0, + "18": 1742.0, + "19": 1686.0, + "20": 1643.0, + "21": 1695.0, + "22": 1733.0, + "23": 1658.0, + "24": 1781.0, + "25": 1666.0, + "26": 1789.0, + "27": 1777.0, + "28": 1803.0, + "29": 1831.0, + "30": 1831.0, + "31": 2000.0, + "32": 1953.0, + "33": 1927.0, + "34": 2012.0, + "35": 2071.0, + "36": 1876.0, + "37": 2178.0, + "38": 2133.0, + "39": 2155.0, + "40": 2357.0, + "41": 2339.0, + "42": 1887.0, + "43": 2351.0, + "44": 2221.0, + "45": 2475.0, + "46": 2480.0, + "47": 2497.0, + "48": 2601.0, + "49": 2909.0, + "50": 2638.0, + "51": 2495.0, + "52": 2790.0, + "53": 2729.0, + "54": 2777.0, + "55": 2626.0, + "56": 2743.0, + "57": 2177.0, + "58": 3682.0, + "59": 2899.0, + "60": 2899.0, + "61": 2826.0, + "62": 3369.0, + "63": 3377.0, + "64": 3714.0, + "65": 2750.0, + "66": 3201.0, + "67": 3778.0, + "68": 3542.0, + "69": 3015.0, + "70": 3264.0, + "71": 3129.0, + "72": 3054.0, + "73": 3365.0, + "74": 3234.0, + "75": 3345.0, + "76": 3367.0, + "77": 3958.0, + "78": 3329.0, + "79": 3194.0, + "80": 2951.0, + "81": 3592.0, + "82": 2943.0, + "83": 3073.0, + "84": 3037.0, + "85": 2541.0, + "86": 3023.0, + "87": 2916.0, + "88": 3027.0, + "89": 3180.0, + "90": 3778.0, + "91": 3014.0, + "92": 2726.0, + "93": 3051.0, + "94": 3000.0, + "95": 3189.0, + "96": 3382.0, + "97": 3490.0, + "98": 3305.0, + "99": 3298.0, + "100": 3311.0 + } + }, + "mem-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 514359808.0, + "2": 514359808.0, + "3": 514359808.0, + "4": 514359808.0, + "5": 514359808.0, + "6": 514359808.0, + "7": 514359808.0, + "8": 514359808.0, + "9": 514359808.0, + "10": 514359808.0, + "11": 514359808.0, + "12": 514359808.0, + "13": 514359808.0, + "14": 514359808.0, + "15": 514359808.0, + "16": 514359808.0, + "17": 514359808.0, + "18": 514359808.0, + "19": 514359808.0, + "20": 514359808.0, + "21": 514359808.0, + "22": 514359808.0, + "23": 514359808.0, + "24": 514359808.0, + "25": 514359808.0, + "26": 514359808.0, + "27": 514359808.0, + "28": 514359808.0, + "29": 514359808.0, + "30": 514359808.0, + "31": 514359808.0, + "32": 514359808.0, + "33": 514359808.0, + "34": 514359808.0, + "35": 514359808.0, + "36": 514359808.0, + "37": 514359808.0, + "38": 514359808.0, + "39": 514359808.0, + "40": 514359808.0, + "41": 514359808.0, + "42": 514359808.0, + "43": 514359808.0, + "44": 514359808.0, + "45": 514359808.0, + "46": 514359808.0, + "47": 514359808.0, + "48": 514359808.0, + "49": 514359808.0, + "50": 514359808.0, + "51": 514359808.0, + "52": 514359808.0, + "53": 514359808.0, + "54": 514359808.0, + "55": 514359808.0, + "56": 514359808.0, + "57": 514359808.0, + "58": 514359808.0, + "59": 514359808.0, + "60": 514359808.0, + "61": 514359808.0, + "62": 514359808.0, + "63": 514359808.0, + "64": 514359808.0, + "65": 514359808.0, + "66": 514359808.0, + "67": 514359808.0, + "68": 514359808.0, + "69": 514359808.0, + "70": 514359808.0, + "71": 514359808.0, + "72": 514359808.0, + "73": 514359808.0, + "74": 514359808.0, + "75": 514359808.0, + "76": 514359808.0, + "77": 514359808.0, + "78": 514359808.0, + "79": 514359808.0, + "80": 514359808.0, + "81": 514359808.0, + "82": 514359808.0, + "83": 514359808.0, + "84": 514359808.0, + "85": 514359808.0, + "86": 514359808.0, + "87": 514359808.0, + "88": 514359808.0, + "89": 514359808.0, + "90": 514359808.0, + "91": 514359808.0, + "92": 514359808.0, + "93": 514359808.0, + "94": 514359808.0, + "95": 514359808.0, + "96": 514359808.0, + "97": 514359808.0, + "98": 514359808.0, + "99": 514359808.0, + "100": 514359808.0 + } + }, + "mem-max-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 1259110912.0, + "2": 1437086208.0, + "3": 1437086208.0, + "4": 1437086208.0, + "5": 1437086208.0, + "6": 1437086208.0, + "7": 1437086208.0, + "8": 1437086208.0, + "9": 1437086208.0, + "10": 1437086208.0, + "11": 1437086208.0, + "12": 1437086208.0, + "13": 1437086208.0, + "14": 1437086208.0, + "15": 1437086208.0, + "16": 1437086208.0, + "17": 1437086208.0, + "18": 1437086208.0, + "19": 1437086208.0, + "20": 1437086208.0, + "21": 1437086208.0, + "22": 1437086208.0, + "23": 1437086208.0, + "24": 1437086208.0, + "25": 1437086208.0, + "26": 1437086208.0, + "27": 1437086208.0, + "28": 1437086208.0, + "29": 1437086208.0, + "30": 1437086208.0, + "31": 1437086208.0, + "32": 1437086208.0, + "33": 1437086208.0, + "34": 1437086208.0, + "35": 1437086208.0, + "36": 1437086208.0, + "37": 1437086208.0, + "38": 1437086208.0, + "39": 1437086208.0, + "40": 1437086208.0, + "41": 1437086208.0, + "42": 1437086208.0, + "43": 1437086208.0, + "44": 1437086208.0, + "45": 1437086208.0, + "46": 1437086208.0, + "47": 1437086208.0, + "48": 1437086208.0, + "49": 1437086208.0, + "50": 1437086208.0, + "51": 1438131712.0, + "52": 1438131712.0, + "53": 1438131712.0, + "54": 1438131712.0, + "55": 1438131712.0, + "56": 1438131712.0, + "57": 1438131712.0, + "58": 1438131712.0, + "59": 1438131712.0, + "60": 1438131712.0, + "61": 1438131712.0, + "62": 1438131712.0, + "63": 1438131712.0, + "64": 1438131712.0, + "65": 1438131712.0, + "66": 1438131712.0, + "67": 1438131712.0, + "68": 1438131712.0, + "69": 1438131712.0, + "70": 1438131712.0, + "71": 1438131712.0, + "72": 1438131712.0, + "73": 1438131712.0, + "74": 1438131712.0, + "75": 1438131712.0, + "76": 1438131712.0, + "77": 1438131712.0, + "78": 1438131712.0, + "79": 1438131712.0, + "80": 1438131712.0, + "81": 1438131712.0, + "82": 1438131712.0, + "83": 1438131712.0, + "84": 1438131712.0, + "85": 1438131712.0, + "86": 1438131712.0, + "87": 1438131712.0, + "88": 1438131712.0, + "89": 1438131712.0, + "90": 1438131712.0, + "91": 1438131712.0, + "92": 1438131712.0, + "93": 1438131712.0, + "94": 1438131712.0, + "95": 1438131712.0, + "96": 1438131712.0, + "97": 1438131712.0, + "98": 1438131712.0, + "99": 1438131712.0, + "100": 1438131712.0 + } + }, + "iteration-time": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": "nan", + "2": 4.63967, + "3": 1.58044, + "4": 1.59341, + "5": 1.49957, + "6": 1.61629, + "7": 1.48635, + "8": 2.19313, + "9": 2.34866, + "10": 1.68381, + "11": 2.12043, + "12": 1.69598, + "13": 2.13363, + "14": 2.32599, + "15": 1.59484, + "16": 1.90532, + "17": 2.29726, + "18": 2.0119, + "19": 2.10462, + "20": 1.83342, + "21": 2.42817, + "22": 1.68203, + "23": 2.12753, + "24": 1.72189, + "25": 2.26514, + "26": 2.12609, + "27": 1.67539, + "28": 2.03755, + "29": 1.38567, + "30": 1.81921, + "31": 1.37318, + "32": 2.81633, + "33": 1.25413, + "34": 1.59957, + "35": 2.11184, + "36": 1.88312, + "37": 1.73004, + "38": 1.47702, + "39": 2.63734, + "40": 1.91747, + "41": 1.87392, + "42": 2.41996, + "43": 1.64846, + "44": 1.73295, + "45": 2.1201, + "46": 1.6429, + "47": 1.96336, + "48": 2.11089, + "49": 1.72422, + "50": 1.75719, + "51": 1.26857, + "52": 1.88631, + "53": 1.74702, + "54": 2.00497, + "55": 1.47315, + "56": 2.74355, + "57": 1.31579, + "58": 1.98611, + "59": 1.63873, + "60": 1.49597, + "61": 2.09588, + "62": 2.11415, + "63": 1.34927, + "64": 1.48441, + "65": 1.69359, + "66": 1.67113, + "67": 1.75275, + "68": 1.20696, + "69": 1.70009, + "70": 1.57882, + "71": 2.18747, + "72": 1.54455, + "73": 1.90761, + "74": 1.84117, + "75": 1.64196, + "76": 1.94248, + "77": 1.53507, + "78": 1.75571, + "79": 1.91817, + "80": 1.8633, + "81": 2.2349, + "82": 1.87368, + "83": 2.05421, + "84": 1.5348, + "85": 1.67617, + "86": 1.64461, + "87": 2.17425, + "88": 1.76563, + "89": 1.94035, + "90": 1.77572, + "91": 1.88463, + "92": 2.05897, + "93": 1.39803, + "94": 1.86429, + "95": 1.97249, + "96": 1.86059, + "97": 1.67634, + "98": 1.73432, + "99": 2.35279, + "100": 1.49691 + } + } +} \ No newline at end of file diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_1node/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_1node/model_config.yaml new file mode 100644 index 00000000000..9436fa2a5e6 --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_1node/model_config.yaml @@ -0,0 +1,54 @@ +ENV_VARS: + CUDA_DEVICE_MAX_CONNECTIONS: 1 + NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0 + NCCL_ALGO: Ring + CUBLAS_WORKSPACE_CONFIG: :4096:8 +MODEL_ARGS: + --num-layers: 12 + --hidden-size: 512 + --num-attention-heads: 8 + --log-params-norm: true + --log-num-zeros-in-grad: true + --log-validation-ppl-to-tensorboard: true + --log-timers-to-tensorboard: true + --tensorboard-dir: ${TENSORBOARD_PATH} + --micro-batch-size: 4 + --global-batch-size: 32 + --seq-length: 1024 + --max-position-embeddings: 1024 + --train-iters: 100 + --timing-log-level: 0 + --lr-decay-iters: 320000 + --save: ${CHECKPOINT_SAVE_PATH} + --load: ${CHECKPOINT_LOAD_PATH} + --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document + --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json + --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt + --split: 949,50,1 + --distributed-backend: nccl + --lr: 0.00015 + --lr-decay-style: cosine + --min-lr: 1.0e-5 + --weight-decay: 1e-2 + --clip-grad: 1.0 + --lr-warmup-fraction: .01 + --log-interval: 1 + --save-interval: 50 + --eval-interval: 1000 + --eval-iters: 10 + --transformer-impl: transformer_engine + --tensor-model-parallel-size: 2 + --pipeline-model-parallel-size: 2 + --deterministic-mode: true + --no-gradient-accumulation-fusion: true + --attention-softmax-in-fp32: true + --use-checkpoint-opt_param-scheduler: true + --use-mcore-models: true + --ckpt-format: torch_dist + --dist-ckpt-strictness: log_all # backward compatibility for TE changes + --data-cache-path: ${DATA_CACHE_PATH} + --bf16: true + --log-memory-to-tensorboard: true + --async-save: true + --use-persistent-ckpt-worker: true +TEST_TYPE: ckpt-resume diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_cp2_nondeterministic_1node/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_cp2_nondeterministic_1node/golden_values_dev_dgx_gb200.json new file mode 100644 index 00000000000..aab973f1fdd --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_cp2_nondeterministic_1node/golden_values_dev_dgx_gb200.json @@ -0,0 +1,537 @@ +{ + "lm loss": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 10.95727, + "2": 10.94891, + "3": 10.9518, + "4": 10.93707, + "5": 10.94343, + "6": 10.93813, + "7": 10.94906, + "8": 10.92915, + "9": 10.94291, + "10": 10.9327, + "11": 10.93261, + "12": 10.92192, + "13": 10.91844, + "14": 10.91221, + "15": 10.89102, + "16": 10.87485, + "17": 10.88662, + "18": 10.86792, + "19": 10.8732, + "20": 10.77511, + "21": 10.76178, + "22": 10.75198, + "23": 10.73998, + "24": 10.70106, + "25": 10.71371, + "26": 10.68552, + "27": 10.64306, + "28": 10.56425, + "29": 10.53593, + "30": 10.52934, + "31": 10.51591, + "32": 10.49935, + "33": 10.46544, + "34": 10.42361, + "35": 10.42258, + "36": 10.41394, + "37": 10.36776, + "38": 10.37996, + "39": 10.34014, + "40": 10.32559, + "41": 10.30513, + "42": 10.29477, + "43": 10.257, + "44": 10.23372, + "45": 10.2453, + "46": 10.21281, + "47": 10.19584, + "48": 10.15328, + "49": 10.14471, + "50": 10.1515, + "51": 10.15987, + "52": 10.1084, + "53": 10.10792, + "54": 10.07446, + "55": 10.04844, + "56": 10.07751, + "57": 10.05889, + "58": 10.07687, + "59": 10.0227, + "60": 10.03903, + "61": 9.99475, + "62": 9.96367, + "63": 10.03992, + "64": 9.99851, + "65": 9.96344, + "66": 9.99444, + "67": 9.96879, + "68": 9.9281, + "69": 9.94598, + "70": 9.9308, + "71": 9.96382, + "72": 9.92607, + "73": 9.91706, + "74": 9.9018, + "75": 9.86961, + "76": 9.91505, + "77": 9.9064, + "78": 9.84568, + "79": 9.85522, + "80": 9.87464, + "81": 9.90489, + "82": 9.84702, + "83": 9.79937, + "84": 9.72985, + "85": 9.72432, + "86": 9.82918, + "87": 9.8655, + "88": 9.83649, + "89": 9.76529, + "90": 9.75735, + "91": 9.77548, + "92": 9.76551, + "93": 9.68781, + "94": 9.77514, + "95": 9.77584, + "96": 9.75058, + "97": 9.67837, + "98": 9.7197, + "99": 9.77323, + "100": 9.65788 + } + }, + "num-zeros": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 639.0, + "2": 608.0, + "3": 624.0, + "4": 612.0, + "5": 625.0, + "6": 641.0, + "7": 653.0, + "8": 660.0, + "9": 625.0, + "10": 621.0, + "11": 652.0, + "12": 601.0, + "13": 620.0, + "14": 677.0, + "15": 577.0, + "16": 639.0, + "17": 642.0, + "18": 657.0, + "19": 672.0, + "20": 615.0, + "21": 685.0, + "22": 655.0, + "23": 713.0, + "24": 625.0, + "25": 639.0, + "26": 684.0, + "27": 672.0, + "28": 700.0, + "29": 719.0, + "30": 689.0, + "31": 777.0, + "32": 813.0, + "33": 806.0, + "34": 756.0, + "35": 727.0, + "36": 793.0, + "37": 867.0, + "38": 749.0, + "39": 802.0, + "40": 829.0, + "41": 796.0, + "42": 720.0, + "43": 853.0, + "44": 893.0, + "45": 937.0, + "46": 912.0, + "47": 915.0, + "48": 962.0, + "49": 971.0, + "50": 1004.0, + "51": 976.0, + "52": 1038.0, + "53": 955.0, + "54": 1092.0, + "55": 969.0, + "56": 1054.0, + "57": 855.0, + "58": 1323.0, + "59": 1030.0, + "60": 1091.0, + "61": 1072.0, + "62": 1175.0, + "63": 1233.0, + "64": 1219.0, + "65": 1003.0, + "66": 1152.0, + "67": 1348.0, + "68": 1201.0, + "69": 1015.0, + "70": 1129.0, + "71": 1091.0, + "72": 1111.0, + "73": 1181.0, + "74": 1136.0, + "75": 1038.0, + "76": 1128.0, + "77": 1245.0, + "78": 1116.0, + "79": 1133.0, + "80": 1104.0, + "81": 1168.0, + "82": 1220.0, + "83": 1013.0, + "84": 1033.0, + "85": 960.0, + "86": 1129.0, + "87": 1125.0, + "88": 1161.0, + "89": 1041.0, + "90": 1250.0, + "91": 1037.0, + "92": 1082.0, + "93": 1156.0, + "94": 1238.0, + "95": 1056.0, + "96": 1181.0, + "97": 1136.0, + "98": 1197.0, + "99": 1211.0, + "100": 1047.0 + } + }, + "mem-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 638631424.0, + "2": 638631424.0, + "3": 638631424.0, + "4": 638631424.0, + "5": 638631424.0, + "6": 638631424.0, + "7": 638631424.0, + "8": 638631424.0, + "9": 638631424.0, + "10": 638631424.0, + "11": 638631424.0, + "12": 638631424.0, + "13": 638631424.0, + "14": 638631424.0, + "15": 638631424.0, + "16": 638631424.0, + "17": 638631424.0, + "18": 638631424.0, + "19": 638631424.0, + "20": 638631424.0, + "21": 638631424.0, + "22": 638631424.0, + "23": 638631424.0, + "24": 638631424.0, + "25": 638631424.0, + "26": 638631424.0, + "27": 638631424.0, + "28": 638631424.0, + "29": 638631424.0, + "30": 638631424.0, + "31": 638631424.0, + "32": 638631424.0, + "33": 638631424.0, + "34": 638631424.0, + "35": 638631424.0, + "36": 638631424.0, + "37": 638631424.0, + "38": 638631424.0, + "39": 638631424.0, + "40": 638631424.0, + "41": 638631424.0, + "42": 638631424.0, + "43": 638631424.0, + "44": 638631424.0, + "45": 638631424.0, + "46": 638631424.0, + "47": 638631424.0, + "48": 638631424.0, + "49": 638631424.0, + "50": 638631424.0, + "51": 638631424.0, + "52": 638631424.0, + "53": 638631424.0, + "54": 638631424.0, + "55": 638631424.0, + "56": 638631424.0, + "57": 638631424.0, + "58": 638631424.0, + "59": 638631424.0, + "60": 638631424.0, + "61": 638631424.0, + "62": 638631424.0, + "63": 638631424.0, + "64": 638631424.0, + "65": 638631424.0, + "66": 638631424.0, + "67": 638631424.0, + "68": 638631424.0, + "69": 638631424.0, + "70": 638631424.0, + "71": 638631424.0, + "72": 638631424.0, + "73": 638631424.0, + "74": 638631424.0, + "75": 638631424.0, + "76": 638631424.0, + "77": 638631424.0, + "78": 638631424.0, + "79": 638631424.0, + "80": 638631424.0, + "81": 638631424.0, + "82": 638631424.0, + "83": 638631424.0, + "84": 638631424.0, + "85": 638631424.0, + "86": 638631424.0, + "87": 638631424.0, + "88": 638631424.0, + "89": 638631424.0, + "90": 638631424.0, + "91": 638631424.0, + "92": 638631424.0, + "93": 638631424.0, + "94": 638631424.0, + "95": 638631424.0, + "96": 638631424.0, + "97": 638631424.0, + "98": 638631424.0, + "99": 638631424.0, + "100": 638631424.0 + } + }, + "mem-max-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 911682048.0, + "2": 1171025920.0, + "3": 1173122560.0, + "4": 1173122560.0, + "5": 1173122560.0, + "6": 1173122560.0, + "7": 1173122560.0, + "8": 1173122560.0, + "9": 1173122560.0, + "10": 1173122560.0, + "11": 1173122560.0, + "12": 1173122560.0, + "13": 1173122560.0, + "14": 1173122560.0, + "15": 1173122560.0, + "16": 1175741952.0, + "17": 1175741952.0, + "18": 1175741952.0, + "19": 1175741952.0, + "20": 1178364416.0, + "21": 1178364416.0, + "22": 1178364416.0, + "23": 1178364416.0, + "24": 1178364416.0, + "25": 1178364416.0, + "26": 1178364416.0, + "27": 1178364416.0, + "28": 1178364416.0, + "29": 1178364416.0, + "30": 1178364416.0, + "31": 1178364416.0, + "32": 1178364416.0, + "33": 1178364416.0, + "34": 1178364416.0, + "35": 1178364416.0, + "36": 1178364416.0, + "37": 1178364416.0, + "38": 1178364416.0, + "39": 1178364416.0, + "40": 1178364416.0, + "41": 1178364416.0, + "42": 1178364416.0, + "43": 1178364416.0, + "44": 1178364416.0, + "45": 1178364416.0, + "46": 1178364416.0, + "47": 1178364416.0, + "48": 1178364416.0, + "49": 1178364416.0, + "50": 1178364416.0, + "51": 1178364416.0, + "52": 1179937792.0, + "53": 1179937792.0, + "54": 1179937792.0, + "55": 1179937792.0, + "56": 1179937792.0, + "57": 1179937792.0, + "58": 1179937792.0, + "59": 1179937792.0, + "60": 1179937792.0, + "61": 1179937792.0, + "62": 1179937792.0, + "63": 1179937792.0, + "64": 1179938304.0, + "65": 1179938304.0, + "66": 1179938304.0, + "67": 1179938304.0, + "68": 1179938304.0, + "69": 1179938304.0, + "70": 1179938304.0, + "71": 1179938304.0, + "72": 1179938304.0, + "73": 1179938304.0, + "74": 1179938304.0, + "75": 1179938304.0, + "76": 1179938304.0, + "77": 1179938304.0, + "78": 1179938304.0, + "79": 1179938304.0, + "80": 1179938304.0, + "81": 1179938304.0, + "82": 1179938304.0, + "83": 1179938304.0, + "84": 1179938304.0, + "85": 1179938304.0, + "86": 1179938304.0, + "87": 1179938304.0, + "88": 1179938304.0, + "89": 1179938304.0, + "90": 1179938304.0, + "91": 1179938304.0, + "92": 1179938304.0, + "93": 1179938304.0, + "94": 1179938304.0, + "95": 1179938304.0, + "96": 1179938304.0, + "97": 1179938304.0, + "98": 1179938304.0, + "99": 1179938304.0, + "100": 1179938304.0 + } + }, + "iteration-time": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": "nan", + "2": 5.85121, + "3": 2.04915, + "4": 1.90053, + "5": 1.55912, + "6": 2.03751, + "7": 1.39854, + "8": 2.25131, + "9": 2.16265, + "10": 1.96754, + "11": 1.90225, + "12": 1.77483, + "13": 1.90829, + "14": 2.09179, + "15": 1.72252, + "16": 1.51498, + "17": 2.15484, + "18": 1.67154, + "19": 5.69475, + "20": 1.76983, + "21": 2.13492, + "22": 1.57505, + "23": 2.12141, + "24": 1.48628, + "25": 2.03498, + "26": 2.23442, + "27": 1.64032, + "28": 1.84396, + "29": 1.56425, + "30": 1.55695, + "31": 1.42748, + "32": 2.50574, + "33": 1.57614, + "34": 1.42113, + "35": 2.20656, + "36": 2.0182, + "37": 1.65544, + "38": 1.37611, + "39": 2.04187, + "40": 1.78305, + "41": 1.88917, + "42": 2.14046, + "43": 1.49828, + "44": 1.80253, + "45": 2.05249, + "46": 1.60904, + "47": 1.74841, + "48": 1.69017, + "49": 1.40422, + "50": 1.84542, + "51": 1.02628, + "52": 2.5179, + "53": 1.9986, + "54": 2.06687, + "55": 1.71982, + "56": 2.37815, + "57": 1.07158, + "58": 2.03529, + "59": 1.40176, + "60": 1.64843, + "61": 1.82064, + "62": 2.02412, + "63": 1.36765, + "64": 1.52555, + "65": 1.65597, + "66": 1.85728, + "67": 1.90578, + "68": 1.32413, + "69": 1.84582, + "70": 1.54774, + "71": 2.18184, + "72": 1.42032, + "73": 1.76812, + "74": 1.43176, + "75": 1.55396, + "76": 1.80903, + "77": 1.79432, + "78": 1.37738, + "79": 1.92617, + "80": 1.49382, + "81": 2.02128, + "82": 1.90476, + "83": 2.53464, + "84": 0.91188, + "85": 1.66267, + "86": 1.5303, + "87": 2.217, + "88": 1.93512, + "89": 1.83711, + "90": 1.80038, + "91": 1.70712, + "92": 2.21105, + "93": 1.38851, + "94": 2.0548, + "95": 1.63607, + "96": 2.29886, + "97": 1.02625, + "98": 1.78943, + "99": 2.18199, + "100": 1.5124 + } + } +} \ No newline at end of file diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_cp2_nondeterministic_1node/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_cp2_nondeterministic_1node/model_config.yaml new file mode 100644 index 00000000000..c682b78eedc --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_cp2_nondeterministic_1node/model_config.yaml @@ -0,0 +1,56 @@ +ENV_VARS: + CUDA_DEVICE_MAX_CONNECTIONS: 1 + NVTE_ALLOW_NONDETERMINISTIC_ALGO: 1 +MODEL_ARGS: + --num-layers: 12 + --hidden-size: 512 + --num-attention-heads: 8 + --log-params-norm: true + --log-num-zeros-in-grad: true + --log-validation-ppl-to-tensorboard: true + --log-timers-to-tensorboard: true + --tensorboard-dir: ${TENSORBOARD_PATH} + --micro-batch-size: 4 + --global-batch-size: 32 + --seq-length: 1024 + --max-position-embeddings: 1024 + --train-iters: 100 + --timing-log-level: 0 + --lr-decay-iters: 320000 + --save: ${CHECKPOINT_SAVE_PATH} + --load: ${CHECKPOINT_LOAD_PATH} + --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document + --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json + --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt + --split: 949,50,1 + --distributed-backend: nccl + --lr: 0.00015 + --lr-decay-style: cosine + --min-lr: 1.0e-5 + --weight-decay: 1e-2 + --clip-grad: 1.0 + --lr-warmup-fraction: .01 + --log-interval: 1 + --save-interval: 50 + --eval-interval: 1000 + --eval-iters: 10 + --transformer-impl: transformer_engine + --tensor-model-parallel-size: 2 + --pipeline-model-parallel-size: 1 + --context-parallel-size: 2 + --sequence-parallel: true + --hidden-dropout: 0.0 + --attention-dropout: 0.0 + --no-gradient-accumulation-fusion: true + --attention-softmax-in-fp32: true + --use-checkpoint-opt_param-scheduler: true + --use-mcore-models: true + --ckpt-format: torch_dist + --dist-ckpt-strictness: log_all # backward compatibility for TE changes + --data-cache-path: ${DATA_CACHE_PATH} + --bf16: true + --attention-backend: flash + --log-memory-to-tensorboard: true + --async-save: true + --use-persistent-ckpt-worker: true +TEST_TYPE: ckpt-resume diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_cross_entropy_loss_fusion_1node/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_cross_entropy_loss_fusion_1node/golden_values_dev_dgx_gb200.json new file mode 100644 index 00000000000..402f169bae6 --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_cross_entropy_loss_fusion_1node/golden_values_dev_dgx_gb200.json @@ -0,0 +1,537 @@ +{ + "lm loss": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 10.93259, + "2": 10.92912, + "3": 10.92983, + "4": 10.92995, + "5": 10.93095, + "6": 10.9284, + "7": 10.92478, + "8": 10.9227, + "9": 10.92096, + "10": 10.92136, + "11": 10.91166, + "12": 10.91917, + "13": 10.91612, + "14": 10.90004, + "15": 10.88228, + "16": 10.86998, + "17": 10.87035, + "18": 10.86545, + "19": 10.86382, + "20": 10.77637, + "21": 10.75923, + "22": 10.75919, + "23": 10.74897, + "24": 10.71537, + "25": 10.71531, + "26": 10.70433, + "27": 10.65687, + "28": 10.58671, + "29": 10.56059, + "30": 10.55328, + "31": 10.52941, + "32": 10.52185, + "33": 10.486, + "34": 10.45518, + "35": 10.45396, + "36": 10.4396, + "37": 10.39899, + "38": 10.39946, + "39": 10.36401, + "40": 10.36156, + "41": 10.33164, + "42": 10.31049, + "43": 10.28599, + "44": 10.25957, + "45": 10.27364, + "46": 10.23728, + "47": 10.22227, + "48": 10.17318, + "49": 10.17861, + "50": 10.18086, + "51": 10.19466, + "52": 10.13959, + "53": 10.13785, + "54": 10.10764, + "55": 10.0825, + "56": 10.11304, + "57": 10.09826, + "58": 10.11106, + "59": 10.05788, + "60": 10.07575, + "61": 10.02908, + "62": 10.002, + "63": 10.07438, + "64": 10.03838, + "65": 10.0052, + "66": 10.03156, + "67": 10.00867, + "68": 9.97182, + "69": 9.9932, + "70": 9.97638, + "71": 10.00194, + "72": 9.98001, + "73": 9.97431, + "74": 9.95857, + "75": 9.93425, + "76": 9.96548, + "77": 9.95871, + "78": 9.90838, + "79": 9.91525, + "80": 9.93262, + "81": 9.95987, + "82": 9.89402, + "83": 9.85935, + "84": 9.78931, + "85": 9.78863, + "86": 9.88604, + "87": 9.91131, + "88": 9.88885, + "89": 9.81862, + "90": 9.81373, + "91": 9.82944, + "92": 9.81732, + "93": 9.75168, + "94": 9.83238, + "95": 9.82701, + "96": 9.80836, + "97": 9.74563, + "98": 9.77927, + "99": 9.82097, + "100": 9.71416 + } + }, + "num-zeros": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 1776.0, + "2": 1669.0, + "3": 1716.0, + "4": 1714.0, + "5": 1682.0, + "6": 1599.0, + "7": 1931.0, + "8": 1758.0, + "9": 1800.0, + "10": 1756.0, + "11": 1683.0, + "12": 1633.0, + "13": 1684.0, + "14": 1819.0, + "15": 1596.0, + "16": 1692.0, + "17": 1808.0, + "18": 1721.0, + "19": 1689.0, + "20": 1641.0, + "21": 1802.0, + "22": 1707.0, + "23": 1650.0, + "24": 1786.0, + "25": 1630.0, + "26": 1729.0, + "27": 1793.0, + "28": 1808.0, + "29": 1864.0, + "30": 1743.0, + "31": 1967.0, + "32": 1949.0, + "33": 1907.0, + "34": 1988.0, + "35": 2018.0, + "36": 1960.0, + "37": 2190.0, + "38": 2100.0, + "39": 2200.0, + "40": 2231.0, + "41": 2437.0, + "42": 1955.0, + "43": 2300.0, + "44": 2132.0, + "45": 2555.0, + "46": 2402.0, + "47": 2512.0, + "48": 2570.0, + "49": 2878.0, + "50": 2540.0, + "51": 2561.0, + "52": 2739.0, + "53": 2696.0, + "54": 2839.0, + "55": 2688.0, + "56": 2751.0, + "57": 2158.0, + "58": 3670.0, + "59": 2940.0, + "60": 3024.0, + "61": 2811.0, + "62": 3352.0, + "63": 3419.0, + "64": 3737.0, + "65": 2695.0, + "66": 3127.0, + "67": 3849.0, + "68": 3263.0, + "69": 3124.0, + "70": 3292.0, + "71": 3173.0, + "72": 3026.0, + "73": 3452.0, + "74": 3361.0, + "75": 3433.0, + "76": 3346.0, + "77": 3896.0, + "78": 3393.0, + "79": 3271.0, + "80": 3129.0, + "81": 3557.0, + "82": 2820.0, + "83": 3099.0, + "84": 2978.0, + "85": 2515.0, + "86": 3031.0, + "87": 2934.0, + "88": 2932.0, + "89": 3107.0, + "90": 3707.0, + "91": 2916.0, + "92": 2778.0, + "93": 3071.0, + "94": 3014.0, + "95": 3419.0, + "96": 3326.0, + "97": 3489.0, + "98": 3336.0, + "99": 3421.0, + "100": 3365.0 + } + }, + "mem-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 464552448.0, + "2": 464552448.0, + "3": 464552448.0, + "4": 464552448.0, + "5": 464552448.0, + "6": 464552448.0, + "7": 464552448.0, + "8": 464552448.0, + "9": 464552448.0, + "10": 464552448.0, + "11": 464552448.0, + "12": 464552448.0, + "13": 464552448.0, + "14": 464552448.0, + "15": 464552448.0, + "16": 464552448.0, + "17": 464552448.0, + "18": 464552448.0, + "19": 464552448.0, + "20": 464552448.0, + "21": 464552448.0, + "22": 464552448.0, + "23": 464552448.0, + "24": 464552448.0, + "25": 464552448.0, + "26": 464552448.0, + "27": 464552448.0, + "28": 464552448.0, + "29": 464552448.0, + "30": 464552448.0, + "31": 464552448.0, + "32": 464552448.0, + "33": 464552448.0, + "34": 464552448.0, + "35": 464552448.0, + "36": 464552448.0, + "37": 464552448.0, + "38": 464552448.0, + "39": 464552448.0, + "40": 464552448.0, + "41": 464552448.0, + "42": 464552448.0, + "43": 464552448.0, + "44": 464552448.0, + "45": 464552448.0, + "46": 464552448.0, + "47": 464552448.0, + "48": 464552448.0, + "49": 464552448.0, + "50": 464552448.0, + "51": 464552448.0, + "52": 464552448.0, + "53": 464552448.0, + "54": 464552448.0, + "55": 464552448.0, + "56": 464552448.0, + "57": 464552448.0, + "58": 464552448.0, + "59": 464552448.0, + "60": 464552448.0, + "61": 464552448.0, + "62": 464552448.0, + "63": 464552448.0, + "64": 464552448.0, + "65": 464552448.0, + "66": 464552448.0, + "67": 464552448.0, + "68": 464552448.0, + "69": 464552448.0, + "70": 464552448.0, + "71": 464552448.0, + "72": 464552448.0, + "73": 464552448.0, + "74": 464552448.0, + "75": 464552448.0, + "76": 464552448.0, + "77": 464552448.0, + "78": 464552448.0, + "79": 464552448.0, + "80": 464552448.0, + "81": 464552448.0, + "82": 464552448.0, + "83": 464552448.0, + "84": 464552448.0, + "85": 464552448.0, + "86": 464552448.0, + "87": 464552448.0, + "88": 464552448.0, + "89": 464552448.0, + "90": 464552448.0, + "91": 464552448.0, + "92": 464552448.0, + "93": 464552448.0, + "94": 464552448.0, + "95": 464552448.0, + "96": 464552448.0, + "97": 464552448.0, + "98": 464552448.0, + "99": 464552448.0, + "100": 464552448.0 + } + }, + "mem-max-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 1728999424.0, + "2": 1787834880.0, + "3": 1787834880.0, + "4": 1787834880.0, + "5": 1787834880.0, + "6": 1787834880.0, + "7": 1787834880.0, + "8": 1787834880.0, + "9": 1787834880.0, + "10": 1787834880.0, + "11": 1787834880.0, + "12": 1787834880.0, + "13": 1787834880.0, + "14": 1787834880.0, + "15": 1787834880.0, + "16": 1787834880.0, + "17": 1787834880.0, + "18": 1787834880.0, + "19": 1787834880.0, + "20": 1787834880.0, + "21": 1787834880.0, + "22": 1787834880.0, + "23": 1787834880.0, + "24": 1787834880.0, + "25": 1787834880.0, + "26": 1787834880.0, + "27": 1787834880.0, + "28": 1787834880.0, + "29": 1787834880.0, + "30": 1787834880.0, + "31": 1787834880.0, + "32": 1787834880.0, + "33": 1787834880.0, + "34": 1787834880.0, + "35": 1787834880.0, + "36": 1787834880.0, + "37": 1787834880.0, + "38": 1787834880.0, + "39": 1787834880.0, + "40": 1787834880.0, + "41": 1787834880.0, + "42": 1787834880.0, + "43": 1787834880.0, + "44": 1787834880.0, + "45": 1787834880.0, + "46": 1787834880.0, + "47": 1787834880.0, + "48": 1787834880.0, + "49": 1787834880.0, + "50": 1787834880.0, + "51": 1788883456.0, + "52": 1788883456.0, + "53": 1788883456.0, + "54": 1788883456.0, + "55": 1788883456.0, + "56": 1788883456.0, + "57": 1788883456.0, + "58": 1788883456.0, + "59": 1788883456.0, + "60": 1788883456.0, + "61": 1788883456.0, + "62": 1788883456.0, + "63": 1788883456.0, + "64": 1788883456.0, + "65": 1788883456.0, + "66": 1788883456.0, + "67": 1788883456.0, + "68": 1788883456.0, + "69": 1788883456.0, + "70": 1788883456.0, + "71": 1788883456.0, + "72": 1788883456.0, + "73": 1788883456.0, + "74": 1788883456.0, + "75": 1788883456.0, + "76": 1788883456.0, + "77": 1788883456.0, + "78": 1788883456.0, + "79": 1788883456.0, + "80": 1788883456.0, + "81": 1788883456.0, + "82": 1788883456.0, + "83": 1788883456.0, + "84": 1788883456.0, + "85": 1788883456.0, + "86": 1788883456.0, + "87": 1788883456.0, + "88": 1788883456.0, + "89": 1788883456.0, + "90": 1788883456.0, + "91": 1788883456.0, + "92": 1788883456.0, + "93": 1788883456.0, + "94": 1788883456.0, + "95": 1788883456.0, + "96": 1788883456.0, + "97": 1788883456.0, + "98": 1788883456.0, + "99": 1788883456.0, + "100": 1788883456.0 + } + }, + "iteration-time": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": "nan", + "2": 6.86906, + "3": 1.69314, + "4": 1.57202, + "5": 1.32253, + "6": 1.42742, + "7": 1.51011, + "8": 1.90198, + "9": 1.90421, + "10": 1.79417, + "11": 1.75284, + "12": 1.48162, + "13": 1.65245, + "14": 1.79916, + "15": 1.41448, + "16": 1.57698, + "17": 1.95732, + "18": 1.75052, + "19": 1.48504, + "20": 1.40544, + "21": 1.70284, + "22": 1.28083, + "23": 1.47023, + "24": 1.41909, + "25": 1.81843, + "26": 2.05699, + "27": 1.7199, + "28": 1.55934, + "29": 1.27651, + "30": 1.69074, + "31": 1.3624, + "32": 2.28511, + "33": 1.31901, + "34": 1.19801, + "35": 1.40903, + "36": 1.77576, + "37": 1.48438, + "38": 1.33416, + "39": 1.82401, + "40": 1.51888, + "41": 1.55862, + "42": 1.55292, + "43": 1.35285, + "44": 1.61383, + "45": 2.4235, + "46": 1.86486, + "47": 1.78046, + "48": 1.99746, + "49": 1.56012, + "50": 1.87873, + "51": 1.01122, + "52": 1.55857, + "53": 1.73737, + "54": 1.79969, + "55": 1.50641, + "56": 1.89979, + "57": 1.05508, + "58": 1.75128, + "59": 1.34324, + "60": 1.66855, + "61": 1.98861, + "62": 2.0261, + "63": 1.64981, + "64": 1.70385, + "65": 1.26179, + "66": 1.61701, + "67": 2.04351, + "68": 1.1398, + "69": 1.5478, + "70": 1.56188, + "71": 1.75339, + "72": 1.48392, + "73": 1.45709, + "74": 1.2907, + "75": 1.56681, + "76": 1.57327, + "77": 1.80419, + "78": 1.702, + "79": 2.1586, + "80": 2.01412, + "81": 1.8335, + "82": 2.12187, + "83": 1.75687, + "84": 1.46026, + "85": 1.79739, + "86": 1.70758, + "87": 2.05242, + "88": 2.05454, + "89": 1.65884, + "90": 1.61796, + "91": 1.78231, + "92": 2.27352, + "93": 1.49874, + "94": 1.47013, + "95": 1.40411, + "96": 1.93056, + "97": 1.51954, + "98": 1.63796, + "99": 2.08477, + "100": 1.35205 + } + } +} \ No newline at end of file diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_cross_entropy_loss_fusion_1node/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_cross_entropy_loss_fusion_1node/model_config.yaml new file mode 100644 index 00000000000..77789c90192 --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_cross_entropy_loss_fusion_1node/model_config.yaml @@ -0,0 +1,53 @@ +ENV_VARS: + CUDA_DEVICE_MAX_CONNECTIONS: 1 + NVTE_ALLOW_NONDETERMINISTIC_ALGO: 1 +MODEL_ARGS: + --num-layers: 12 + --hidden-size: 512 + --num-attention-heads: 8 + --log-params-norm: true + --log-num-zeros-in-grad: true + --log-validation-ppl-to-tensorboard: true + --log-timers-to-tensorboard: true + --tensorboard-dir: ${TENSORBOARD_PATH} + --micro-batch-size: 4 + --global-batch-size: 32 + --seq-length: 1024 + --max-position-embeddings: 1024 + --train-iters: 100 + --timing-log-level: 0 + --lr-decay-iters: 320000 + --save: ${CHECKPOINT_SAVE_PATH} + --load: ${CHECKPOINT_LOAD_PATH} + --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document + --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json + --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt + --split: 949,50,1 + --distributed-backend: nccl + --lr: 0.00015 + --lr-decay-style: cosine + --min-lr: 1.0e-5 + --weight-decay: 1e-2 + --clip-grad: 1.0 + --lr-warmup-fraction: .01 + --log-interval: 1 + --save-interval: 50 + --eval-interval: 1000 + --eval-iters: 10 + --transformer-impl: transformer_engine + --tensor-model-parallel-size: 2 + --pipeline-model-parallel-size: 2 + --cross-entropy-loss-fusion: true + --no-gradient-accumulation-fusion: true + --attention-softmax-in-fp32: true + --use-checkpoint-opt_param-scheduler: true + --use-mcore-models: true + --ckpt-format: torch_dist + --dist-ckpt-optim-fully-reshardable: true + --dist-ckpt-strictness: log_all # backward compatibility for TE changes + --data-cache-path: ${DATA_CACHE_PATH} + --bf16: true + --log-memory-to-tensorboard: true + --async-save: true + --use-persistent-ckpt-worker: true +TEST_TYPE: ckpt-resume diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_ddp_average_in_collective_1node/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_ddp_average_in_collective_1node/golden_values_dev_dgx_gb200.json new file mode 100644 index 00000000000..a21aff2a8af --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_ddp_average_in_collective_1node/golden_values_dev_dgx_gb200.json @@ -0,0 +1,537 @@ +{ + "lm loss": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 10.93086, + "2": 10.92833, + "3": 10.92866, + "4": 10.9288, + "5": 10.93088, + "6": 10.92771, + "7": 10.92336, + "8": 10.92159, + "9": 10.92031, + "10": 10.91964, + "11": 10.91027, + "12": 10.91814, + "13": 10.91523, + "14": 10.89825, + "15": 10.88042, + "16": 10.86841, + "17": 10.86882, + "18": 10.86258, + "19": 10.86185, + "20": 10.77348, + "21": 10.75516, + "22": 10.75521, + "23": 10.74508, + "24": 10.71106, + "25": 10.71082, + "26": 10.69968, + "27": 10.65192, + "28": 10.58249, + "29": 10.55636, + "30": 10.54863, + "31": 10.52499, + "32": 10.51761, + "33": 10.48144, + "34": 10.451, + "35": 10.44994, + "36": 10.43486, + "37": 10.39411, + "38": 10.39546, + "39": 10.36016, + "40": 10.35798, + "41": 10.32782, + "42": 10.30704, + "43": 10.28206, + "44": 10.25604, + "45": 10.27039, + "46": 10.23404, + "47": 10.21882, + "48": 10.17023, + "49": 10.1759, + "50": 10.17855, + "51": 10.19228, + "52": 10.13723, + "53": 10.13552, + "54": 10.10521, + "55": 10.08055, + "56": 10.11116, + "57": 10.09599, + "58": 10.10931, + "59": 10.05623, + "60": 10.07399, + "61": 10.02775, + "62": 10.0003, + "63": 10.07322, + "64": 10.03695, + "65": 10.00341, + "66": 10.03038, + "67": 10.0072, + "68": 9.97019, + "69": 9.9915, + "70": 9.97453, + "71": 10.00065, + "72": 9.97779, + "73": 9.97166, + "74": 9.95604, + "75": 9.93149, + "76": 9.96344, + "77": 9.95639, + "78": 9.90552, + "79": 9.91218, + "80": 9.92898, + "81": 9.95655, + "82": 9.89202, + "83": 9.85589, + "84": 9.78695, + "85": 9.78572, + "86": 9.88392, + "87": 9.91028, + "88": 9.88719, + "89": 9.81819, + "90": 9.81249, + "91": 9.82779, + "92": 9.81659, + "93": 9.74998, + "94": 9.8309, + "95": 9.82648, + "96": 9.80702, + "97": 9.74393, + "98": 9.77873, + "99": 9.82058, + "100": 9.71398 + } + }, + "num-zeros": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 1731.0, + "2": 1691.0, + "3": 1669.0, + "4": 1581.0, + "5": 1644.0, + "6": 1685.0, + "7": 1815.0, + "8": 1709.0, + "9": 1781.0, + "10": 1748.0, + "11": 1630.0, + "12": 1732.0, + "13": 1742.0, + "14": 1815.0, + "15": 1549.0, + "16": 1654.0, + "17": 1749.0, + "18": 1742.0, + "19": 1686.0, + "20": 1643.0, + "21": 1695.0, + "22": 1733.0, + "23": 1658.0, + "24": 1781.0, + "25": 1666.0, + "26": 1789.0, + "27": 1777.0, + "28": 1803.0, + "29": 1831.0, + "30": 1831.0, + "31": 2000.0, + "32": 1953.0, + "33": 1927.0, + "34": 2012.0, + "35": 2071.0, + "36": 1876.0, + "37": 2178.0, + "38": 2133.0, + "39": 2155.0, + "40": 2357.0, + "41": 2339.0, + "42": 1887.0, + "43": 2351.0, + "44": 2221.0, + "45": 2475.0, + "46": 2480.0, + "47": 2497.0, + "48": 2601.0, + "49": 2909.0, + "50": 2638.0, + "51": 2495.0, + "52": 2790.0, + "53": 2729.0, + "54": 2777.0, + "55": 2626.0, + "56": 2743.0, + "57": 2177.0, + "58": 3682.0, + "59": 2899.0, + "60": 2899.0, + "61": 2826.0, + "62": 3369.0, + "63": 3377.0, + "64": 3714.0, + "65": 2750.0, + "66": 3201.0, + "67": 3778.0, + "68": 3542.0, + "69": 3015.0, + "70": 3264.0, + "71": 3129.0, + "72": 3054.0, + "73": 3365.0, + "74": 3234.0, + "75": 3345.0, + "76": 3367.0, + "77": 3958.0, + "78": 3329.0, + "79": 3194.0, + "80": 2951.0, + "81": 3592.0, + "82": 2943.0, + "83": 3073.0, + "84": 3037.0, + "85": 2541.0, + "86": 3023.0, + "87": 2916.0, + "88": 3027.0, + "89": 3180.0, + "90": 3778.0, + "91": 3014.0, + "92": 2726.0, + "93": 3051.0, + "94": 3000.0, + "95": 3189.0, + "96": 3382.0, + "97": 3490.0, + "98": 3305.0, + "99": 3298.0, + "100": 3311.0 + } + }, + "mem-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 514359808.0, + "2": 514359808.0, + "3": 514359808.0, + "4": 514359808.0, + "5": 514359808.0, + "6": 514359808.0, + "7": 514359808.0, + "8": 514359808.0, + "9": 514359808.0, + "10": 514359808.0, + "11": 514359808.0, + "12": 514359808.0, + "13": 514359808.0, + "14": 514359808.0, + "15": 514359808.0, + "16": 514359808.0, + "17": 514359808.0, + "18": 514359808.0, + "19": 514359808.0, + "20": 514359808.0, + "21": 514359808.0, + "22": 514359808.0, + "23": 514359808.0, + "24": 514359808.0, + "25": 514359808.0, + "26": 514359808.0, + "27": 514359808.0, + "28": 514359808.0, + "29": 514359808.0, + "30": 514359808.0, + "31": 514359808.0, + "32": 514359808.0, + "33": 514359808.0, + "34": 514359808.0, + "35": 514359808.0, + "36": 514359808.0, + "37": 514359808.0, + "38": 514359808.0, + "39": 514359808.0, + "40": 514359808.0, + "41": 514359808.0, + "42": 514359808.0, + "43": 514359808.0, + "44": 514359808.0, + "45": 514359808.0, + "46": 514359808.0, + "47": 514359808.0, + "48": 514359808.0, + "49": 514359808.0, + "50": 514359808.0, + "51": 514359808.0, + "52": 514359808.0, + "53": 514359808.0, + "54": 514359808.0, + "55": 514359808.0, + "56": 514359808.0, + "57": 514359808.0, + "58": 514359808.0, + "59": 514359808.0, + "60": 514359808.0, + "61": 514359808.0, + "62": 514359808.0, + "63": 514359808.0, + "64": 514359808.0, + "65": 514359808.0, + "66": 514359808.0, + "67": 514359808.0, + "68": 514359808.0, + "69": 514359808.0, + "70": 514359808.0, + "71": 514359808.0, + "72": 514359808.0, + "73": 514359808.0, + "74": 514359808.0, + "75": 514359808.0, + "76": 514359808.0, + "77": 514359808.0, + "78": 514359808.0, + "79": 514359808.0, + "80": 514359808.0, + "81": 514359808.0, + "82": 514359808.0, + "83": 514359808.0, + "84": 514359808.0, + "85": 514359808.0, + "86": 514359808.0, + "87": 514359808.0, + "88": 514359808.0, + "89": 514359808.0, + "90": 514359808.0, + "91": 514359808.0, + "92": 514359808.0, + "93": 514359808.0, + "94": 514359808.0, + "95": 514359808.0, + "96": 514359808.0, + "97": 514359808.0, + "98": 514359808.0, + "99": 514359808.0, + "100": 514359808.0 + } + }, + "mem-max-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 1259110912.0, + "2": 1437086208.0, + "3": 1437086208.0, + "4": 1437086208.0, + "5": 1437086208.0, + "6": 1437086208.0, + "7": 1437086208.0, + "8": 1437086208.0, + "9": 1437086208.0, + "10": 1437086208.0, + "11": 1437086208.0, + "12": 1437086208.0, + "13": 1437086208.0, + "14": 1437086208.0, + "15": 1437086208.0, + "16": 1437086208.0, + "17": 1437086208.0, + "18": 1437086208.0, + "19": 1437086208.0, + "20": 1437086208.0, + "21": 1437086208.0, + "22": 1437086208.0, + "23": 1437086208.0, + "24": 1437086208.0, + "25": 1437086208.0, + "26": 1437086208.0, + "27": 1437086208.0, + "28": 1437086208.0, + "29": 1437086208.0, + "30": 1437086208.0, + "31": 1437086208.0, + "32": 1437086208.0, + "33": 1437086208.0, + "34": 1437086208.0, + "35": 1437086208.0, + "36": 1437086208.0, + "37": 1437086208.0, + "38": 1437086208.0, + "39": 1437086208.0, + "40": 1437086208.0, + "41": 1437086208.0, + "42": 1437086208.0, + "43": 1437086208.0, + "44": 1437086208.0, + "45": 1437086208.0, + "46": 1437086208.0, + "47": 1437086208.0, + "48": 1437086208.0, + "49": 1437086208.0, + "50": 1437086208.0, + "51": 1438131712.0, + "52": 1438131712.0, + "53": 1438131712.0, + "54": 1438131712.0, + "55": 1438131712.0, + "56": 1438131712.0, + "57": 1438131712.0, + "58": 1438131712.0, + "59": 1438131712.0, + "60": 1438131712.0, + "61": 1438131712.0, + "62": 1438131712.0, + "63": 1438131712.0, + "64": 1438131712.0, + "65": 1438131712.0, + "66": 1438131712.0, + "67": 1438131712.0, + "68": 1438131712.0, + "69": 1438131712.0, + "70": 1438131712.0, + "71": 1438131712.0, + "72": 1438131712.0, + "73": 1438131712.0, + "74": 1438131712.0, + "75": 1438131712.0, + "76": 1438131712.0, + "77": 1438131712.0, + "78": 1438131712.0, + "79": 1438131712.0, + "80": 1438131712.0, + "81": 1438131712.0, + "82": 1438131712.0, + "83": 1438131712.0, + "84": 1438131712.0, + "85": 1438131712.0, + "86": 1438131712.0, + "87": 1438131712.0, + "88": 1438131712.0, + "89": 1438131712.0, + "90": 1438131712.0, + "91": 1438131712.0, + "92": 1438131712.0, + "93": 1438131712.0, + "94": 1438131712.0, + "95": 1438131712.0, + "96": 1438131712.0, + "97": 1438131712.0, + "98": 1438131712.0, + "99": 1438131712.0, + "100": 1438131712.0 + } + }, + "iteration-time": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": "nan", + "2": 4.81473, + "3": 2.09453, + "4": 1.86984, + "5": 1.56676, + "6": 1.80992, + "7": 1.7198, + "8": 2.12982, + "9": 2.28507, + "10": 1.61307, + "11": 1.51036, + "12": 1.60587, + "13": 1.77033, + "14": 1.85991, + "15": 1.66399, + "16": 1.65099, + "17": 2.28711, + "18": 1.75458, + "19": 1.87636, + "20": 1.64474, + "21": 1.96717, + "22": 1.48716, + "23": 1.96156, + "24": 1.85233, + "25": 1.80976, + "26": 2.05999, + "27": 1.73756, + "28": 1.67494, + "29": 1.36873, + "30": 1.58467, + "31": 1.46416, + "32": 2.34186, + "33": 1.57787, + "34": 1.39533, + "35": 2.15177, + "36": 2.08982, + "37": 1.79399, + "38": 1.37801, + "39": 2.39593, + "40": 1.97004, + "41": 1.88566, + "42": 2.43752, + "43": 1.58207, + "44": 1.69065, + "45": 2.24127, + "46": 1.75904, + "47": 1.94181, + "48": 1.98932, + "49": 1.72371, + "50": 1.76697, + "51": 1.40392, + "52": 1.78108, + "53": 1.82488, + "54": 1.89291, + "55": 1.6651, + "56": 2.52069, + "57": 1.35678, + "58": 1.94647, + "59": 1.61547, + "60": 1.52417, + "61": 2.07485, + "62": 2.18871, + "63": 1.43064, + "64": 1.76189, + "65": 1.4102, + "66": 1.80604, + "67": 1.88948, + "68": 1.29699, + "69": 1.76349, + "70": 1.46986, + "71": 1.91983, + "72": 1.64698, + "73": 1.78924, + "74": 1.5544, + "75": 1.73358, + "76": 2.02105, + "77": 1.54779, + "78": 1.72857, + "79": 1.96934, + "80": 1.85736, + "81": 2.01788, + "82": 2.10514, + "83": 1.91832, + "84": 1.56669, + "85": 1.67457, + "86": 1.6547, + "87": 2.15931, + "88": 1.83824, + "89": 1.92897, + "90": 1.64696, + "91": 1.99756, + "92": 2.10698, + "93": 1.47651, + "94": 1.78543, + "95": 1.95868, + "96": 1.83602, + "97": 1.57637, + "98": 1.88355, + "99": 2.32003, + "100": 1.43637 + } + } +} \ No newline at end of file diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_ddp_average_in_collective_1node/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_ddp_average_in_collective_1node/model_config.yaml new file mode 100644 index 00000000000..aacc077e937 --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_ddp_average_in_collective_1node/model_config.yaml @@ -0,0 +1,55 @@ +ENV_VARS: + CUDA_DEVICE_MAX_CONNECTIONS: 1 + NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0 + NCCL_ALGO: Ring + CUBLAS_WORKSPACE_CONFIG: :4096:8 +MODEL_ARGS: + --num-layers: 12 + --hidden-size: 512 + --num-attention-heads: 8 + --log-params-norm: true + --log-num-zeros-in-grad: true + --log-validation-ppl-to-tensorboard: true + --log-timers-to-tensorboard: true + --tensorboard-dir: ${TENSORBOARD_PATH} + --micro-batch-size: 4 + --global-batch-size: 32 + --seq-length: 1024 + --max-position-embeddings: 1024 + --train-iters: 100 + --timing-log-level: 0 + --lr-decay-iters: 320000 + --save: ${CHECKPOINT_SAVE_PATH} + --load: ${CHECKPOINT_LOAD_PATH} + --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document + --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json + --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt + --split: 949,50,1 + --distributed-backend: nccl + --lr: 0.00015 + --lr-decay-style: cosine + --min-lr: 1.0e-5 + --weight-decay: 1e-2 + --clip-grad: 1.0 + --lr-warmup-fraction: .01 + --log-interval: 1 + --save-interval: 50 + --eval-interval: 1000 + --eval-iters: 10 + --transformer-impl: transformer_engine + --tensor-model-parallel-size: 2 + --pipeline-model-parallel-size: 2 + --ddp-average-in-collective: true + --deterministic-mode: true + --no-gradient-accumulation-fusion: true + --attention-softmax-in-fp32: true + --use-checkpoint-opt_param-scheduler: true + --use-mcore-models: true + --ckpt-format: torch_dist + --dist-ckpt-strictness: log_all # backward compatibility for TE changes + --data-cache-path: ${DATA_CACHE_PATH} + --bf16: true + --log-memory-to-tensorboard: true + --async-save: true + --use-persistent-ckpt-worker: true +TEST_TYPE: ckpt-resume diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_defer_embedding_wgrad_compute_1node/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_defer_embedding_wgrad_compute_1node/golden_values_dev_dgx_gb200.json new file mode 100644 index 00000000000..746b31c3f88 --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_defer_embedding_wgrad_compute_1node/golden_values_dev_dgx_gb200.json @@ -0,0 +1,537 @@ +{ + "lm loss": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 10.93086, + "2": 10.92833, + "3": 10.92867, + "4": 10.92885, + "5": 10.93089, + "6": 10.92768, + "7": 10.92339, + "8": 10.92161, + "9": 10.92031, + "10": 10.91963, + "11": 10.91026, + "12": 10.91814, + "13": 10.91528, + "14": 10.89825, + "15": 10.88044, + "16": 10.86845, + "17": 10.86882, + "18": 10.86255, + "19": 10.86184, + "20": 10.77349, + "21": 10.75515, + "22": 10.75525, + "23": 10.7451, + "24": 10.71105, + "25": 10.71078, + "26": 10.69967, + "27": 10.65193, + "28": 10.5825, + "29": 10.55638, + "30": 10.54863, + "31": 10.52498, + "32": 10.51762, + "33": 10.48146, + "34": 10.45103, + "35": 10.44996, + "36": 10.43489, + "37": 10.39411, + "38": 10.39543, + "39": 10.36014, + "40": 10.35802, + "41": 10.32785, + "42": 10.307, + "43": 10.28209, + "44": 10.25605, + "45": 10.27043, + "46": 10.23405, + "47": 10.21883, + "48": 10.17023, + "49": 10.17586, + "50": 10.17856, + "51": 10.19226, + "52": 10.13722, + "53": 10.13551, + "54": 10.10522, + "55": 10.08051, + "56": 10.11117, + "57": 10.09597, + "58": 10.10933, + "59": 10.05625, + "60": 10.07399, + "61": 10.02772, + "62": 10.00029, + "63": 10.07319, + "64": 10.0369, + "65": 10.00343, + "66": 10.03036, + "67": 10.00721, + "68": 9.97016, + "69": 9.99154, + "70": 9.97454, + "71": 10.00064, + "72": 9.97777, + "73": 9.97168, + "74": 9.956, + "75": 9.93149, + "76": 9.96344, + "77": 9.95634, + "78": 9.90551, + "79": 9.91217, + "80": 9.92898, + "81": 9.95658, + "82": 9.89201, + "83": 9.85591, + "84": 9.78695, + "85": 9.78569, + "86": 9.88392, + "87": 9.91031, + "88": 9.88719, + "89": 9.81821, + "90": 9.81249, + "91": 9.82781, + "92": 9.81661, + "93": 9.74996, + "94": 9.83091, + "95": 9.82648, + "96": 9.80705, + "97": 9.7439, + "98": 9.77871, + "99": 9.82058, + "100": 9.71401 + } + }, + "num-zeros": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 76.0, + "2": 63.0, + "3": 65.0, + "4": 73.0, + "5": 70.0, + "6": 60.0, + "7": 65.0, + "8": 61.0, + "9": 60.0, + "10": 68.0, + "11": 69.0, + "12": 55.0, + "13": 72.0, + "14": 74.0, + "15": 52.0, + "16": 66.0, + "17": 56.0, + "18": 62.0, + "19": 57.0, + "20": 72.0, + "21": 60.0, + "22": 60.0, + "23": 57.0, + "24": 72.0, + "25": 71.0, + "26": 77.0, + "27": 66.0, + "28": 68.0, + "29": 63.0, + "30": 68.0, + "31": 80.0, + "32": 75.0, + "33": 60.0, + "34": 71.0, + "35": 74.0, + "36": 76.0, + "37": 70.0, + "38": 64.0, + "39": 76.0, + "40": 78.0, + "41": 72.0, + "42": 81.0, + "43": 73.0, + "44": 87.0, + "45": 94.0, + "46": 59.0, + "47": 76.0, + "48": 82.0, + "49": 75.0, + "50": 80.0, + "51": 74.0, + "52": 76.0, + "53": 76.0, + "54": 68.0, + "55": 58.0, + "56": 68.0, + "57": 73.0, + "58": 104.0, + "59": 94.0, + "60": 84.0, + "61": 85.0, + "62": 77.0, + "63": 71.0, + "64": 92.0, + "65": 78.0, + "66": 97.0, + "67": 89.0, + "68": 82.0, + "69": 98.0, + "70": 82.0, + "71": 96.0, + "72": 87.0, + "73": 94.0, + "74": 80.0, + "75": 80.0, + "76": 69.0, + "77": 100.0, + "78": 88.0, + "79": 90.0, + "80": 70.0, + "81": 102.0, + "82": 60.0, + "83": 93.0, + "84": 85.0, + "85": 69.0, + "86": 81.0, + "87": 75.0, + "88": 80.0, + "89": 84.0, + "90": 81.0, + "91": 74.0, + "92": 62.0, + "93": 65.0, + "94": 81.0, + "95": 91.0, + "96": 83.0, + "97": 88.0, + "98": 66.0, + "99": 72.0, + "100": 91.0 + } + }, + "mem-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 542802432.0, + "2": 542802432.0, + "3": 542802432.0, + "4": 542802432.0, + "5": 542802432.0, + "6": 542802432.0, + "7": 542802432.0, + "8": 542802432.0, + "9": 542802432.0, + "10": 542802432.0, + "11": 542802432.0, + "12": 542802432.0, + "13": 542802432.0, + "14": 542802432.0, + "15": 542802432.0, + "16": 542802432.0, + "17": 542802432.0, + "18": 542802432.0, + "19": 542802432.0, + "20": 542802432.0, + "21": 542802432.0, + "22": 542802432.0, + "23": 542802432.0, + "24": 542802432.0, + "25": 542802432.0, + "26": 542802432.0, + "27": 542802432.0, + "28": 542802432.0, + "29": 542802432.0, + "30": 542802432.0, + "31": 542802432.0, + "32": 542802432.0, + "33": 542802432.0, + "34": 542802432.0, + "35": 542802432.0, + "36": 542802432.0, + "37": 542802432.0, + "38": 542802432.0, + "39": 542802432.0, + "40": 542802432.0, + "41": 542802432.0, + "42": 542802432.0, + "43": 542802432.0, + "44": 542802432.0, + "45": 542802432.0, + "46": 542802432.0, + "47": 542802432.0, + "48": 542802432.0, + "49": 542802432.0, + "50": 542802432.0, + "51": 542802432.0, + "52": 542802432.0, + "53": 542802432.0, + "54": 542802432.0, + "55": 542802432.0, + "56": 542802432.0, + "57": 542802432.0, + "58": 542802432.0, + "59": 542802432.0, + "60": 542802432.0, + "61": 542802432.0, + "62": 542802432.0, + "63": 542802432.0, + "64": 542802432.0, + "65": 542802432.0, + "66": 542802432.0, + "67": 542802432.0, + "68": 542802432.0, + "69": 542802432.0, + "70": 542802432.0, + "71": 542802432.0, + "72": 542802432.0, + "73": 542802432.0, + "74": 542802432.0, + "75": 542802432.0, + "76": 542802432.0, + "77": 542802432.0, + "78": 542802432.0, + "79": 542802432.0, + "80": 542802432.0, + "81": 542802432.0, + "82": 542802432.0, + "83": 542802432.0, + "84": 542802432.0, + "85": 542802432.0, + "86": 542802432.0, + "87": 542802432.0, + "88": 542802432.0, + "89": 542802432.0, + "90": 542802432.0, + "91": 542802432.0, + "92": 542802432.0, + "93": 542802432.0, + "94": 542802432.0, + "95": 542802432.0, + "96": 542802432.0, + "97": 542802432.0, + "98": 542802432.0, + "99": 542802432.0, + "100": 542802432.0 + } + }, + "mem-max-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 1726382592.0, + "2": 1905144320.0, + "3": 1905144320.0, + "4": 1905144320.0, + "5": 1905144320.0, + "6": 1905144320.0, + "7": 1905144320.0, + "8": 1905144320.0, + "9": 1905144320.0, + "10": 1905144320.0, + "11": 1905144320.0, + "12": 1905144320.0, + "13": 1905144320.0, + "14": 1905144320.0, + "15": 1905144320.0, + "16": 1905144320.0, + "17": 1905144320.0, + "18": 1905144320.0, + "19": 1905144320.0, + "20": 1905144320.0, + "21": 1905144320.0, + "22": 1905144320.0, + "23": 1905144320.0, + "24": 1905144320.0, + "25": 1905144320.0, + "26": 1905144320.0, + "27": 1905144320.0, + "28": 1905144320.0, + "29": 1905144320.0, + "30": 1905144320.0, + "31": 1905144320.0, + "32": 1905144320.0, + "33": 1905144320.0, + "34": 1905144320.0, + "35": 1905144320.0, + "36": 1905144320.0, + "37": 1905144320.0, + "38": 1905144320.0, + "39": 1905144320.0, + "40": 1905144320.0, + "41": 1905144320.0, + "42": 1905144320.0, + "43": 1905144320.0, + "44": 1905144320.0, + "45": 1905144320.0, + "46": 1905144320.0, + "47": 1905144320.0, + "48": 1905144320.0, + "49": 1905144320.0, + "50": 1905144320.0, + "51": 1905930752.0, + "52": 1905930752.0, + "53": 1905930752.0, + "54": 1905930752.0, + "55": 1905930752.0, + "56": 1905930752.0, + "57": 1905930752.0, + "58": 1905930752.0, + "59": 1905930752.0, + "60": 1905930752.0, + "61": 1905930752.0, + "62": 1905930752.0, + "63": 1905930752.0, + "64": 1905930752.0, + "65": 1905930752.0, + "66": 1905930752.0, + "67": 1905930752.0, + "68": 1905930752.0, + "69": 1905930752.0, + "70": 1905930752.0, + "71": 1905930752.0, + "72": 1905930752.0, + "73": 1905930752.0, + "74": 1905930752.0, + "75": 1905930752.0, + "76": 1905930752.0, + "77": 1905930752.0, + "78": 1905930752.0, + "79": 1905930752.0, + "80": 1905930752.0, + "81": 1905930752.0, + "82": 1905930752.0, + "83": 1905930752.0, + "84": 1905930752.0, + "85": 1905930752.0, + "86": 1905930752.0, + "87": 1905930752.0, + "88": 1905930752.0, + "89": 1905930752.0, + "90": 1905930752.0, + "91": 1905930752.0, + "92": 1905930752.0, + "93": 1905930752.0, + "94": 1905930752.0, + "95": 1905930752.0, + "96": 1905930752.0, + "97": 1905930752.0, + "98": 1905930752.0, + "99": 1905930752.0, + "100": 1905930752.0 + } + }, + "iteration-time": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": "nan", + "2": 4.77093, + "3": 2.00972, + "4": 1.995, + "5": 1.56147, + "6": 1.60153, + "7": 1.37722, + "8": 2.20244, + "9": 2.24495, + "10": 1.86104, + "11": 2.06265, + "12": 1.68463, + "13": 1.82427, + "14": 1.88402, + "15": 1.60829, + "16": 1.77207, + "17": 2.00294, + "18": 1.86099, + "19": 1.91095, + "20": 1.7563, + "21": 1.72699, + "22": 1.62013, + "23": 2.0429, + "24": 1.46404, + "25": 2.08873, + "26": 2.31405, + "27": 1.45356, + "28": 1.74206, + "29": 1.40044, + "30": 1.63679, + "31": 1.5254, + "32": 2.45261, + "33": 1.48425, + "34": 1.614, + "35": 2.01304, + "36": 1.93212, + "37": 1.83303, + "38": 1.23902, + "39": 1.93774, + "40": 1.74239, + "41": 1.8622, + "42": 2.04557, + "43": 1.82174, + "44": 1.80106, + "45": 2.07996, + "46": 1.5475, + "47": 1.66734, + "48": 1.93365, + "49": 1.38407, + "50": 1.65545, + "51": 1.14169, + "52": 1.72797, + "53": 1.75077, + "54": 1.91392, + "55": 1.92503, + "56": 3.48234, + "57": 1.25329, + "58": 2.11296, + "59": 1.72368, + "60": 1.69621, + "61": 2.30612, + "62": 2.10489, + "63": 1.60869, + "64": 2.03916, + "65": 1.75772, + "66": 1.83987, + "67": 1.93896, + "68": 1.31509, + "69": 1.74073, + "70": 1.72843, + "71": 2.16538, + "72": 1.6367, + "73": 1.7422, + "74": 1.5244, + "75": 1.67888, + "76": 1.82499, + "77": 1.71431, + "78": 1.79197, + "79": 2.04495, + "80": 1.83611, + "81": 2.0013, + "82": 1.89405, + "83": 1.90476, + "84": 1.43999, + "85": 1.73582, + "86": 1.45086, + "87": 2.16513, + "88": 1.82369, + "89": 1.96507, + "90": 1.65476, + "91": 1.67164, + "92": 2.12668, + "93": 1.57701, + "94": 1.7392, + "95": 2.0081, + "96": 1.8941, + "97": 1.71479, + "98": 1.82127, + "99": 2.35655, + "100": 1.65236 + } + } +} \ No newline at end of file diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_defer_embedding_wgrad_compute_1node/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_defer_embedding_wgrad_compute_1node/model_config.yaml new file mode 100644 index 00000000000..f600f587ae9 --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_defer_embedding_wgrad_compute_1node/model_config.yaml @@ -0,0 +1,55 @@ +ENV_VARS: + CUDA_DEVICE_MAX_CONNECTIONS: 1 + NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0 + NCCL_ALGO: Ring + CUBLAS_WORKSPACE_CONFIG: :4096:8 +MODEL_ARGS: + --num-layers: 12 + --hidden-size: 512 + --num-attention-heads: 8 + --log-params-norm: true + --log-num-zeros-in-grad: true + --log-validation-ppl-to-tensorboard: true + --log-timers-to-tensorboard: true + --tensorboard-dir: ${TENSORBOARD_PATH} + --micro-batch-size: 4 + --global-batch-size: 32 + --seq-length: 1024 + --max-position-embeddings: 1024 + --train-iters: 100 + --timing-log-level: 0 + --lr-decay-iters: 320000 + --save: ${CHECKPOINT_SAVE_PATH} + --load: ${CHECKPOINT_LOAD_PATH} + --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document + --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json + --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt + --split: 949,50,1 + --distributed-backend: nccl + --lr: 0.00015 + --lr-decay-style: cosine + --min-lr: 1.0e-5 + --weight-decay: 1e-2 + --clip-grad: 1.0 + --lr-warmup-fraction: .01 + --log-interval: 1 + --save-interval: 50 + --eval-interval: 1000 + --eval-iters: 10 + --transformer-impl: transformer_engine + --tensor-model-parallel-size: 2 + --pipeline-model-parallel-size: 2 + --defer-embedding-wgrad-compute: true + --wgrad-deferral-limit: 2 + --deterministic-mode: true + --attention-softmax-in-fp32: true + --use-checkpoint-opt_param-scheduler: true + --use-mcore-models: true + --ckpt-format: torch_dist + --dist-ckpt-strictness: log_all # backward compatibility for TE changes + --data-cache-path: ${DATA_CACHE_PATH} + --bf16: true + --log-memory-to-tensorboard: true + --async-save: true + --use-persistent-ckpt-worker: true +TEST_TYPE: ckpt-resume diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_reshard_1x4xNone_1node/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_reshard_1x4xNone_1node/golden_values_dev_dgx_gb200.json new file mode 100644 index 00000000000..35e03d6e4ca --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_reshard_1x4xNone_1node/golden_values_dev_dgx_gb200.json @@ -0,0 +1,537 @@ +{ + "lm loss": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 10.93259, + "2": 10.92912, + "3": 10.92982, + "4": 10.92995, + "5": 10.93097, + "6": 10.92838, + "7": 10.92478, + "8": 10.92268, + "9": 10.92099, + "10": 10.92135, + "11": 10.91168, + "12": 10.91921, + "13": 10.91613, + "14": 10.90003, + "15": 10.88228, + "16": 10.87, + "17": 10.87031, + "18": 10.86546, + "19": 10.86381, + "20": 10.77638, + "21": 10.75919, + "22": 10.75924, + "23": 10.74898, + "24": 10.71536, + "25": 10.71528, + "26": 10.70432, + "27": 10.6569, + "28": 10.58674, + "29": 10.56055, + "30": 10.55329, + "31": 10.52938, + "32": 10.52186, + "33": 10.486, + "34": 10.45516, + "35": 10.45396, + "36": 10.43956, + "37": 10.39902, + "38": 10.39954, + "39": 10.36402, + "40": 10.36152, + "41": 10.33166, + "42": 10.31049, + "43": 10.286, + "44": 10.25962, + "45": 10.27364, + "46": 10.23729, + "47": 10.22227, + "48": 10.17322, + "49": 10.17861, + "50": 10.18086, + "51": 10.19463, + "52": 10.1396, + "53": 10.13783, + "54": 10.10765, + "55": 10.08249, + "56": 10.11303, + "57": 10.09824, + "58": 10.11106, + "59": 10.05788, + "60": 10.07577, + "61": 10.0291, + "62": 10.00202, + "63": 10.07436, + "64": 10.03837, + "65": 10.00521, + "66": 10.03156, + "67": 10.00863, + "68": 9.97179, + "69": 9.99315, + "70": 9.97637, + "71": 10.00193, + "72": 9.97998, + "73": 9.97431, + "74": 9.95858, + "75": 9.93426, + "76": 9.96552, + "77": 9.95871, + "78": 9.90836, + "79": 9.91528, + "80": 9.93262, + "81": 9.9599, + "82": 9.89402, + "83": 9.85935, + "84": 9.7893, + "85": 9.78862, + "86": 9.88605, + "87": 9.91131, + "88": 9.88882, + "89": 9.8186, + "90": 9.81369, + "91": 9.82942, + "92": 9.81731, + "93": 9.75166, + "94": 9.8324, + "95": 9.82704, + "96": 9.80835, + "97": 9.74564, + "98": 9.77929, + "99": 9.82101, + "100": 9.71413 + } + }, + "num-zeros": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 1842.0, + "2": 1666.0, + "3": 1669.0, + "4": 1676.0, + "5": 1673.0, + "6": 1667.0, + "7": 1854.0, + "8": 1735.0, + "9": 1819.0, + "10": 1705.0, + "11": 1633.0, + "12": 1661.0, + "13": 1678.0, + "14": 1916.0, + "15": 1516.0, + "16": 1592.0, + "17": 1731.0, + "18": 1740.0, + "19": 1689.0, + "20": 1663.0, + "21": 1656.0, + "22": 1702.0, + "23": 1605.0, + "24": 1683.0, + "25": 1600.0, + "26": 1796.0, + "27": 1869.0, + "28": 1799.0, + "29": 1863.0, + "30": 1751.0, + "31": 1996.0, + "32": 1822.0, + "33": 1989.0, + "34": 1969.0, + "35": 2071.0, + "36": 1947.0, + "37": 2205.0, + "38": 2039.0, + "39": 2152.0, + "40": 2192.0, + "41": 2398.0, + "42": 1894.0, + "43": 2416.0, + "44": 2212.0, + "45": 2580.0, + "46": 2401.0, + "47": 2548.0, + "48": 2740.0, + "49": 2877.0, + "50": 2585.0, + "51": 2463.0, + "52": 2762.0, + "53": 2695.0, + "54": 2726.0, + "55": 2696.0, + "56": 2703.0, + "57": 2237.0, + "58": 3590.0, + "59": 2940.0, + "60": 2897.0, + "61": 2865.0, + "62": 3261.0, + "63": 3257.0, + "64": 3719.0, + "65": 2634.0, + "66": 3076.0, + "67": 3829.0, + "68": 3363.0, + "69": 3014.0, + "70": 3346.0, + "71": 3257.0, + "72": 2987.0, + "73": 3508.0, + "74": 3297.0, + "75": 3466.0, + "76": 3269.0, + "77": 3832.0, + "78": 3423.0, + "79": 3221.0, + "80": 3109.0, + "81": 3729.0, + "82": 2907.0, + "83": 3084.0, + "84": 2989.0, + "85": 2642.0, + "86": 3014.0, + "87": 2964.0, + "88": 2977.0, + "89": 3130.0, + "90": 3602.0, + "91": 2839.0, + "92": 2789.0, + "93": 3114.0, + "94": 2950.0, + "95": 3291.0, + "96": 3283.0, + "97": 3528.0, + "98": 3282.0, + "99": 3348.0, + "100": 3234.0 + } + }, + "mem-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 464552448.0, + "2": 464552448.0, + "3": 464552448.0, + "4": 464552448.0, + "5": 464552448.0, + "6": 464552448.0, + "7": 464552448.0, + "8": 464552448.0, + "9": 464552448.0, + "10": 464552448.0, + "11": 464552448.0, + "12": 464552448.0, + "13": 464552448.0, + "14": 464552448.0, + "15": 464552448.0, + "16": 464552448.0, + "17": 464552448.0, + "18": 464552448.0, + "19": 464552448.0, + "20": 464552448.0, + "21": 464552448.0, + "22": 464552448.0, + "23": 464552448.0, + "24": 464552448.0, + "25": 464552448.0, + "26": 464552448.0, + "27": 464552448.0, + "28": 464552448.0, + "29": 464552448.0, + "30": 464552448.0, + "31": 464552448.0, + "32": 464552448.0, + "33": 464552448.0, + "34": 464552448.0, + "35": 464552448.0, + "36": 464552448.0, + "37": 464552448.0, + "38": 464552448.0, + "39": 464552448.0, + "40": 464552448.0, + "41": 464552448.0, + "42": 464552448.0, + "43": 464552448.0, + "44": 464552448.0, + "45": 464552448.0, + "46": 464552448.0, + "47": 464552448.0, + "48": 464552448.0, + "49": 464552448.0, + "50": 464552448.0, + "51": 464552448.0, + "52": 464552448.0, + "53": 464552448.0, + "54": 464552448.0, + "55": 464552448.0, + "56": 464552448.0, + "57": 464552448.0, + "58": 464552448.0, + "59": 464552448.0, + "60": 464552448.0, + "61": 464552448.0, + "62": 464552448.0, + "63": 464552448.0, + "64": 464552448.0, + "65": 464552448.0, + "66": 464552448.0, + "67": 464552448.0, + "68": 464552448.0, + "69": 464552448.0, + "70": 464552448.0, + "71": 464552448.0, + "72": 464552448.0, + "73": 464552448.0, + "74": 464552448.0, + "75": 464552448.0, + "76": 464552448.0, + "77": 464552448.0, + "78": 464552448.0, + "79": 464552448.0, + "80": 464552448.0, + "81": 464552448.0, + "82": 464552448.0, + "83": 464552448.0, + "84": 464552448.0, + "85": 464552448.0, + "86": 464552448.0, + "87": 464552448.0, + "88": 464552448.0, + "89": 464552448.0, + "90": 464552448.0, + "91": 464552448.0, + "92": 464552448.0, + "93": 464552448.0, + "94": 464552448.0, + "95": 464552448.0, + "96": 464552448.0, + "97": 464552448.0, + "98": 464552448.0, + "99": 464552448.0, + "100": 464552448.0 + } + }, + "mem-max-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 1196458496.0, + "2": 1374695936.0, + "3": 1374695936.0, + "4": 1374695936.0, + "5": 1374695936.0, + "6": 1374695936.0, + "7": 1374695936.0, + "8": 1374695936.0, + "9": 1374695936.0, + "10": 1374695936.0, + "11": 1374695936.0, + "12": 1374695936.0, + "13": 1374695936.0, + "14": 1374695936.0, + "15": 1374695936.0, + "16": 1374695936.0, + "17": 1374695936.0, + "18": 1374695936.0, + "19": 1374695936.0, + "20": 1374695936.0, + "21": 1374695936.0, + "22": 1374695936.0, + "23": 1374695936.0, + "24": 1374695936.0, + "25": 1374695936.0, + "26": 1374695936.0, + "27": 1374695936.0, + "28": 1374695936.0, + "29": 1374695936.0, + "30": 1374695936.0, + "31": 1374695936.0, + "32": 1374695936.0, + "33": 1374695936.0, + "34": 1374695936.0, + "35": 1374695936.0, + "36": 1374695936.0, + "37": 1374695936.0, + "38": 1374695936.0, + "39": 1374695936.0, + "40": 1374695936.0, + "41": 1374695936.0, + "42": 1374695936.0, + "43": 1374695936.0, + "44": 1374695936.0, + "45": 1374695936.0, + "46": 1374695936.0, + "47": 1374695936.0, + "48": 1374695936.0, + "49": 1374695936.0, + "50": 1374695936.0, + "51": 1375744512.0, + "52": 1375744512.0, + "53": 1375744512.0, + "54": 1375744512.0, + "55": 1375744512.0, + "56": 1375744512.0, + "57": 1375744512.0, + "58": 1375744512.0, + "59": 1375744512.0, + "60": 1375744512.0, + "61": 1375744512.0, + "62": 1375744512.0, + "63": 1375744512.0, + "64": 1375744512.0, + "65": 1375744512.0, + "66": 1375744512.0, + "67": 1375744512.0, + "68": 1375744512.0, + "69": 1375744512.0, + "70": 1375744512.0, + "71": 1375744512.0, + "72": 1375744512.0, + "73": 1375744512.0, + "74": 1375744512.0, + "75": 1375744512.0, + "76": 1375744512.0, + "77": 1375744512.0, + "78": 1375744512.0, + "79": 1375744512.0, + "80": 1375744512.0, + "81": 1375744512.0, + "82": 1375744512.0, + "83": 1375744512.0, + "84": 1375744512.0, + "85": 1375744512.0, + "86": 1375744512.0, + "87": 1375744512.0, + "88": 1375744512.0, + "89": 1375744512.0, + "90": 1375744512.0, + "91": 1375744512.0, + "92": 1375744512.0, + "93": 1375744512.0, + "94": 1375744512.0, + "95": 1375744512.0, + "96": 1375744512.0, + "97": 1375744512.0, + "98": 1375744512.0, + "99": 1375744512.0, + "100": 1375744512.0 + } + }, + "iteration-time": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": "nan", + "2": 5.43178, + "3": 2.11204, + "4": 1.57881, + "5": 1.25677, + "6": 1.66575, + "7": 1.1159, + "8": 1.76283, + "9": 1.84403, + "10": 1.86994, + "11": 1.94765, + "12": 1.73758, + "13": 2.3284, + "14": 1.98527, + "15": 1.83671, + "16": 1.97567, + "17": 2.452, + "18": 1.57899, + "19": 1.74843, + "20": 1.43465, + "21": 2.2229, + "22": 1.66071, + "23": 2.09685, + "24": 1.75803, + "25": 2.23128, + "26": 2.53191, + "27": 2.01473, + "28": 1.80989, + "29": 1.59971, + "30": 1.89057, + "31": 1.67378, + "32": 2.6672, + "33": 1.46266, + "34": 1.36834, + "35": 1.8787, + "36": 2.33778, + "37": 1.78251, + "38": 1.26104, + "39": 2.24518, + "40": 1.90219, + "41": 2.14212, + "42": 2.70056, + "43": 1.82774, + "44": 2.01226, + "45": 2.40442, + "46": 1.26065, + "47": 1.75868, + "48": 1.92171, + "49": 1.47451, + "50": 1.74815, + "51": 1.28009, + "52": 2.0084, + "53": 1.72222, + "54": 1.885, + "55": 1.78412, + "56": 2.72441, + "57": 1.54464, + "58": 2.01158, + "59": 1.86486, + "60": 1.66307, + "61": 2.37798, + "62": 2.06309, + "63": 1.66317, + "64": 1.70626, + "65": 1.6454, + "66": 1.73807, + "67": 1.50109, + "68": 1.45802, + "69": 1.95859, + "70": 1.86557, + "71": 2.6301, + "72": 2.77939, + "73": 2.10753, + "74": 1.80629, + "75": 1.91951, + "76": 2.14099, + "77": 1.82286, + "78": 1.70552, + "79": 1.79363, + "80": 1.78159, + "81": 2.04362, + "82": 2.296, + "83": 2.02433, + "84": 1.76213, + "85": 2.19422, + "86": 1.79144, + "87": 2.14923, + "88": 2.13274, + "89": 2.17052, + "90": 1.78888, + "91": 2.1383, + "92": 2.24417, + "93": 1.42999, + "94": 2.11636, + "95": 2.25751, + "96": 1.98919, + "97": 1.41422, + "98": 1.91417, + "99": 2.61613, + "100": 1.60046 + } + } +} \ No newline at end of file diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_reshard_1x4xNone_1node/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_reshard_1x4xNone_1node/model_config.yaml new file mode 100644 index 00000000000..a413b7ebb0c --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_pp2_resume_torch_dist_reshard_1x4xNone_1node/model_config.yaml @@ -0,0 +1,52 @@ +ENV_VARS: + CUDA_DEVICE_MAX_CONNECTIONS: 1 + NVTE_ALLOW_NONDETERMINISTIC_ALGO: 1 +MODEL_ARGS: + --num-layers: 12 + --hidden-size: 512 + --num-attention-heads: 8 + --log-params-norm: true + --log-num-zeros-in-grad: true + --log-validation-ppl-to-tensorboard: true + --log-timers-to-tensorboard: true + --tensorboard-dir: ${TENSORBOARD_PATH} + --micro-batch-size: 4 + --global-batch-size: 32 + --seq-length: 1024 + --max-position-embeddings: 1024 + --train-iters: 100 + --timing-log-level: 0 + --lr-decay-iters: 320000 + --save: ${CHECKPOINT_SAVE_PATH} + --load: ${CHECKPOINT_LOAD_PATH} + --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document + --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json + --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt + --split: 949,50,1 + --distributed-backend: nccl + --lr: 0.00015 + --lr-decay-style: cosine + --min-lr: 1.0e-5 + --weight-decay: 1e-2 + --clip-grad: 1.0 + --lr-warmup-fraction: .01 + --log-interval: 1 + --save-interval: 50 + --eval-interval: 1000 + --eval-iters: 10 + --transformer-impl: transformer_engine + --tensor-model-parallel-size: 2 + --pipeline-model-parallel-size: 2 + --no-gradient-accumulation-fusion: true + --attention-softmax-in-fp32: true + --use-checkpoint-opt_param-scheduler: true + --use-mcore-models: true + --ckpt-format: torch_dist + --dist-ckpt-optim-fully-reshardable: true + --dist-ckpt-strictness: log_all # backward compatibility for TE changes + --data-cache-path: ${DATA_CACHE_PATH} + --bf16: true + --log-memory-to-tensorboard: true + --async-save: true + --use-persistent-ckpt-worker: true +TEST_TYPE: ckpt-resume diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_zp_z3_resume_fsdp_dtensor_1node/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_zp_z3_resume_fsdp_dtensor_1node/golden_values_dev_dgx_gb200.json new file mode 100644 index 00000000000..4a1cdc7dcfd --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_zp_z3_resume_fsdp_dtensor_1node/golden_values_dev_dgx_gb200.json @@ -0,0 +1,537 @@ +{ + "lm loss": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 10.95307, + "2": 10.94856, + "3": 10.95609, + "4": 10.94434, + "5": 10.94637, + "6": 10.94557, + "7": 10.95292, + "8": 10.93872, + "9": 10.94966, + "10": 10.93736, + "11": 10.93794, + "12": 10.93034, + "13": 10.92636, + "14": 10.92328, + "15": 10.90361, + "16": 10.89261, + "17": 10.90125, + "18": 10.88605, + "19": 10.88664, + "20": 10.80354, + "21": 10.79325, + "22": 10.78345, + "23": 10.77589, + "24": 10.73859, + "25": 10.75397, + "26": 10.72401, + "27": 10.68087, + "28": 10.60753, + "29": 10.58173, + "30": 10.57586, + "31": 10.55865, + "32": 10.54534, + "33": 10.51316, + "34": 10.47157, + "35": 10.47226, + "36": 10.46033, + "37": 10.41614, + "38": 10.42803, + "39": 10.38981, + "40": 10.37324, + "41": 10.35698, + "42": 10.34258, + "43": 10.30625, + "44": 10.2833, + "45": 10.29373, + "46": 10.26051, + "47": 10.24268, + "48": 10.2011, + "49": 10.1927, + "50": 10.19551, + "51": 10.20263, + "52": 10.15249, + "53": 10.15361, + "54": 10.11924, + "55": 10.09675, + "56": 10.11854, + "57": 10.10417, + "58": 10.11747, + "59": 10.06702, + "60": 10.08112, + "61": 10.03586, + "62": 10.00888, + "63": 10.07618, + "64": 10.03927, + "65": 10.01139, + "66": 10.03452, + "67": 10.01084, + "68": 9.97493, + "69": 9.99636, + "70": 9.97873, + "71": 10.00559, + "72": 9.98062, + "73": 9.97765, + "74": 9.95924, + "75": 9.9335, + "76": 9.96535, + "77": 9.9598, + "78": 9.9074, + "79": 9.91725, + "80": 9.93605, + "81": 9.96051, + "82": 9.89289, + "83": 9.85645, + "84": 9.78625, + "85": 9.78401, + "86": 9.88327, + "87": 9.90989, + "88": 9.88512, + "89": 9.81694, + "90": 9.80996, + "91": 9.82887, + "92": 9.81612, + "93": 9.74684, + "94": 9.83238, + "95": 9.8273, + "96": 9.80575, + "97": 9.74064, + "98": 9.77907, + "99": 9.82087, + "100": 9.71283 + } + }, + "num-zeros": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 1879.0, + "2": 1840.0, + "3": 1763.0, + "4": 1799.0, + "5": 1826.0, + "6": 1799.0, + "7": 1948.0, + "8": 1699.0, + "9": 1844.0, + "10": 1821.0, + "11": 1760.0, + "12": 1737.0, + "13": 1900.0, + "14": 1871.0, + "15": 1586.0, + "16": 1801.0, + "17": 1831.0, + "18": 1844.0, + "19": 1719.0, + "20": 1793.0, + "21": 1923.0, + "22": 1777.0, + "23": 1785.0, + "24": 1835.0, + "25": 1854.0, + "26": 1897.0, + "27": 1952.0, + "28": 1885.0, + "29": 2125.0, + "30": 1983.0, + "31": 2149.0, + "32": 2115.0, + "33": 2121.0, + "34": 2212.0, + "35": 2218.0, + "36": 2118.0, + "37": 2349.0, + "38": 2207.0, + "39": 2256.0, + "40": 2381.0, + "41": 2353.0, + "42": 2050.0, + "43": 2518.0, + "44": 2259.0, + "45": 2659.0, + "46": 2517.0, + "47": 2552.0, + "48": 2676.0, + "49": 2840.0, + "50": 2551.0, + "51": 2495.0, + "52": 2765.0, + "53": 2808.0, + "54": 2907.0, + "55": 2675.0, + "56": 2865.0, + "57": 2242.0, + "58": 3810.0, + "59": 3052.0, + "60": 3190.0, + "61": 2925.0, + "62": 3385.0, + "63": 3602.0, + "64": 3723.0, + "65": 2785.0, + "66": 3298.0, + "67": 4072.0, + "68": 3407.0, + "69": 3096.0, + "70": 3528.0, + "71": 3225.0, + "72": 3092.0, + "73": 3475.0, + "74": 3362.0, + "75": 3195.0, + "76": 3320.0, + "77": 3944.0, + "78": 3522.0, + "79": 3281.0, + "80": 3241.0, + "81": 3763.0, + "82": 2964.0, + "83": 3107.0, + "84": 3076.0, + "85": 2865.0, + "86": 2990.0, + "87": 3080.0, + "88": 3170.0, + "89": 3342.0, + "90": 3855.0, + "91": 2997.0, + "92": 2860.0, + "93": 3091.0, + "94": 3006.0, + "95": 3393.0, + "96": 3323.0, + "97": 3534.0, + "98": 3263.0, + "99": 3271.0, + "100": 3436.0 + } + }, + "mem-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 394290688.0, + "2": 394290688.0, + "3": 394290688.0, + "4": 394290688.0, + "5": 394290688.0, + "6": 394290688.0, + "7": 394290688.0, + "8": 394290688.0, + "9": 394290688.0, + "10": 394290688.0, + "11": 394290688.0, + "12": 394290688.0, + "13": 394290688.0, + "14": 394290688.0, + "15": 394290688.0, + "16": 394290688.0, + "17": 394290688.0, + "18": 394290688.0, + "19": 394290688.0, + "20": 394290688.0, + "21": 394290688.0, + "22": 394290688.0, + "23": 394290688.0, + "24": 394290688.0, + "25": 394290688.0, + "26": 394290688.0, + "27": 394290688.0, + "28": 394290688.0, + "29": 394290688.0, + "30": 394290688.0, + "31": 394290688.0, + "32": 394290688.0, + "33": 394290688.0, + "34": 394290688.0, + "35": 394290688.0, + "36": 394290688.0, + "37": 394290688.0, + "38": 394290688.0, + "39": 394290688.0, + "40": 394290688.0, + "41": 394290688.0, + "42": 394290688.0, + "43": 394290688.0, + "44": 394290688.0, + "45": 394290688.0, + "46": 394290688.0, + "47": 394290688.0, + "48": 394290688.0, + "49": 394290688.0, + "50": 394290688.0, + "51": 394290688.0, + "52": 394290688.0, + "53": 394290688.0, + "54": 394290688.0, + "55": 394290688.0, + "56": 394290688.0, + "57": 394290688.0, + "58": 394290688.0, + "59": 394290688.0, + "60": 394290688.0, + "61": 394290688.0, + "62": 394290688.0, + "63": 394290688.0, + "64": 394290688.0, + "65": 394290688.0, + "66": 394290688.0, + "67": 394290688.0, + "68": 394290688.0, + "69": 394290688.0, + "70": 394290688.0, + "71": 394290688.0, + "72": 394290688.0, + "73": 394290688.0, + "74": 394290688.0, + "75": 394290688.0, + "76": 394290688.0, + "77": 394290688.0, + "78": 394290688.0, + "79": 394290688.0, + "80": 394290688.0, + "81": 394290688.0, + "82": 394290688.0, + "83": 394290688.0, + "84": 394290688.0, + "85": 394290688.0, + "86": 394290688.0, + "87": 394290688.0, + "88": 394290688.0, + "89": 394290688.0, + "90": 394290688.0, + "91": 394290688.0, + "92": 394290688.0, + "93": 394290688.0, + "94": 394290688.0, + "95": 394290688.0, + "96": 394290688.0, + "97": 394290688.0, + "98": 394290688.0, + "99": 394290688.0, + "100": 394290688.0 + } + }, + "mem-max-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 1642060800.0, + "2": 1642060800.0, + "3": 1642060800.0, + "4": 1642060800.0, + "5": 1642060800.0, + "6": 1642060800.0, + "7": 1642060800.0, + "8": 1642060800.0, + "9": 1642060800.0, + "10": 1642060800.0, + "11": 1642060800.0, + "12": 1642060800.0, + "13": 1642060800.0, + "14": 1642060800.0, + "15": 1642060800.0, + "16": 1642060800.0, + "17": 1642060800.0, + "18": 1642060800.0, + "19": 1642060800.0, + "20": 1642060800.0, + "21": 1642060800.0, + "22": 1642060800.0, + "23": 1642060800.0, + "24": 1642060800.0, + "25": 1642060800.0, + "26": 1642060800.0, + "27": 1642060800.0, + "28": 1642060800.0, + "29": 1642060800.0, + "30": 1642060800.0, + "31": 1642060800.0, + "32": 1642060800.0, + "33": 1642060800.0, + "34": 1642060800.0, + "35": 1642060800.0, + "36": 1642060800.0, + "37": 1642060800.0, + "38": 1642060800.0, + "39": 1642060800.0, + "40": 1642060800.0, + "41": 1642060800.0, + "42": 1642060800.0, + "43": 1642060800.0, + "44": 1642060800.0, + "45": 1642060800.0, + "46": 1642060800.0, + "47": 1642060800.0, + "48": 1642060800.0, + "49": 1642060800.0, + "50": 1642060800.0, + "51": 1642060800.0, + "52": 1642060800.0, + "53": 1642060800.0, + "54": 1642060800.0, + "55": 1642060800.0, + "56": 1642060800.0, + "57": 1642060800.0, + "58": 1642060800.0, + "59": 1642060800.0, + "60": 1642060800.0, + "61": 1642060800.0, + "62": 1642060800.0, + "63": 1642060800.0, + "64": 1642060800.0, + "65": 1642060800.0, + "66": 1642060800.0, + "67": 1642060800.0, + "68": 1642060800.0, + "69": 1642060800.0, + "70": 1642060800.0, + "71": 1642060800.0, + "72": 1642060800.0, + "73": 1642060800.0, + "74": 1642060800.0, + "75": 1642060800.0, + "76": 1642060800.0, + "77": 1642060800.0, + "78": 1642060800.0, + "79": 1642060800.0, + "80": 1642060800.0, + "81": 1642060800.0, + "82": 1642060800.0, + "83": 1642060800.0, + "84": 1642060800.0, + "85": 1642060800.0, + "86": 1642060800.0, + "87": 1642060800.0, + "88": 1642060800.0, + "89": 1642060800.0, + "90": 1642060800.0, + "91": 1642060800.0, + "92": 1642060800.0, + "93": 1642060800.0, + "94": 1642060800.0, + "95": 1642060800.0, + "96": 1642060800.0, + "97": 1642060800.0, + "98": 1642060800.0, + "99": 1642060800.0, + "100": 1642060800.0 + } + }, + "iteration-time": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": "nan", + "2": 3.44044, + "3": 1.09132, + "4": 0.91551, + "5": 0.7129, + "6": 0.98539, + "7": 0.97618, + "8": 1.16941, + "9": 1.40389, + "10": 0.6131, + "11": 1.0429, + "12": 0.99352, + "13": 0.88123, + "14": 1.26324, + "15": 0.7096, + "16": 0.87855, + "17": 1.01579, + "18": 0.97794, + "19": 0.9756, + "20": 0.94218, + "21": 0.95493, + "22": 0.91029, + "23": 1.02444, + "24": 1.44288, + "25": 0.56656, + "26": 0.85132, + "27": 0.94663, + "28": 1.01233, + "29": 0.84541, + "30": 0.80132, + "31": 0.71911, + "32": 1.49348, + "33": 0.95046, + "34": 0.45084, + "35": 1.06203, + "36": 1.22109, + "37": 0.99784, + "38": 0.58876, + "39": 1.03921, + "40": 1.37732, + "41": 0.9811, + "42": 1.09855, + "43": 0.98879, + "44": 1.24364, + "45": 0.79178, + "46": 0.7893, + "47": 1.16897, + "48": 1.11308, + "49": 0.8317, + "50": 1.12236, + "51": 0.51194, + "52": 0.90605, + "53": 1.25287, + "54": 1.29748, + "55": 1.14487, + "56": 1.03478, + "57": 0.77131, + "58": 1.28308, + "59": 0.98927, + "60": 0.687, + "61": 1.12292, + "62": 1.13383, + "63": 1.06608, + "64": 1.18823, + "65": 0.81087, + "66": 0.8859, + "67": 1.13009, + "68": 0.92921, + "69": 1.35935, + "70": 0.94987, + "71": 1.15206, + "72": 1.10847, + "73": 1.01601, + "74": 0.72294, + "75": 1.03371, + "76": 1.28661, + "77": 1.28537, + "78": 1.28457, + "79": 1.82932, + "80": 1.13795, + "81": 1.06766, + "82": 1.22098, + "83": 1.1978, + "84": 1.07228, + "85": 0.71667, + "86": 0.92405, + "87": 1.70792, + "88": 0.96319, + "89": 0.81455, + "90": 1.30538, + "91": 0.92867, + "92": 1.53302, + "93": 0.97906, + "94": 1.0981, + "95": 1.02227, + "96": 0.98261, + "97": 1.02742, + "98": 1.12531, + "99": 1.3547, + "100": 0.94993 + } + } +} \ No newline at end of file diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_zp_z3_resume_fsdp_dtensor_1node/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_zp_z3_resume_fsdp_dtensor_1node/model_config.yaml new file mode 100644 index 00000000000..88e1a817a05 --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp2_zp_z3_resume_fsdp_dtensor_1node/model_config.yaml @@ -0,0 +1,55 @@ +ENV_VARS: + NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0 + NCCL_ALGO: Ring + CUBLAS_WORKSPACE_CONFIG: :4096:8 +MODEL_ARGS: + --num-layers: 12 + --hidden-size: 512 + --num-attention-heads: 8 + --log-params-norm: true + --log-num-zeros-in-grad: true + --log-validation-ppl-to-tensorboard: true + --log-timers-to-tensorboard: true + --log-memory-to-tensorboard: true + --tensorboard-dir: ${TENSORBOARD_PATH} + --micro-batch-size: 4 + --global-batch-size: 32 + --seq-length: 1024 + --max-position-embeddings: 1024 + --train-iters: 100 + --timing-log-level: 0 + --lr-decay-iters: 320000 + --save: ${CHECKPOINT_SAVE_PATH} + --load: ${CHECKPOINT_LOAD_PATH} + --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document + --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json + --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt + --split: 949,50,1 + --distributed-backend: nccl + --lr: 0.00015 + --lr-decay-style: cosine + --min-lr: 1.0e-5 + --weight-decay: 1e-2 + --clip-grad: 1.0 + --lr-warmup-fraction: .01 + --log-interval: 1 + --save-interval: 50 + --eval-interval: 1000 + --eval-iters: 10 + --transformer-impl: transformer_engine + --tensor-model-parallel-size: 2 + --use-megatron-fsdp: true + --calculate-per-token-loss: true + --data-parallel-sharding-strategy: optim_grads_params + --use-distributed-optimizer: true + --deterministic-mode: true + --no-gradient-accumulation-fusion: true + --attention-softmax-in-fp32: true + --use-checkpoint-opt_param-scheduler: true + --use-mcore-models: true + --ckpt-format: fsdp_dtensor + --data-cache-path: ${DATA_CACHE_PATH} + --bf16: true + --async-save: true + --use-persistent-ckpt-worker: true +TEST_TYPE: regular diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_dist_optimizer_overlap_grad_reduce_param_gather_1node/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_dist_optimizer_overlap_grad_reduce_param_gather_1node/golden_values_dev_dgx_gb200.json new file mode 100644 index 00000000000..40726e38439 --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_dist_optimizer_overlap_grad_reduce_param_gather_1node/golden_values_dev_dgx_gb200.json @@ -0,0 +1,287 @@ +{ + "lm loss": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 10.89465, + "2": 10.88327, + "3": 10.88075, + "4": 10.88361, + "5": 10.88287, + "6": 10.875, + "7": 10.87677, + "8": 10.88171, + "9": 10.88078, + "10": 10.87893, + "11": 10.87526, + "12": 10.86941, + "13": 10.86653, + "14": 10.86405, + "15": 10.8404, + "16": 10.83432, + "17": 10.84502, + "18": 10.81955, + "19": 10.84023, + "20": 10.75963, + "21": 10.74833, + "22": 10.73542, + "23": 10.72861, + "24": 10.70175, + "25": 10.69385, + "26": 10.68277, + "27": 10.65028, + "28": 10.58845, + "29": 10.5527, + "30": 10.53012, + "31": 10.52065, + "32": 10.51506, + "33": 10.47611, + "34": 10.45066, + "35": 10.45166, + "36": 10.42459, + "37": 10.39344, + "38": 10.39768, + "39": 10.36673, + "40": 10.35691, + "41": 10.33049, + "42": 10.30995, + "43": 10.28971, + "44": 10.26685, + "45": 10.27231, + "46": 10.23941, + "47": 10.2249, + "48": 10.18112, + "49": 10.1825, + "50": 10.18095 + } + }, + "num-zeros": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 1815.0, + "2": 1778.0, + "3": 1884.0, + "4": 1723.0, + "5": 1861.0, + "6": 1834.0, + "7": 1939.0, + "8": 1787.0, + "9": 1863.0, + "10": 1827.0, + "11": 1824.0, + "12": 1776.0, + "13": 1751.0, + "14": 1921.0, + "15": 1778.0, + "16": 1802.0, + "17": 1812.0, + "18": 1776.0, + "19": 1781.0, + "20": 1731.0, + "21": 1944.0, + "22": 1875.0, + "23": 1834.0, + "24": 1839.0, + "25": 1770.0, + "26": 1904.0, + "27": 1826.0, + "28": 1782.0, + "29": 1928.0, + "30": 1923.0, + "31": 2037.0, + "32": 1915.0, + "33": 1951.0, + "34": 2059.0, + "35": 2146.0, + "36": 2031.0, + "37": 2301.0, + "38": 2103.0, + "39": 2248.0, + "40": 2226.0, + "41": 2352.0, + "42": 1925.0, + "43": 2435.0, + "44": 2241.0, + "45": 2601.0, + "46": 2495.0, + "47": 2557.0, + "48": 2705.0, + "49": 2785.0, + "50": 2545.0 + } + }, + "mem-allocated-bytes": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 398193152.0, + "2": 398193152.0, + "3": 398193152.0, + "4": 398193152.0, + "5": 398193152.0, + "6": 398193152.0, + "7": 398193152.0, + "8": 398193152.0, + "9": 398193152.0, + "10": 398193152.0, + "11": 398193152.0, + "12": 398193152.0, + "13": 398193152.0, + "14": 398193152.0, + "15": 398193152.0, + "16": 398193152.0, + "17": 398193152.0, + "18": 398193152.0, + "19": 398193152.0, + "20": 398193152.0, + "21": 398193152.0, + "22": 398193152.0, + "23": 398193152.0, + "24": 398193152.0, + "25": 398193152.0, + "26": 398193152.0, + "27": 398193152.0, + "28": 398193152.0, + "29": 398193152.0, + "30": 398193152.0, + "31": 398193152.0, + "32": 398193152.0, + "33": 398193152.0, + "34": 398193152.0, + "35": 398193152.0, + "36": 398193152.0, + "37": 398193152.0, + "38": 398193152.0, + "39": 398193152.0, + "40": 398193152.0, + "41": 398193152.0, + "42": 398193152.0, + "43": 398193152.0, + "44": 398193152.0, + "45": 398193152.0, + "46": 398193152.0, + "47": 398193152.0, + "48": 398193152.0, + "49": 398193152.0, + "50": 398193152.0 + } + }, + "mem-max-allocated-bytes": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 1509896704.0, + "2": 1642407424.0, + "3": 1642407424.0, + "4": 1642407424.0, + "5": 1642407424.0, + "6": 1642407424.0, + "7": 1642407424.0, + "8": 1642407424.0, + "9": 1642407424.0, + "10": 1642407424.0, + "11": 1642407424.0, + "12": 1642407424.0, + "13": 1642407424.0, + "14": 1642407424.0, + "15": 1642407424.0, + "16": 1642407424.0, + "17": 1642407424.0, + "18": 1642407424.0, + "19": 1642407424.0, + "20": 1642407424.0, + "21": 1642407424.0, + "22": 1642407424.0, + "23": 1642407424.0, + "24": 1642407424.0, + "25": 1642407424.0, + "26": 1642407424.0, + "27": 1642407424.0, + "28": 1642407424.0, + "29": 1642407424.0, + "30": 1642407424.0, + "31": 1642407424.0, + "32": 1642407424.0, + "33": 1642407424.0, + "34": 1642407424.0, + "35": 1642407424.0, + "36": 1642407424.0, + "37": 1642407424.0, + "38": 1642407424.0, + "39": 1642407424.0, + "40": 1642407424.0, + "41": 1642407424.0, + "42": 1642407424.0, + "43": 1642407424.0, + "44": 1642407424.0, + "45": 1642407424.0, + "46": 1642407424.0, + "47": 1642407424.0, + "48": 1642407424.0, + "49": 1642407424.0, + "50": 1642407424.0 + } + }, + "iteration-time": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": "nan", + "2": 4.92236, + "3": 2.33606, + "4": 2.14516, + "5": 1.83016, + "6": 2.16862, + "7": 1.76235, + "8": 2.43585, + "9": 2.43675, + "10": 2.11674, + "11": 2.35369, + "12": 1.95978, + "13": 2.13313, + "14": 2.44773, + "15": 1.85169, + "16": 2.11244, + "17": 2.63204, + "18": 1.95751, + "19": 2.19845, + "20": 1.94686, + "21": 2.31649, + "22": 1.77509, + "23": 2.31029, + "24": 1.79652, + "25": 2.35545, + "26": 2.31143, + "27": 1.63427, + "28": 1.77833, + "29": 1.55928, + "30": 1.77389, + "31": 1.76954, + "32": 2.8591, + "33": 1.52099, + "34": 1.52307, + "35": 2.1913, + "36": 1.94399, + "37": 1.85372, + "38": 1.59739, + "39": 2.50792, + "40": 1.92749, + "41": 2.12413, + "42": 2.37404, + "43": 1.80019, + "44": 1.8608, + "45": 2.26604, + "46": 1.70302, + "47": 1.87015, + "48": 2.10855, + "49": 1.62615, + "50": 1.91035 + } + } +} \ No newline at end of file diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_dist_optimizer_overlap_grad_reduce_param_gather_1node/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_dist_optimizer_overlap_grad_reduce_param_gather_1node/model_config.yaml new file mode 100644 index 00000000000..11a72d75951 --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_dist_optimizer_overlap_grad_reduce_param_gather_1node/model_config.yaml @@ -0,0 +1,57 @@ +ENV_VARS: + CUDA_DEVICE_MAX_CONNECTIONS: 1 + NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0 + NCCL_ALGO: Ring + CUBLAS_WORKSPACE_CONFIG: :4096:8 +MODEL_ARGS: + --num-layers: 12 + --hidden-size: 512 + --num-attention-heads: 8 + --log-params-norm: true + --log-num-zeros-in-grad: true + --log-validation-ppl-to-tensorboard: true + --log-timers-to-tensorboard: true + --tensorboard-dir: ${TENSORBOARD_PATH} + --micro-batch-size: 4 + --global-batch-size: 32 + --seq-length: 1024 + --max-position-embeddings: 1024 + --train-iters: 50 + --timing-log-level: 0 + --lr-decay-iters: 320000 + --save: ${CHECKPOINT_SAVE_PATH} + --load: ${CHECKPOINT_LOAD_PATH} + --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document + --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json + --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt + --split: 949,50,1 + --distributed-backend: nccl + --lr: 0.00015 + --lr-decay-style: cosine + --min-lr: 1.0e-5 + --weight-decay: 1e-2 + --clip-grad: 1.0 + --lr-warmup-fraction: .01 + --log-interval: 1 + --save-interval: 10000 + --eval-interval: 1000 + --eval-iters: 10 + --transformer-impl: transformer_engine + --tensor-model-parallel-size: 4 + --pipeline-model-parallel-size: 1 + --use-distributed-optimizer: true + --overlap-grad-reduce: true + --overlap-param-gather: true + --deterministic-mode: true + --no-gradient-accumulation-fusion: true + --attention-softmax-in-fp32: true + --use-mcore-models: true + --ckpt-format: torch_dist + --dist-ckpt-strictness: log_all # backward compatibility for TE changes + --data-cache-path: ${DATA_CACHE_PATH} + --bf16: true + --attention-backend: unfused + --log-memory-to-tensorboard: true + --async-save: true + --use-persistent-ckpt-worker: true +TEST_TYPE: regular diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_1node/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_1node/golden_values_dev_dgx_gb200.json new file mode 100644 index 00000000000..2f50d1852c0 --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_1node/golden_values_dev_dgx_gb200.json @@ -0,0 +1,537 @@ +{ + "lm loss": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 10.89463, + "2": 10.88332, + "3": 10.88064, + "4": 10.88309, + "5": 10.88332, + "6": 10.87492, + "7": 10.87689, + "8": 10.88182, + "9": 10.88055, + "10": 10.879, + "11": 10.8752, + "12": 10.87014, + "13": 10.86618, + "14": 10.86433, + "15": 10.83999, + "16": 10.83442, + "17": 10.84483, + "18": 10.81981, + "19": 10.84023, + "20": 10.75974, + "21": 10.74842, + "22": 10.73585, + "23": 10.72846, + "24": 10.70204, + "25": 10.69386, + "26": 10.68294, + "27": 10.65042, + "28": 10.58861, + "29": 10.55276, + "30": 10.52991, + "31": 10.52067, + "32": 10.51487, + "33": 10.47672, + "34": 10.45098, + "35": 10.45204, + "36": 10.42467, + "37": 10.3935, + "38": 10.39796, + "39": 10.36701, + "40": 10.35717, + "41": 10.33082, + "42": 10.31015, + "43": 10.28972, + "44": 10.26722, + "45": 10.27276, + "46": 10.23956, + "47": 10.22509, + "48": 10.18113, + "49": 10.18221, + "50": 10.18109, + "51": 10.18054, + "52": 10.13848, + "53": 10.14381, + "54": 10.10815, + "55": 10.08196, + "56": 10.10878, + "57": 10.09685, + "58": 10.11263, + "59": 10.06067, + "60": 10.08081, + "61": 10.029, + "62": 10.00208, + "63": 10.07351, + "64": 10.02973, + "65": 10.00767, + "66": 10.02968, + "67": 10.00724, + "68": 9.96811, + "69": 9.99247, + "70": 9.96962, + "71": 9.99959, + "72": 9.97932, + "73": 9.97019, + "74": 9.9547, + "75": 9.93058, + "76": 9.96382, + "77": 9.95341, + "78": 9.90812, + "79": 9.91503, + "80": 9.92521, + "81": 9.95289, + "82": 9.88577, + "83": 9.85328, + "84": 9.79572, + "85": 9.78589, + "86": 9.87801, + "87": 9.90219, + "88": 9.87507, + "89": 9.82262, + "90": 9.81115, + "91": 9.82642, + "92": 9.81572, + "93": 9.75371, + "94": 9.82278, + "95": 9.81996, + "96": 9.80283, + "97": 9.74847, + "98": 9.77372, + "99": 9.81917, + "100": 9.71201 + } + }, + "num-zeros": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 1922.0, + "2": 1807.0, + "3": 1882.0, + "4": 1775.0, + "5": 1912.0, + "6": 1714.0, + "7": 1974.0, + "8": 1935.0, + "9": 1833.0, + "10": 1932.0, + "11": 1923.0, + "12": 1769.0, + "13": 1796.0, + "14": 2001.0, + "15": 1767.0, + "16": 1851.0, + "17": 1834.0, + "18": 1796.0, + "19": 1749.0, + "20": 1655.0, + "21": 1824.0, + "22": 1766.0, + "23": 1820.0, + "24": 1931.0, + "25": 1746.0, + "26": 1895.0, + "27": 1859.0, + "28": 1860.0, + "29": 1939.0, + "30": 1870.0, + "31": 2013.0, + "32": 1982.0, + "33": 2040.0, + "34": 2102.0, + "35": 2075.0, + "36": 2026.0, + "37": 2314.0, + "38": 2141.0, + "39": 2189.0, + "40": 2255.0, + "41": 2396.0, + "42": 1953.0, + "43": 2389.0, + "44": 2234.0, + "45": 2588.0, + "46": 2583.0, + "47": 2581.0, + "48": 2757.0, + "49": 2847.0, + "50": 2589.0, + "51": 2438.0, + "52": 2832.0, + "53": 2666.0, + "54": 2900.0, + "55": 2729.0, + "56": 2824.0, + "57": 2182.0, + "58": 3842.0, + "59": 2971.0, + "60": 3042.0, + "61": 2689.0, + "62": 3269.0, + "63": 3494.0, + "64": 3553.0, + "65": 2767.0, + "66": 3146.0, + "67": 3952.0, + "68": 3552.0, + "69": 3054.0, + "70": 3377.0, + "71": 3173.0, + "72": 2909.0, + "73": 3460.0, + "74": 3310.0, + "75": 3240.0, + "76": 3197.0, + "77": 3644.0, + "78": 3322.0, + "79": 3270.0, + "80": 3197.0, + "81": 3406.0, + "82": 2921.0, + "83": 2929.0, + "84": 2873.0, + "85": 2722.0, + "86": 3294.0, + "87": 2854.0, + "88": 3109.0, + "89": 3067.0, + "90": 3859.0, + "91": 2975.0, + "92": 3083.0, + "93": 3074.0, + "94": 3298.0, + "95": 3271.0, + "96": 3531.0, + "97": 3566.0, + "98": 3646.0, + "99": 3246.0, + "100": 3560.0 + } + }, + "mem-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 398193152.0, + "2": 398193152.0, + "3": 398193152.0, + "4": 398193152.0, + "5": 398193152.0, + "6": 398193152.0, + "7": 398193152.0, + "8": 398193152.0, + "9": 398193152.0, + "10": 398193152.0, + "11": 398193152.0, + "12": 398193152.0, + "13": 398193152.0, + "14": 398193152.0, + "15": 398193152.0, + "16": 398193152.0, + "17": 398193152.0, + "18": 398193152.0, + "19": 398193152.0, + "20": 398193152.0, + "21": 398193152.0, + "22": 398193152.0, + "23": 398193152.0, + "24": 398193152.0, + "25": 398193152.0, + "26": 398193152.0, + "27": 398193152.0, + "28": 398193152.0, + "29": 398193152.0, + "30": 398193152.0, + "31": 398193152.0, + "32": 398193152.0, + "33": 398193152.0, + "34": 398193152.0, + "35": 398193152.0, + "36": 398193152.0, + "37": 398193152.0, + "38": 398193152.0, + "39": 398193152.0, + "40": 398193152.0, + "41": 398193152.0, + "42": 398193152.0, + "43": 398193152.0, + "44": 398193152.0, + "45": 398193152.0, + "46": 398193152.0, + "47": 398193152.0, + "48": 398193152.0, + "49": 398193152.0, + "50": 398193152.0, + "51": 398193152.0, + "52": 398193152.0, + "53": 398193152.0, + "54": 398193152.0, + "55": 398193152.0, + "56": 398193152.0, + "57": 398193152.0, + "58": 398193152.0, + "59": 398193152.0, + "60": 398193152.0, + "61": 398193152.0, + "62": 398193152.0, + "63": 398193152.0, + "64": 398193152.0, + "65": 398193152.0, + "66": 398193152.0, + "67": 398193152.0, + "68": 398193152.0, + "69": 398193152.0, + "70": 398193152.0, + "71": 398193152.0, + "72": 398193152.0, + "73": 398193152.0, + "74": 398193152.0, + "75": 398193152.0, + "76": 398193152.0, + "77": 398193152.0, + "78": 398193152.0, + "79": 398193152.0, + "80": 398193152.0, + "81": 398193152.0, + "82": 398193152.0, + "83": 398193152.0, + "84": 398193152.0, + "85": 398193152.0, + "86": 398193152.0, + "87": 398193152.0, + "88": 398193152.0, + "89": 398193152.0, + "90": 398193152.0, + "91": 398193152.0, + "92": 398193152.0, + "93": 398193152.0, + "94": 398193152.0, + "95": 398193152.0, + "96": 398193152.0, + "97": 398193152.0, + "98": 398193152.0, + "99": 398193152.0, + "100": 398193152.0 + } + }, + "mem-max-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 1028344320.0, + "2": 1158360064.0, + "3": 1158360064.0, + "4": 1158360064.0, + "5": 1158360064.0, + "6": 1158360064.0, + "7": 1158360064.0, + "8": 1158360064.0, + "9": 1158360064.0, + "10": 1158360064.0, + "11": 1158360064.0, + "12": 1158360064.0, + "13": 1158360064.0, + "14": 1158360064.0, + "15": 1158360064.0, + "16": 1158360064.0, + "17": 1158360064.0, + "18": 1158360064.0, + "19": 1158360064.0, + "20": 1158360064.0, + "21": 1158360064.0, + "22": 1158360064.0, + "23": 1158360064.0, + "24": 1158360064.0, + "25": 1158360064.0, + "26": 1158360064.0, + "27": 1158360064.0, + "28": 1158360064.0, + "29": 1158360064.0, + "30": 1158360064.0, + "31": 1158360064.0, + "32": 1158360064.0, + "33": 1158360064.0, + "34": 1158360064.0, + "35": 1158360064.0, + "36": 1158360064.0, + "37": 1158360064.0, + "38": 1158360064.0, + "39": 1158360064.0, + "40": 1158360064.0, + "41": 1158360064.0, + "42": 1158360064.0, + "43": 1158360064.0, + "44": 1158360064.0, + "45": 1158360064.0, + "46": 1158360064.0, + "47": 1158360064.0, + "48": 1158360064.0, + "49": 1158360064.0, + "50": 1158360064.0, + "51": 1158360064.0, + "52": 1158360064.0, + "53": 1158360064.0, + "54": 1158360064.0, + "55": 1158360064.0, + "56": 1158360064.0, + "57": 1158360064.0, + "58": 1158360064.0, + "59": 1158360064.0, + "60": 1158360064.0, + "61": 1158360064.0, + "62": 1158360064.0, + "63": 1158360064.0, + "64": 1158360064.0, + "65": 1158360064.0, + "66": 1158360064.0, + "67": 1158360064.0, + "68": 1158360064.0, + "69": 1158360064.0, + "70": 1158360064.0, + "71": 1158360064.0, + "72": 1158360064.0, + "73": 1158360064.0, + "74": 1158360064.0, + "75": 1158360064.0, + "76": 1158360064.0, + "77": 1158360064.0, + "78": 1158360064.0, + "79": 1158360064.0, + "80": 1158360064.0, + "81": 1158360064.0, + "82": 1158360064.0, + "83": 1158360064.0, + "84": 1158360064.0, + "85": 1158360064.0, + "86": 1158360064.0, + "87": 1158360064.0, + "88": 1158360064.0, + "89": 1158360064.0, + "90": 1158360064.0, + "91": 1158360064.0, + "92": 1158360064.0, + "93": 1158360064.0, + "94": 1158360064.0, + "95": 1158360064.0, + "96": 1158360064.0, + "97": 1158360064.0, + "98": 1158360064.0, + "99": 1158360064.0, + "100": 1158360064.0 + } + }, + "iteration-time": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": "nan", + "2": 4.79268, + "3": 2.09767, + "4": 1.81293, + "5": 1.59164, + "6": 1.99316, + "7": 1.56275, + "8": 2.1037, + "9": 2.27297, + "10": 1.94185, + "11": 2.11559, + "12": 1.82854, + "13": 1.9378, + "14": 2.14021, + "15": 1.70395, + "16": 1.86406, + "17": 2.43661, + "18": 1.76308, + "19": 2.05825, + "20": 1.72435, + "21": 2.10476, + "22": 1.61156, + "23": 2.05388, + "24": 1.65755, + "25": 2.22636, + "26": 2.44264, + "27": 1.93195, + "28": 1.96022, + "29": 1.67786, + "30": 1.86133, + "31": 1.67328, + "32": 2.76013, + "33": 1.56483, + "34": 1.49171, + "35": 2.42408, + "36": 2.10328, + "37": 2.04228, + "38": 1.73011, + "39": 2.66311, + "40": 2.17804, + "41": 2.29505, + "42": 2.81957, + "43": 2.05039, + "44": 2.01172, + "45": 2.57501, + "46": 1.84495, + "47": 2.04703, + "48": 2.32685, + "49": 1.82339, + "50": 2.15125, + "51": 1.76133, + "52": 2.24317, + "53": 2.05372, + "54": 2.33969, + "55": 2.00823, + "56": 2.69029, + "57": 1.49424, + "58": 2.19142, + "59": 1.89228, + "60": 1.92027, + "61": 2.17663, + "62": 2.5544, + "63": 1.94823, + "64": 2.07746, + "65": 2.94697, + "66": 2.80953, + "67": 2.38812, + "68": 1.66867, + "69": 2.26151, + "70": 2.10078, + "71": 2.71592, + "72": 2.07445, + "73": 2.12652, + "74": 1.97662, + "75": 1.9601, + "76": 2.21406, + "77": 2.15326, + "78": 2.36, + "79": 2.53131, + "80": 2.23723, + "81": 2.55105, + "82": 2.52331, + "83": 2.28359, + "84": 1.79563, + "85": 1.94316, + "86": 1.9746, + "87": 2.58456, + "88": 2.3507, + "89": 2.30514, + "90": 2.20037, + "91": 2.28019, + "92": 2.67417, + "93": 2.01659, + "94": 2.48718, + "95": 2.2981, + "96": 2.37457, + "97": 2.00115, + "98": 2.11485, + "99": 3.01584, + "100": 2.02465 + } + } +} \ No newline at end of file diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_1node/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_1node/model_config.yaml new file mode 100644 index 00000000000..bd6f526c209 --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_1node/model_config.yaml @@ -0,0 +1,56 @@ +ENV_VARS: + CUDA_DEVICE_MAX_CONNECTIONS: 1 + NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0 + NCCL_ALGO: Ring + CUBLAS_WORKSPACE_CONFIG: :4096:8 +MODEL_ARGS: + --num-layers: 12 + --hidden-size: 512 + --num-attention-heads: 8 + --log-params-norm: true + --log-num-zeros-in-grad: true + --log-validation-ppl-to-tensorboard: true + --log-timers-to-tensorboard: true + --tensorboard-dir: ${TENSORBOARD_PATH} + --micro-batch-size: 4 + --global-batch-size: 32 + --seq-length: 1024 + --max-position-embeddings: 1024 + --train-iters: 100 + --timing-log-level: 0 + --lr-decay-iters: 320000 + --save: ${CHECKPOINT_SAVE_PATH} + --load: ${CHECKPOINT_LOAD_PATH} + --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document + --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json + --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt + --split: 949,50,1 + --distributed-backend: nccl + --lr: 0.00015 + --lr-decay-style: cosine + --min-lr: 1.0e-5 + --weight-decay: 1e-2 + --clip-grad: 1.0 + --lr-warmup-fraction: .01 + --log-interval: 1 + --save-interval: 50 + --eval-interval: 1000 + --eval-iters: 10 + --transformer-impl: transformer_engine + --tensor-model-parallel-size: 4 + --pipeline-model-parallel-size: 1 + --use-distributed-optimizer: true + --overlap-grad-reduce: true + --deterministic-mode: true + --no-gradient-accumulation-fusion: true + --attention-softmax-in-fp32: true + --use-checkpoint-opt_param-scheduler: true + --use-mcore-models: true + --ckpt-format: torch_dist + --dist-ckpt-strictness: log_all # backward compatibility for TE changes + --data-cache-path: ${DATA_CACHE_PATH} + --bf16: true + --log-memory-to-tensorboard: true + --async-save: true + --use-persistent-ckpt-worker: true +TEST_TYPE: ckpt-resume diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_param_gather_1node/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_param_gather_1node/golden_values_dev_dgx_gb200.json new file mode 100644 index 00000000000..1dc40207f4e --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_param_gather_1node/golden_values_dev_dgx_gb200.json @@ -0,0 +1,537 @@ +{ + "lm loss": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 10.89463, + "2": 10.88332, + "3": 10.88064, + "4": 10.88309, + "5": 10.88332, + "6": 10.87492, + "7": 10.87689, + "8": 10.88182, + "9": 10.88055, + "10": 10.879, + "11": 10.8752, + "12": 10.87014, + "13": 10.86618, + "14": 10.86433, + "15": 10.83999, + "16": 10.83442, + "17": 10.84483, + "18": 10.81981, + "19": 10.84023, + "20": 10.75974, + "21": 10.74842, + "22": 10.73585, + "23": 10.72846, + "24": 10.70204, + "25": 10.69386, + "26": 10.68294, + "27": 10.65042, + "28": 10.58861, + "29": 10.55276, + "30": 10.52991, + "31": 10.52067, + "32": 10.51487, + "33": 10.47672, + "34": 10.45098, + "35": 10.45204, + "36": 10.42467, + "37": 10.3935, + "38": 10.39796, + "39": 10.36701, + "40": 10.35717, + "41": 10.33082, + "42": 10.31015, + "43": 10.28972, + "44": 10.26722, + "45": 10.27276, + "46": 10.23956, + "47": 10.22509, + "48": 10.18113, + "49": 10.18221, + "50": 10.18109, + "51": 10.18054, + "52": 10.13848, + "53": 10.14381, + "54": 10.10815, + "55": 10.08196, + "56": 10.10878, + "57": 10.09685, + "58": 10.11263, + "59": 10.06067, + "60": 10.08081, + "61": 10.029, + "62": 10.00208, + "63": 10.07351, + "64": 10.02973, + "65": 10.00767, + "66": 10.02968, + "67": 10.00724, + "68": 9.96811, + "69": 9.99247, + "70": 9.96962, + "71": 9.99959, + "72": 9.97932, + "73": 9.97019, + "74": 9.9547, + "75": 9.93058, + "76": 9.96382, + "77": 9.95341, + "78": 9.90812, + "79": 9.91503, + "80": 9.92521, + "81": 9.95289, + "82": 9.88577, + "83": 9.85328, + "84": 9.79572, + "85": 9.78589, + "86": 9.87801, + "87": 9.90219, + "88": 9.87507, + "89": 9.82262, + "90": 9.81115, + "91": 9.82642, + "92": 9.81572, + "93": 9.75371, + "94": 9.82278, + "95": 9.81996, + "96": 9.80283, + "97": 9.74847, + "98": 9.77372, + "99": 9.81917, + "100": 9.71201 + } + }, + "num-zeros": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 1922.0, + "2": 1807.0, + "3": 1882.0, + "4": 1775.0, + "5": 1912.0, + "6": 1714.0, + "7": 1974.0, + "8": 1935.0, + "9": 1833.0, + "10": 1932.0, + "11": 1923.0, + "12": 1769.0, + "13": 1796.0, + "14": 2001.0, + "15": 1767.0, + "16": 1851.0, + "17": 1834.0, + "18": 1796.0, + "19": 1749.0, + "20": 1655.0, + "21": 1824.0, + "22": 1766.0, + "23": 1820.0, + "24": 1931.0, + "25": 1746.0, + "26": 1895.0, + "27": 1859.0, + "28": 1860.0, + "29": 1939.0, + "30": 1870.0, + "31": 2013.0, + "32": 1982.0, + "33": 2040.0, + "34": 2102.0, + "35": 2075.0, + "36": 2026.0, + "37": 2314.0, + "38": 2141.0, + "39": 2189.0, + "40": 2255.0, + "41": 2396.0, + "42": 1953.0, + "43": 2389.0, + "44": 2234.0, + "45": 2588.0, + "46": 2583.0, + "47": 2581.0, + "48": 2757.0, + "49": 2847.0, + "50": 2589.0, + "51": 2438.0, + "52": 2832.0, + "53": 2666.0, + "54": 2900.0, + "55": 2729.0, + "56": 2824.0, + "57": 2182.0, + "58": 3842.0, + "59": 2971.0, + "60": 3042.0, + "61": 2689.0, + "62": 3269.0, + "63": 3494.0, + "64": 3553.0, + "65": 2767.0, + "66": 3146.0, + "67": 3952.0, + "68": 3552.0, + "69": 3054.0, + "70": 3377.0, + "71": 3173.0, + "72": 2909.0, + "73": 3460.0, + "74": 3310.0, + "75": 3240.0, + "76": 3197.0, + "77": 3644.0, + "78": 3322.0, + "79": 3270.0, + "80": 3197.0, + "81": 3406.0, + "82": 2921.0, + "83": 2929.0, + "84": 2873.0, + "85": 2722.0, + "86": 3294.0, + "87": 2854.0, + "88": 3109.0, + "89": 3067.0, + "90": 3859.0, + "91": 2975.0, + "92": 3083.0, + "93": 3074.0, + "94": 3298.0, + "95": 3271.0, + "96": 3531.0, + "97": 3566.0, + "98": 3646.0, + "99": 3246.0, + "100": 3560.0 + } + }, + "mem-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 398193152.0, + "2": 398193152.0, + "3": 398193152.0, + "4": 398193152.0, + "5": 398193152.0, + "6": 398193152.0, + "7": 398193152.0, + "8": 398193152.0, + "9": 398193152.0, + "10": 398193152.0, + "11": 398193152.0, + "12": 398193152.0, + "13": 398193152.0, + "14": 398193152.0, + "15": 398193152.0, + "16": 398193152.0, + "17": 398193152.0, + "18": 398193152.0, + "19": 398193152.0, + "20": 398193152.0, + "21": 398193152.0, + "22": 398193152.0, + "23": 398193152.0, + "24": 398193152.0, + "25": 398193152.0, + "26": 398193152.0, + "27": 398193152.0, + "28": 398193152.0, + "29": 398193152.0, + "30": 398193152.0, + "31": 398193152.0, + "32": 398193152.0, + "33": 398193152.0, + "34": 398193152.0, + "35": 398193152.0, + "36": 398193152.0, + "37": 398193152.0, + "38": 398193152.0, + "39": 398193152.0, + "40": 398193152.0, + "41": 398193152.0, + "42": 398193152.0, + "43": 398193152.0, + "44": 398193152.0, + "45": 398193152.0, + "46": 398193152.0, + "47": 398193152.0, + "48": 398193152.0, + "49": 398193152.0, + "50": 398193152.0, + "51": 398193152.0, + "52": 398193152.0, + "53": 398193152.0, + "54": 398193152.0, + "55": 398193152.0, + "56": 398193152.0, + "57": 398193152.0, + "58": 398193152.0, + "59": 398193152.0, + "60": 398193152.0, + "61": 398193152.0, + "62": 398193152.0, + "63": 398193152.0, + "64": 398193152.0, + "65": 398193152.0, + "66": 398193152.0, + "67": 398193152.0, + "68": 398193152.0, + "69": 398193152.0, + "70": 398193152.0, + "71": 398193152.0, + "72": 398193152.0, + "73": 398193152.0, + "74": 398193152.0, + "75": 398193152.0, + "76": 398193152.0, + "77": 398193152.0, + "78": 398193152.0, + "79": 398193152.0, + "80": 398193152.0, + "81": 398193152.0, + "82": 398193152.0, + "83": 398193152.0, + "84": 398193152.0, + "85": 398193152.0, + "86": 398193152.0, + "87": 398193152.0, + "88": 398193152.0, + "89": 398193152.0, + "90": 398193152.0, + "91": 398193152.0, + "92": 398193152.0, + "93": 398193152.0, + "94": 398193152.0, + "95": 398193152.0, + "96": 398193152.0, + "97": 398193152.0, + "98": 398193152.0, + "99": 398193152.0, + "100": 398193152.0 + } + }, + "mem-max-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 1028344320.0, + "2": 1158360064.0, + "3": 1158360064.0, + "4": 1158360064.0, + "5": 1158360064.0, + "6": 1158360064.0, + "7": 1158360064.0, + "8": 1158360064.0, + "9": 1158360064.0, + "10": 1158360064.0, + "11": 1158360064.0, + "12": 1158360064.0, + "13": 1158360064.0, + "14": 1158360064.0, + "15": 1158360064.0, + "16": 1158360064.0, + "17": 1158360064.0, + "18": 1158360064.0, + "19": 1158360064.0, + "20": 1158360064.0, + "21": 1158360064.0, + "22": 1158360064.0, + "23": 1158360064.0, + "24": 1158360064.0, + "25": 1158360064.0, + "26": 1158360064.0, + "27": 1158360064.0, + "28": 1158360064.0, + "29": 1158360064.0, + "30": 1158360064.0, + "31": 1158360064.0, + "32": 1158360064.0, + "33": 1158360064.0, + "34": 1158360064.0, + "35": 1158360064.0, + "36": 1158360064.0, + "37": 1158360064.0, + "38": 1158360064.0, + "39": 1158360064.0, + "40": 1158360064.0, + "41": 1158360064.0, + "42": 1158360064.0, + "43": 1158360064.0, + "44": 1158360064.0, + "45": 1158360064.0, + "46": 1158360064.0, + "47": 1158360064.0, + "48": 1158360064.0, + "49": 1158360064.0, + "50": 1158360064.0, + "51": 1158360064.0, + "52": 1158360064.0, + "53": 1158360064.0, + "54": 1158360064.0, + "55": 1158360064.0, + "56": 1158360064.0, + "57": 1158360064.0, + "58": 1158360064.0, + "59": 1158360064.0, + "60": 1158360064.0, + "61": 1158360064.0, + "62": 1158360064.0, + "63": 1158360064.0, + "64": 1158360064.0, + "65": 1158360064.0, + "66": 1158360064.0, + "67": 1158360064.0, + "68": 1158360064.0, + "69": 1158360064.0, + "70": 1158360064.0, + "71": 1158360064.0, + "72": 1158360064.0, + "73": 1158360064.0, + "74": 1158360064.0, + "75": 1158360064.0, + "76": 1158360064.0, + "77": 1158360064.0, + "78": 1158360064.0, + "79": 1158360064.0, + "80": 1158360064.0, + "81": 1158360064.0, + "82": 1158360064.0, + "83": 1158360064.0, + "84": 1158360064.0, + "85": 1158360064.0, + "86": 1158360064.0, + "87": 1158360064.0, + "88": 1158360064.0, + "89": 1158360064.0, + "90": 1158360064.0, + "91": 1158360064.0, + "92": 1158360064.0, + "93": 1158360064.0, + "94": 1158360064.0, + "95": 1158360064.0, + "96": 1158360064.0, + "97": 1158360064.0, + "98": 1158360064.0, + "99": 1158360064.0, + "100": 1158360064.0 + } + }, + "iteration-time": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": "nan", + "2": 4.85798, + "3": 2.33423, + "4": 2.05939, + "5": 1.76155, + "6": 2.25749, + "7": 1.82574, + "8": 2.86123, + "9": 2.75986, + "10": 2.30218, + "11": 2.51313, + "12": 1.96936, + "13": 2.32936, + "14": 2.44053, + "15": 1.6882, + "16": 2.07562, + "17": 2.45031, + "18": 1.77682, + "19": 2.04643, + "20": 1.69018, + "21": 2.07033, + "22": 1.65666, + "23": 2.24254, + "24": 1.77985, + "25": 2.42995, + "26": 2.52764, + "27": 1.95918, + "28": 1.8617, + "29": 1.56131, + "30": 1.6832, + "31": 1.59847, + "32": 2.68201, + "33": 1.56255, + "34": 1.51903, + "35": 2.11919, + "36": 2.07754, + "37": 1.81292, + "38": 1.4702, + "39": 2.20223, + "40": 1.85535, + "41": 1.91807, + "42": 2.4418, + "43": 1.70148, + "44": 1.7631, + "45": 2.33838, + "46": 1.65679, + "47": 1.87192, + "48": 2.1227, + "49": 1.6291, + "50": 1.96288, + "51": 1.27296, + "52": 2.25003, + "53": 2.15011, + "54": 2.16143, + "55": 1.93844, + "56": 2.72902, + "57": 1.37822, + "58": 2.06414, + "59": 1.79292, + "60": 1.90403, + "61": 2.14706, + "62": 2.4544, + "63": 1.76096, + "64": 1.99473, + "65": 1.90693, + "66": 2.19214, + "67": 2.28766, + "68": 1.53796, + "69": 2.15298, + "70": 1.84606, + "71": 2.45438, + "72": 1.76053, + "73": 1.69193, + "74": 1.79054, + "75": 1.65917, + "76": 1.89196, + "77": 1.97729, + "78": 2.08552, + "79": 2.34258, + "80": 2.09338, + "81": 2.39216, + "82": 2.27974, + "83": 2.03946, + "84": 1.78054, + "85": 2.06032, + "86": 1.88863, + "87": 2.57279, + "88": 2.27021, + "89": 2.24989, + "90": 2.06671, + "91": 2.19184, + "92": 2.6413, + "93": 1.77297, + "94": 2.25038, + "95": 2.06645, + "96": 2.28973, + "97": 1.91464, + "98": 2.12594, + "99": 2.88998, + "100": 2.06171 + } + } +} \ No newline at end of file diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_param_gather_1node/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_param_gather_1node/model_config.yaml new file mode 100644 index 00000000000..12acfaf8402 --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_param_gather_1node/model_config.yaml @@ -0,0 +1,60 @@ +ENV_VARS: + CUDA_DEVICE_MAX_CONNECTIONS: 1 + NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0 + NCCL_ALGO: Ring + CUBLAS_WORKSPACE_CONFIG: :4096:8 +MODEL_ARGS: + --num-layers: 12 + --hidden-size: 512 + --num-attention-heads: 8 + --log-params-norm: true + --log-num-zeros-in-grad: true + --log-validation-ppl-to-tensorboard: true + --log-timers-to-tensorboard: true + --tensorboard-dir: ${TENSORBOARD_PATH} + --micro-batch-size: 4 + --global-batch-size: 32 + --seq-length: 1024 + --max-position-embeddings: 1024 + --train-iters: 100 + --timing-log-level: 0 + --lr-decay-iters: 320000 + --save: ${CHECKPOINT_SAVE_PATH} + --load: ${CHECKPOINT_LOAD_PATH} + --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document + --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json + --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt + --split: 949,50,1 + --distributed-backend: nccl + --lr: 0.00015 + --lr-decay-style: cosine + --min-lr: 1.0e-5 + --weight-decay: 1e-2 + --clip-grad: 1.0 + --lr-warmup-fraction: .01 + --log-interval: 1 + --save-interval: 50 + --eval-interval: 1000 + --eval-iters: 10 + --transformer-impl: transformer_engine + --tensor-model-parallel-size: 4 + --pipeline-model-parallel-size: 1 + --use-distributed-optimizer: true + --overlap-grad-reduce: true + --overlap-param-gather: true + --deterministic-mode: true + --no-gradient-accumulation-fusion: true + --attention-softmax-in-fp32: true + --use-checkpoint-opt_param-scheduler: true + --use-mcore-models: true + --ckpt-format: torch_dist + --dist-ckpt-optim-fully-reshardable: true + --dist-ckpt-strictness: log_all # backward compatibility for TE changes + --data-cache-path: ${DATA_CACHE_PATH} + --bf16: true + --log-memory-to-tensorboard: true + --async-save: true + --async-strategy: mcore + --use-persistent-ckpt-worker: true +TEST_TYPE: ckpt-resume +LAUNCHER: ft_launcher diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_resume_torch_dist_qk_layernorm_test_mode_1node/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_resume_torch_dist_qk_layernorm_test_mode_1node/golden_values_dev_dgx_gb200.json new file mode 100644 index 00000000000..eb68302393d --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_resume_torch_dist_qk_layernorm_test_mode_1node/golden_values_dev_dgx_gb200.json @@ -0,0 +1,537 @@ +{ + "lm loss": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 10.89534, + "2": 10.88313, + "3": 10.88058, + "4": 10.8822, + "5": 10.88321, + "6": 10.87484, + "7": 10.87757, + "8": 10.88301, + "9": 10.88054, + "10": 10.87887, + "11": 10.87541, + "12": 10.86979, + "13": 10.86712, + "14": 10.86521, + "15": 10.84124, + "16": 10.83552, + "17": 10.84586, + "18": 10.82014, + "19": 10.84135, + "20": 10.76033, + "21": 10.7515, + "22": 10.73896, + "23": 10.72982, + "24": 10.70361, + "25": 10.69585, + "26": 10.68595, + "27": 10.65365, + "28": 10.592, + "29": 10.55593, + "30": 10.53219, + "31": 10.52302, + "32": 10.51797, + "33": 10.47935, + "34": 10.45411, + "35": 10.454, + "36": 10.42698, + "37": 10.39699, + "38": 10.40023, + "39": 10.36898, + "40": 10.35931, + "41": 10.33342, + "42": 10.31186, + "43": 10.29213, + "44": 10.26934, + "45": 10.27395, + "46": 10.24145, + "47": 10.227, + "48": 10.18305, + "49": 10.18406, + "50": 10.18231, + "51": 10.18171, + "52": 10.14054, + "53": 10.14639, + "54": 10.11019, + "55": 10.08362, + "56": 10.11023, + "57": 10.09879, + "58": 10.11422, + "59": 10.06263, + "60": 10.08229, + "61": 10.03027, + "62": 10.00397, + "63": 10.07492, + "64": 10.03142, + "65": 10.00976, + "66": 10.03138, + "67": 10.00902, + "68": 9.97028, + "69": 9.99473, + "70": 9.97175, + "71": 10.00142, + "72": 9.98221, + "73": 9.97407, + "74": 9.95811, + "75": 9.93459, + "76": 9.96651, + "77": 9.9566, + "78": 9.91141, + "79": 9.91914, + "80": 9.9304, + "81": 9.95666, + "82": 9.88909, + "83": 9.85746, + "84": 9.79774, + "85": 9.78932, + "86": 9.87974, + "87": 9.90388, + "88": 9.87781, + "89": 9.82415, + "90": 9.81292, + "91": 9.8278, + "92": 9.81697, + "93": 9.75605, + "94": 9.82456, + "95": 9.8217, + "96": 9.80484, + "97": 9.75029, + "98": 9.77497, + "99": 9.82037, + "100": 9.7128 + } + }, + "num-zeros": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 1879.0, + "2": 1827.0, + "3": 1946.0, + "4": 1786.0, + "5": 1934.0, + "6": 1687.0, + "7": 1969.0, + "8": 1907.0, + "9": 1922.0, + "10": 1838.0, + "11": 1809.0, + "12": 1821.0, + "13": 1859.0, + "14": 2036.0, + "15": 1774.0, + "16": 1853.0, + "17": 1932.0, + "18": 1854.0, + "19": 1797.0, + "20": 1814.0, + "21": 1858.0, + "22": 1779.0, + "23": 1827.0, + "24": 1837.0, + "25": 1757.0, + "26": 1871.0, + "27": 1867.0, + "28": 1949.0, + "29": 2044.0, + "30": 1926.0, + "31": 2023.0, + "32": 1969.0, + "33": 2007.0, + "34": 2111.0, + "35": 2177.0, + "36": 2119.0, + "37": 2240.0, + "38": 2178.0, + "39": 2220.0, + "40": 2368.0, + "41": 2364.0, + "42": 2047.0, + "43": 2445.0, + "44": 2287.0, + "45": 2599.0, + "46": 2556.0, + "47": 2488.0, + "48": 2719.0, + "49": 2822.0, + "50": 2595.0, + "51": 2602.0, + "52": 2804.0, + "53": 2786.0, + "54": 3027.0, + "55": 2718.0, + "56": 2783.0, + "57": 2218.0, + "58": 3910.0, + "59": 3107.0, + "60": 3134.0, + "61": 2834.0, + "62": 3274.0, + "63": 3534.0, + "64": 3542.0, + "65": 2765.0, + "66": 3175.0, + "67": 4116.0, + "68": 3516.0, + "69": 3088.0, + "70": 3237.0, + "71": 3162.0, + "72": 3061.0, + "73": 3470.0, + "74": 3406.0, + "75": 3253.0, + "76": 3291.0, + "77": 3771.0, + "78": 3318.0, + "79": 3240.0, + "80": 3152.0, + "81": 3521.0, + "82": 2878.0, + "83": 3065.0, + "84": 2924.0, + "85": 2831.0, + "86": 3152.0, + "87": 3004.0, + "88": 3149.0, + "89": 3162.0, + "90": 3938.0, + "91": 2951.0, + "92": 2982.0, + "93": 3147.0, + "94": 3175.0, + "95": 3284.0, + "96": 3419.0, + "97": 3522.0, + "98": 3569.0, + "99": 3208.0, + "100": 3476.0 + } + }, + "mem-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 397748736.0, + "2": 397748736.0, + "3": 397748736.0, + "4": 397748736.0, + "5": 397748736.0, + "6": 397748736.0, + "7": 397748736.0, + "8": 397748736.0, + "9": 397748736.0, + "10": 397748736.0, + "11": 397748736.0, + "12": 397748736.0, + "13": 397748736.0, + "14": 397748736.0, + "15": 397748736.0, + "16": 397748736.0, + "17": 397748736.0, + "18": 397748736.0, + "19": 397748736.0, + "20": 397748736.0, + "21": 397748736.0, + "22": 397748736.0, + "23": 397748736.0, + "24": 397748736.0, + "25": 397748736.0, + "26": 397748736.0, + "27": 397748736.0, + "28": 397748736.0, + "29": 397748736.0, + "30": 397748736.0, + "31": 397748736.0, + "32": 397748736.0, + "33": 397748736.0, + "34": 397748736.0, + "35": 397748736.0, + "36": 397748736.0, + "37": 397748736.0, + "38": 397748736.0, + "39": 397748736.0, + "40": 397748736.0, + "41": 397748736.0, + "42": 397748736.0, + "43": 397748736.0, + "44": 397748736.0, + "45": 397748736.0, + "46": 397748736.0, + "47": 397748736.0, + "48": 397748736.0, + "49": 397748736.0, + "50": 397748736.0, + "51": 397748736.0, + "52": 397748736.0, + "53": 397748736.0, + "54": 397748736.0, + "55": 397748736.0, + "56": 397748736.0, + "57": 397748736.0, + "58": 397748736.0, + "59": 397748736.0, + "60": 397748736.0, + "61": 397748736.0, + "62": 397748736.0, + "63": 397748736.0, + "64": 397748736.0, + "65": 397748736.0, + "66": 397748736.0, + "67": 397748736.0, + "68": 397748736.0, + "69": 397748736.0, + "70": 397748736.0, + "71": 397748736.0, + "72": 397748736.0, + "73": 397748736.0, + "74": 397748736.0, + "75": 397748736.0, + "76": 397748736.0, + "77": 397748736.0, + "78": 397748736.0, + "79": 397748736.0, + "80": 397748736.0, + "81": 397748736.0, + "82": 397748736.0, + "83": 397748736.0, + "84": 397748736.0, + "85": 397748736.0, + "86": 397748736.0, + "87": 397748736.0, + "88": 397748736.0, + "89": 397748736.0, + "90": 397748736.0, + "91": 397748736.0, + "92": 397748736.0, + "93": 397748736.0, + "94": 397748736.0, + "95": 397748736.0, + "96": 397748736.0, + "97": 397748736.0, + "98": 397748736.0, + "99": 397748736.0, + "100": 397748736.0 + } + }, + "mem-max-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 1053147648.0, + "2": 1185178624.0, + "3": 1185178624.0, + "4": 1185178624.0, + "5": 1185178624.0, + "6": 1185178624.0, + "7": 1185178624.0, + "8": 1185178624.0, + "9": 1185178624.0, + "10": 1185178624.0, + "11": 1185178624.0, + "12": 1185178624.0, + "13": 1185178624.0, + "14": 1185178624.0, + "15": 1185178624.0, + "16": 1185178624.0, + "17": 1185178624.0, + "18": 1185178624.0, + "19": 1185178624.0, + "20": 1185178624.0, + "21": 1185178624.0, + "22": 1185178624.0, + "23": 1185178624.0, + "24": 1185178624.0, + "25": 1185178624.0, + "26": 1185178624.0, + "27": 1185178624.0, + "28": 1185178624.0, + "29": 1185178624.0, + "30": 1185178624.0, + "31": 1185178624.0, + "32": 1185178624.0, + "33": 1185178624.0, + "34": 1185178624.0, + "35": 1185178624.0, + "36": 1185178624.0, + "37": 1185178624.0, + "38": 1185178624.0, + "39": 1185178624.0, + "40": 1185178624.0, + "41": 1185178624.0, + "42": 1185178624.0, + "43": 1185178624.0, + "44": 1185178624.0, + "45": 1185178624.0, + "46": 1185178624.0, + "47": 1185178624.0, + "48": 1185178624.0, + "49": 1185178624.0, + "50": 1185178624.0, + "51": 1185178624.0, + "52": 1185178624.0, + "53": 1185178624.0, + "54": 1185178624.0, + "55": 1185178624.0, + "56": 1185178624.0, + "57": 1185178624.0, + "58": 1185178624.0, + "59": 1185178624.0, + "60": 1185178624.0, + "61": 1185178624.0, + "62": 1185178624.0, + "63": 1185178624.0, + "64": 1185178624.0, + "65": 1185178624.0, + "66": 1185178624.0, + "67": 1185178624.0, + "68": 1185178624.0, + "69": 1185178624.0, + "70": 1185178624.0, + "71": 1185178624.0, + "72": 1185178624.0, + "73": 1185178624.0, + "74": 1185178624.0, + "75": 1185178624.0, + "76": 1185178624.0, + "77": 1185178624.0, + "78": 1185178624.0, + "79": 1185178624.0, + "80": 1185178624.0, + "81": 1185178624.0, + "82": 1185178624.0, + "83": 1185178624.0, + "84": 1185178624.0, + "85": 1185178624.0, + "86": 1185178624.0, + "87": 1185178624.0, + "88": 1185178624.0, + "89": 1185178624.0, + "90": 1185178624.0, + "91": 1185178624.0, + "92": 1185178624.0, + "93": 1185178624.0, + "94": 1185178624.0, + "95": 1185178624.0, + "96": 1185178624.0, + "97": 1185178624.0, + "98": 1185178624.0, + "99": 1185178624.0, + "100": 1185178624.0 + } + }, + "iteration-time": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": "nan", + "2": 4.73349, + "3": 2.15983, + "4": 1.88941, + "5": 1.57172, + "6": 2.14479, + "7": 1.6003, + "8": 2.13612, + "9": 2.30247, + "10": 1.92529, + "11": 1.96824, + "12": 1.86263, + "13": 1.89185, + "14": 2.17849, + "15": 1.6867, + "16": 1.76569, + "17": 2.30083, + "18": 1.68893, + "19": 1.93145, + "20": 1.74937, + "21": 2.17003, + "22": 1.57677, + "23": 2.04527, + "24": 1.62416, + "25": 2.19726, + "26": 2.24821, + "27": 1.64124, + "28": 1.77819, + "29": 1.63684, + "30": 1.60146, + "31": 1.60366, + "32": 2.68043, + "33": 1.44457, + "34": 1.52986, + "35": 2.24933, + "36": 2.00822, + "37": 1.79668, + "38": 1.47126, + "39": 2.17536, + "40": 1.77231, + "41": 1.89414, + "42": 2.40791, + "43": 1.60653, + "44": 1.95761, + "45": 2.177, + "46": 1.59332, + "47": 1.90437, + "48": 1.96553, + "49": 1.55566, + "50": 1.86143, + "51": 1.22312, + "52": 2.04217, + "53": 1.96848, + "54": 2.40717, + "55": 1.76452, + "56": 2.54219, + "57": 1.23046, + "58": 2.0047, + "59": 1.55501, + "60": 1.57163, + "61": 1.93399, + "62": 2.18865, + "63": 1.57273, + "64": 1.91969, + "65": 1.67746, + "66": 1.77917, + "67": 2.0028, + "68": 1.33153, + "69": 1.93831, + "70": 1.73118, + "71": 2.24164, + "72": 1.75068, + "73": 1.74409, + "74": 1.63697, + "75": 1.61654, + "76": 1.90661, + "77": 1.63289, + "78": 1.75427, + "79": 2.07391, + "80": 1.77912, + "81": 2.14573, + "82": 2.01988, + "83": 1.93264, + "84": 1.51221, + "85": 1.70984, + "86": 1.70674, + "87": 2.19525, + "88": 1.93034, + "89": 1.83377, + "90": 1.82253, + "91": 1.91076, + "92": 2.20521, + "93": 1.60272, + "94": 1.98571, + "95": 1.71085, + "96": 1.92934, + "97": 1.66744, + "98": 1.76841, + "99": 2.35658, + "100": 1.62165 + } + } +} \ No newline at end of file diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_resume_torch_dist_qk_layernorm_test_mode_1node/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_resume_torch_dist_qk_layernorm_test_mode_1node/model_config.yaml new file mode 100644 index 00000000000..addb335475f --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp1_resume_torch_dist_qk_layernorm_test_mode_1node/model_config.yaml @@ -0,0 +1,56 @@ +ENV_VARS: + CUDA_DEVICE_MAX_CONNECTIONS: 1 + NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0 + NCCL_ALGO: Ring + CUBLAS_WORKSPACE_CONFIG: :4096:8 +MODEL_ARGS: + --num-layers: 12 + --hidden-size: 512 + --num-attention-heads: 8 + --log-params-norm: true + --log-num-zeros-in-grad: true + --log-validation-ppl-to-tensorboard: true + --log-timers-to-tensorboard: true + --tensorboard-dir: ${TENSORBOARD_PATH} + --micro-batch-size: 4 + --global-batch-size: 32 + --seq-length: 1024 + --max-position-embeddings: 1024 + --train-iters: 100 + --timing-log-level: 0 + --lr-decay-iters: 320000 + --save: ${CHECKPOINT_SAVE_PATH} + --load: ${CHECKPOINT_LOAD_PATH} + --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document + --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json + --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt + --split: 949,50,1 + --distributed-backend: nccl + --lr: 0.00015 + --lr-decay-style: cosine + --min-lr: 1.0e-5 + --weight-decay: 1e-2 + --clip-grad: 1.0 + --lr-warmup-fraction: .01 + --log-interval: 1 + --save-interval: 50 + --eval-interval: 1000 + --eval-iters: 10 + --transformer-impl: transformer_engine + --tensor-model-parallel-size: 4 + --pipeline-model-parallel-size: 1 + --qk-layernorm: true + --test-mode: true + --deterministic-mode: true + --no-gradient-accumulation-fusion: true + --attention-softmax-in-fp32: true + --use-checkpoint-opt_param-scheduler: true + --use-mcore-models: true + --ckpt-format: torch_dist + --dist-ckpt-strictness: log_all # backward compatibility for TE changes + --data-cache-path: ${DATA_CACHE_PATH} + --bf16: true + --log-memory-to-tensorboard: true + --async-save: true + --use-persistent-ckpt-worker: true +TEST_TYPE: ckpt-resume diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp2_resume_torch_dist_reshard_8x1xNone_1node/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp2_resume_torch_dist_reshard_8x1xNone_1node/golden_values_dev_dgx_gb200.json new file mode 100644 index 00000000000..e80138a7f38 --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp2_resume_torch_dist_reshard_8x1xNone_1node/golden_values_dev_dgx_gb200.json @@ -0,0 +1,537 @@ +{ + "lm loss": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 10.89526, + "2": 10.88355, + "3": 10.88128, + "4": 10.88343, + "5": 10.88378, + "6": 10.87665, + "7": 10.8776, + "8": 10.88243, + "9": 10.88104, + "10": 10.87976, + "11": 10.87612, + "12": 10.8706, + "13": 10.86686, + "14": 10.8653, + "15": 10.84172, + "16": 10.83554, + "17": 10.84619, + "18": 10.82135, + "19": 10.84194, + "20": 10.7625, + "21": 10.75146, + "22": 10.73818, + "23": 10.73178, + "24": 10.70504, + "25": 10.69726, + "26": 10.6866, + "27": 10.65455, + "28": 10.59238, + "29": 10.55656, + "30": 10.53377, + "31": 10.52445, + "32": 10.51891, + "33": 10.48052, + "34": 10.45428, + "35": 10.45581, + "36": 10.42805, + "37": 10.3973, + "38": 10.4016, + "39": 10.37019, + "40": 10.36069, + "41": 10.33395, + "42": 10.31364, + "43": 10.29323, + "44": 10.27037, + "45": 10.27551, + "46": 10.24211, + "47": 10.22779, + "48": 10.18381, + "49": 10.18505, + "50": 10.18346, + "51": 10.18244, + "52": 10.14059, + "53": 10.14631, + "54": 10.11036, + "55": 10.08404, + "56": 10.11039, + "57": 10.09876, + "58": 10.11426, + "59": 10.06228, + "60": 10.08257, + "61": 10.03026, + "62": 10.00388, + "63": 10.07451, + "64": 10.03122, + "65": 10.0097, + "66": 10.03108, + "67": 10.00894, + "68": 9.9699, + "69": 9.99453, + "70": 9.97148, + "71": 10.0009, + "72": 9.98163, + "73": 9.97309, + "74": 9.95746, + "75": 9.93383, + "76": 9.96606, + "77": 9.95605, + "78": 9.91127, + "79": 9.9187, + "80": 9.92948, + "81": 9.95629, + "82": 9.88768, + "83": 9.85651, + "84": 9.79714, + "85": 9.78873, + "86": 9.87926, + "87": 9.90317, + "88": 9.87692, + "89": 9.82291, + "90": 9.81186, + "91": 9.82783, + "92": 9.81666, + "93": 9.75521, + "94": 9.82396, + "95": 9.82067, + "96": 9.8038, + "97": 9.74973, + "98": 9.77442, + "99": 9.81978, + "100": 9.71258 + } + }, + "num-zeros": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 1838.0, + "2": 1877.0, + "3": 1796.0, + "4": 1894.0, + "5": 1929.0, + "6": 1909.0, + "7": 1956.0, + "8": 1766.0, + "9": 1837.0, + "10": 1833.0, + "11": 1781.0, + "12": 1755.0, + "13": 1771.0, + "14": 1973.0, + "15": 1796.0, + "16": 1952.0, + "17": 1849.0, + "18": 1823.0, + "19": 1696.0, + "20": 1841.0, + "21": 1966.0, + "22": 1728.0, + "23": 1863.0, + "24": 1854.0, + "25": 1716.0, + "26": 1828.0, + "27": 1872.0, + "28": 1856.0, + "29": 1978.0, + "30": 1844.0, + "31": 1988.0, + "32": 1996.0, + "33": 2020.0, + "34": 2130.0, + "35": 2131.0, + "36": 2073.0, + "37": 2244.0, + "38": 2173.0, + "39": 2248.0, + "40": 2251.0, + "41": 2302.0, + "42": 1985.0, + "43": 2438.0, + "44": 2240.0, + "45": 2536.0, + "46": 2505.0, + "47": 2402.0, + "48": 2648.0, + "49": 2867.0, + "50": 2623.0, + "51": 2449.0, + "52": 2763.0, + "53": 2608.0, + "54": 2896.0, + "55": 2658.0, + "56": 2761.0, + "57": 2170.0, + "58": 3828.0, + "59": 3085.0, + "60": 2981.0, + "61": 2772.0, + "62": 3293.0, + "63": 3355.0, + "64": 3634.0, + "65": 2690.0, + "66": 3153.0, + "67": 3968.0, + "68": 3612.0, + "69": 3020.0, + "70": 3399.0, + "71": 3246.0, + "72": 3000.0, + "73": 3423.0, + "74": 3421.0, + "75": 3201.0, + "76": 3197.0, + "77": 3734.0, + "78": 3285.0, + "79": 3326.0, + "80": 3154.0, + "81": 3544.0, + "82": 2831.0, + "83": 2924.0, + "84": 2798.0, + "85": 2767.0, + "86": 3253.0, + "87": 2998.0, + "88": 3150.0, + "89": 3087.0, + "90": 3902.0, + "91": 2954.0, + "92": 2983.0, + "93": 3117.0, + "94": 3168.0, + "95": 3276.0, + "96": 3532.0, + "97": 3564.0, + "98": 3563.0, + "99": 3320.0, + "100": 3435.0 + } + }, + "mem-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 348123648.0, + "2": 348123648.0, + "3": 348123648.0, + "4": 348123648.0, + "5": 348123648.0, + "6": 348123648.0, + "7": 348123648.0, + "8": 348123648.0, + "9": 348123648.0, + "10": 348123648.0, + "11": 348123648.0, + "12": 348123648.0, + "13": 348123648.0, + "14": 348123648.0, + "15": 348123648.0, + "16": 348123648.0, + "17": 348123648.0, + "18": 348123648.0, + "19": 348123648.0, + "20": 348123648.0, + "21": 348123648.0, + "22": 348123648.0, + "23": 348123648.0, + "24": 348123648.0, + "25": 348123648.0, + "26": 348123648.0, + "27": 348123648.0, + "28": 348123648.0, + "29": 348123648.0, + "30": 348123648.0, + "31": 348123648.0, + "32": 348123648.0, + "33": 348123648.0, + "34": 348123648.0, + "35": 348123648.0, + "36": 348123648.0, + "37": 348123648.0, + "38": 348123648.0, + "39": 348123648.0, + "40": 348123648.0, + "41": 348123648.0, + "42": 348123648.0, + "43": 348123648.0, + "44": 348123648.0, + "45": 348123648.0, + "46": 348123648.0, + "47": 348123648.0, + "48": 348123648.0, + "49": 348123648.0, + "50": 348123648.0, + "51": 348123648.0, + "52": 348123648.0, + "53": 348123648.0, + "54": 348123648.0, + "55": 348123648.0, + "56": 348123648.0, + "57": 348123648.0, + "58": 348123648.0, + "59": 348123648.0, + "60": 348123648.0, + "61": 348123648.0, + "62": 348123648.0, + "63": 348123648.0, + "64": 348123648.0, + "65": 348123648.0, + "66": 348123648.0, + "67": 348123648.0, + "68": 348123648.0, + "69": 348123648.0, + "70": 348123648.0, + "71": 348123648.0, + "72": 348123648.0, + "73": 348123648.0, + "74": 348123648.0, + "75": 348123648.0, + "76": 348123648.0, + "77": 348123648.0, + "78": 348123648.0, + "79": 348123648.0, + "80": 348123648.0, + "81": 348123648.0, + "82": 348123648.0, + "83": 348123648.0, + "84": 348123648.0, + "85": 348123648.0, + "86": 348123648.0, + "87": 348123648.0, + "88": 348123648.0, + "89": 348123648.0, + "90": 348123648.0, + "91": 348123648.0, + "92": 348123648.0, + "93": 348123648.0, + "94": 348123648.0, + "95": 348123648.0, + "96": 348123648.0, + "97": 348123648.0, + "98": 348123648.0, + "99": 348123648.0, + "100": 348123648.0 + } + }, + "mem-max-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 957041152.0, + "2": 1087323648.0, + "3": 1087323648.0, + "4": 1087323648.0, + "5": 1087323648.0, + "6": 1087323648.0, + "7": 1087323648.0, + "8": 1087323648.0, + "9": 1087323648.0, + "10": 1087323648.0, + "11": 1087323648.0, + "12": 1087323648.0, + "13": 1087323648.0, + "14": 1087323648.0, + "15": 1087323648.0, + "16": 1087323648.0, + "17": 1087323648.0, + "18": 1087323648.0, + "19": 1087323648.0, + "20": 1087323648.0, + "21": 1087323648.0, + "22": 1087323648.0, + "23": 1087323648.0, + "24": 1087323648.0, + "25": 1087323648.0, + "26": 1087323648.0, + "27": 1087323648.0, + "28": 1087323648.0, + "29": 1087323648.0, + "30": 1087323648.0, + "31": 1087323648.0, + "32": 1087323648.0, + "33": 1087323648.0, + "34": 1087323648.0, + "35": 1087323648.0, + "36": 1087323648.0, + "37": 1087323648.0, + "38": 1087323648.0, + "39": 1087323648.0, + "40": 1087323648.0, + "41": 1087323648.0, + "42": 1087323648.0, + "43": 1087323648.0, + "44": 1087323648.0, + "45": 1087323648.0, + "46": 1087323648.0, + "47": 1087323648.0, + "48": 1087323648.0, + "49": 1087323648.0, + "50": 1087323648.0, + "51": 1087323648.0, + "52": 1087323648.0, + "53": 1087323648.0, + "54": 1087323648.0, + "55": 1087323648.0, + "56": 1087323648.0, + "57": 1087323648.0, + "58": 1087323648.0, + "59": 1087323648.0, + "60": 1087323648.0, + "61": 1087323648.0, + "62": 1087323648.0, + "63": 1087323648.0, + "64": 1087323648.0, + "65": 1087323648.0, + "66": 1087323648.0, + "67": 1087323648.0, + "68": 1087323648.0, + "69": 1087323648.0, + "70": 1087323648.0, + "71": 1087323648.0, + "72": 1087323648.0, + "73": 1087323648.0, + "74": 1087323648.0, + "75": 1087323648.0, + "76": 1087323648.0, + "77": 1087323648.0, + "78": 1087323648.0, + "79": 1087323648.0, + "80": 1087323648.0, + "81": 1087323648.0, + "82": 1087323648.0, + "83": 1087323648.0, + "84": 1087323648.0, + "85": 1087323648.0, + "86": 1087323648.0, + "87": 1087323648.0, + "88": 1087323648.0, + "89": 1087323648.0, + "90": 1087323648.0, + "91": 1087323648.0, + "92": 1087323648.0, + "93": 1087323648.0, + "94": 1087323648.0, + "95": 1087323648.0, + "96": 1087323648.0, + "97": 1087323648.0, + "98": 1087323648.0, + "99": 1087323648.0, + "100": 1087323648.0 + } + }, + "iteration-time": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": "nan", + "2": 5.33383, + "3": 2.08622, + "4": 1.77734, + "5": 1.55199, + "6": 1.95304, + "7": 1.51306, + "8": 2.09998, + "9": 2.13872, + "10": 1.81982, + "11": 2.02226, + "12": 1.72473, + "13": 1.99756, + "14": 2.27508, + "15": 1.85465, + "16": 2.10279, + "17": 2.66852, + "18": 1.89021, + "19": 2.30676, + "20": 2.00722, + "21": 2.35738, + "22": 1.95565, + "23": 2.31878, + "24": 1.75527, + "25": 2.36271, + "26": 2.64549, + "27": 1.99175, + "28": 1.97844, + "29": 1.6433, + "30": 1.91792, + "31": 1.7799, + "32": 3.00541, + "33": 1.78776, + "34": 1.57537, + "35": 2.28932, + "36": 2.30836, + "37": 1.96596, + "38": 1.52181, + "39": 2.42981, + "40": 2.08329, + "41": 2.1796, + "42": 2.62047, + "43": 1.88805, + "44": 1.78893, + "45": 2.24602, + "46": 1.68018, + "47": 1.80837, + "48": 2.12482, + "49": 1.56454, + "50": 1.85997, + "51": 1.17807, + "52": 2.10459, + "53": 2.16294, + "54": 2.31912, + "55": 2.01947, + "56": 2.70614, + "57": 1.44101, + "58": 2.11397, + "59": 2.04734, + "60": 1.86782, + "61": 2.10854, + "62": 2.44403, + "63": 1.64686, + "64": 1.91434, + "65": 1.81729, + "66": 2.00572, + "67": 2.05745, + "68": 1.44922, + "69": 1.94902, + "70": 1.89487, + "71": 2.25102, + "72": 1.73557, + "73": 1.85327, + "74": 1.75185, + "75": 1.76643, + "76": 1.86109, + "77": 1.90515, + "78": 1.9862, + "79": 2.28594, + "80": 1.92388, + "81": 2.39685, + "82": 2.27414, + "83": 2.20932, + "84": 1.73248, + "85": 1.76128, + "86": 1.805, + "87": 2.18071, + "88": 1.97452, + "89": 1.89156, + "90": 1.85672, + "91": 1.95745, + "92": 2.29995, + "93": 1.59425, + "94": 1.97124, + "95": 1.83215, + "96": 1.97626, + "97": 1.63251, + "98": 1.8657, + "99": 2.42793, + "100": 1.66972 + } + } +} \ No newline at end of file diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp2_resume_torch_dist_reshard_8x1xNone_1node/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp2_resume_torch_dist_reshard_8x1xNone_1node/model_config.yaml new file mode 100644 index 00000000000..630da63ed62 --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_te_tp4_pp2_resume_torch_dist_reshard_8x1xNone_1node/model_config.yaml @@ -0,0 +1,54 @@ +ENV_VARS: + CUDA_DEVICE_MAX_CONNECTIONS: 1 + NVTE_ALLOW_NONDETERMINISTIC_ALGO: 1 +MODEL_ARGS: + --num-layers: 12 + --hidden-size: 512 + --num-attention-heads: 8 + --log-params-norm: true + --log-num-zeros-in-grad: true + --log-validation-ppl-to-tensorboard: true + --log-timers-to-tensorboard: true + --tensorboard-dir: ${TENSORBOARD_PATH} + --micro-batch-size: 4 + --global-batch-size: 32 + --seq-length: 1024 + --max-position-embeddings: 1024 + --train-iters: 100 + --timing-log-level: 0 + --lr-decay-iters: 320000 + --save: ${CHECKPOINT_SAVE_PATH} + --load: ${CHECKPOINT_LOAD_PATH} + --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document + --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json + --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt + --split: 949,50,1 + --distributed-backend: nccl + --lr: 0.00015 + --lr-decay-style: cosine + --min-lr: 1.0e-5 + --weight-decay: 1e-2 + --clip-grad: 1.0 + --lr-warmup-fraction: .01 + --log-interval: 1 + --save-interval: 50 + --eval-interval: 1000 + --eval-iters: 10 + --transformer-impl: transformer_engine + --tensor-model-parallel-size: 4 + --pipeline-model-parallel-size: 1 + --use-distributed-optimizer: true + --async-save: true + --ckpt-fully-parallel-save: true + --no-gradient-accumulation-fusion: true + --attention-softmax-in-fp32: true + --use-checkpoint-opt_param-scheduler: true + --use-mcore-models: true + --ckpt-format: torch_dist + --dist-ckpt-optim-fully-reshardable: true + --dist-ckpt-strictness: log_all # backward compatibility for TE changes + --data-cache-path: ${DATA_CACHE_PATH} + --bf16: true + --log-memory-to-tensorboard: true + --use-persistent-ckpt-worker: true +TEST_TYPE: ckpt-resume diff --git a/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp2_pp2_uninstall_te_1node/model_config.yaml b/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp2_pp2_uninstall_te_1node/model_config.yaml new file mode 100644 index 00000000000..d44645ba5e3 --- /dev/null +++ b/tests/functional_tests/test_cases/gpt/gpt3_mcore_tp2_pp2_uninstall_te_1node/model_config.yaml @@ -0,0 +1,59 @@ +ENV_VARS: + CUDA_DEVICE_MAX_CONNECTIONS: 1 + NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0 + NCCL_ALGO: Ring + CUBLAS_WORKSPACE_CONFIG: :4096:8 + SKIP_PYTEST: 1 +BEFORE_SCRIPT: | + pip uninstall -y transformer_engine || true + pip uninstall -y Apex || true ## TODO: remove once Apex dependency has been removed completely +MODEL_ARGS: + --num-layers: 12 + --hidden-size: 512 + --num-attention-heads: 8 + --log-params-norm: true + --log-num-zeros-in-grad: true + --log-validation-ppl-to-tensorboard: true + --log-timers-to-tensorboard: true + --tensorboard-dir: ${TENSORBOARD_PATH} + --micro-batch-size: 4 + --global-batch-size: 32 + --seq-length: 1024 + --max-position-embeddings: 1024 + --train-iters: 50 + --timing-log-level: 0 + --lr-decay-iters: 320000 + --save: ${CHECKPOINT_SAVE_PATH} + --load: ${CHECKPOINT_LOAD_PATH} + --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document + --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json + --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt + --split: 949,50,1 + --distributed-backend: nccl + --lr: 0.00015 + --lr-decay-style: cosine + --min-lr: 1.0e-5 + --weight-decay: 1e-2 + --clip-grad: 1.0 + --lr-warmup-fraction: .01 + --log-interval: 1 + --save-interval: 10000 + --eval-interval: 1000 + --eval-iters: 10 + --transformer-impl: local + --tensor-model-parallel-size: 2 + --pipeline-model-parallel-size: 2 + --no-persist-layer-norm: true + --no-masked-softmax-fusion: true + --deterministic-mode: true + --no-gradient-accumulation-fusion: true + --use-mcore-models: true + --ckpt-format: torch_dist + --dist-ckpt-strictness: log_all # backward compatibility for TE changes + --data-cache-path: ${DATA_CACHE_PATH} + --fp16: true + --apply-query-key-layer-scaling: true + --log-memory-to-tensorboard: true + --async-save: true + --use-persistent-ckpt-worker: true +TEST_TYPE: regular diff --git a/tests/functional_tests/test_cases/hybrid/hybrid_dynamic_inference_tp1_pp1_dp8_583m/golden_values_dev_dgx_h100.json b/tests/functional_tests/test_cases/hybrid/hybrid_dynamic_inference_tp1_pp1_dp8_583m/golden_values_dev_dgx_h100.json index f126da7402c..df80109e0dc 100644 --- a/tests/functional_tests/test_cases/hybrid/hybrid_dynamic_inference_tp1_pp1_dp8_583m/golden_values_dev_dgx_h100.json +++ b/tests/functional_tests/test_cases/hybrid/hybrid_dynamic_inference_tp1_pp1_dp8_583m/golden_values_dev_dgx_h100.json @@ -34,8 +34,8 @@ 1261, 15353 ], - "latency": 1.7664601802825928, - "ttft": 0.09903407096862793, + "latency": 2.3884456157684326, + "ttft": 0.10706686973571777, "cuda_graph_request_count_map": null, "step_count": 30, "top_n_logprobs": null, diff --git a/tests/functional_tests/test_cases/moe/deepseek_proxy_fsdp_ep2_fsdp2_1node/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/moe/deepseek_proxy_fsdp_ep2_fsdp2_1node/golden_values_dev_dgx_gb200.json index f3be3ae48a3..c825c2f98ce 100644 --- a/tests/functional_tests/test_cases/moe/deepseek_proxy_fsdp_ep2_fsdp2_1node/golden_values_dev_dgx_gb200.json +++ b/tests/functional_tests/test_cases/moe/deepseek_proxy_fsdp_ep2_fsdp2_1node/golden_values_dev_dgx_gb200.json @@ -233,55 +233,55 @@ "step_interval": 1, "values": { "1": "nan", - "2": 7.9241, - "3": 0.14054, - "4": 0.14617, - "5": 0.13812, - "6": 0.13462, - "7": 0.13823, - "8": 0.12549, - "9": 0.13315, - "10": 0.13723, - "11": 0.13167, - "12": 0.12839, - "13": 0.12284, - "14": 0.12524, - "15": 0.13108, - "16": 0.12445, - "17": 0.12506, - "18": 0.1217, - "19": 0.12602, - "20": 0.12442, - "21": 0.12158, - "22": 0.12384, - "23": 0.12268, - "24": 0.1232, - "25": 0.11859, - "26": 0.12908, - "27": 0.12049, - "28": 0.12893, - "29": 0.11972, - "30": 0.1193, - "31": 0.12398, - "32": 0.11942, - "33": 0.1224, - "34": 0.11969, - "35": 0.12075, - "36": 0.12532, - "37": 0.12074, - "38": 0.11755, - "39": 0.11831, - "40": 0.12014, - "41": 0.12022, - "42": 0.12347, - "43": 0.12316, - "44": 0.12387, - "45": 0.12015, - "46": 0.11998, - "47": 0.11701, - "48": 0.11954, - "49": 0.1272, - "50": 0.12168 + "2": 8.05941, + "3": 0.14914, + "4": 0.14573, + "5": 0.14253, + "6": 0.13588, + "7": 0.14273, + "8": 0.12487, + "9": 0.13154, + "10": 0.13251, + "11": 0.13189, + "12": 0.12689, + "13": 0.127, + "14": 0.12702, + "15": 0.13138, + "16": 0.12836, + "17": 0.12972, + "18": 0.12581, + "19": 0.12707, + "20": 0.12675, + "21": 0.12105, + "22": 0.12349, + "23": 0.12498, + "24": 0.12617, + "25": 0.1226, + "26": 0.13116, + "27": 0.12186, + "28": 0.12987, + "29": 0.12199, + "30": 0.1191, + "31": 0.13055, + "32": 0.12292, + "33": 0.1211, + "34": 0.12095, + "35": 0.12517, + "36": 0.12585, + "37": 0.12497, + "38": 0.12145, + "39": 0.12709, + "40": 0.12078, + "41": 0.1203, + "42": 0.12413, + "43": 0.12297, + "44": 0.12183, + "45": 0.12411, + "46": 0.12497, + "47": 0.12098, + "48": 0.12062, + "49": 0.12456, + "50": 0.12174 } } } \ No newline at end of file diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp1_pp2_resume_torch_dist_reshard_2x1x4_te_8experts2parallel_dist_optimizer_1node/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp1_pp2_resume_torch_dist_reshard_2x1x4_te_8experts2parallel_dist_optimizer_1node/golden_values_dev_dgx_gb200.json new file mode 100644 index 00000000000..c3784e4ea95 --- /dev/null +++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp1_pp2_resume_torch_dist_reshard_2x1x4_te_8experts2parallel_dist_optimizer_1node/golden_values_dev_dgx_gb200.json @@ -0,0 +1,537 @@ +{ + "lm loss": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 10.93377, + "2": 10.92607, + "3": 10.92767, + "4": 10.92453, + "5": 10.9276, + "6": 10.92131, + "7": 10.92152, + "8": 10.9216, + "9": 10.92891, + "10": 10.92274, + "11": 10.91482, + "12": 10.9179, + "13": 10.91931, + "14": 10.9049, + "15": 10.89547, + "16": 10.88092, + "17": 10.89674, + "18": 10.88076, + "19": 10.881, + "20": 10.82438, + "21": 10.82525, + "22": 10.8137, + "23": 10.80432, + "24": 10.77737, + "25": 10.77677, + "26": 10.76095, + "27": 10.7548, + "28": 10.70135, + "29": 10.66806, + "30": 10.64356, + "31": 10.62623, + "32": 10.61876, + "33": 10.58643, + "34": 10.55239, + "35": 10.5495, + "36": 10.53104, + "37": 10.5024, + "38": 10.51075, + "39": 10.47152, + "40": 10.45474, + "41": 10.43256, + "42": 10.41866, + "43": 10.40232, + "44": 10.36574, + "45": 10.37376, + "46": 10.33992, + "47": 10.32368, + "48": 10.27298, + "49": 10.26663, + "50": 10.27906, + "51": 10.27133, + "52": 10.22994, + "53": 10.23144, + "54": 10.20189, + "55": 10.16916, + "56": 10.19596, + "57": 10.17322, + "58": 10.1908, + "59": 10.13762, + "60": 10.1475, + "61": 10.10791, + "62": 10.07631, + "63": 10.1365, + "64": 10.09569, + "65": 10.06841, + "66": 10.09405, + "67": 10.06682, + "68": 10.03287, + "69": 10.05337, + "70": 10.03913, + "71": 10.05443, + "72": 10.03596, + "73": 10.03662, + "74": 10.01818, + "75": 9.98747, + "76": 10.01611, + "77": 10.01346, + "78": 9.96201, + "79": 9.97655, + "80": 9.99362, + "81": 10.01457, + "82": 9.94848, + "83": 9.92998, + "84": 9.85406, + "85": 9.85383, + "86": 9.94233, + "87": 9.97246, + "88": 9.94695, + "89": 9.87772, + "90": 9.87819, + "91": 9.89905, + "92": 9.88148, + "93": 9.81813, + "94": 9.89313, + "95": 9.87741, + "96": 9.86001, + "97": 9.80194, + "98": 9.83796, + "99": 9.87556, + "100": 9.76742 + } + }, + "num-zeros": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 36891.0, + "2": 37810.0, + "3": 37270.0, + "4": 37438.0, + "5": 36860.0, + "6": 35806.0, + "7": 36797.0, + "8": 37014.0, + "9": 37425.0, + "10": 38208.0, + "11": 37028.0, + "12": 36359.0, + "13": 37200.0, + "14": 37135.0, + "15": 36691.0, + "16": 36560.0, + "17": 37516.0, + "18": 37037.0, + "19": 37618.0, + "20": 37100.0, + "21": 37005.0, + "22": 37272.0, + "23": 37127.0, + "24": 37850.0, + "25": 35977.0, + "26": 37134.0, + "27": 37184.0, + "28": 36000.0, + "29": 36028.0, + "30": 36332.0, + "31": 36439.0, + "32": 36840.0, + "33": 35943.0, + "34": 35991.0, + "35": 36498.0, + "36": 35209.0, + "37": 36195.0, + "38": 35967.0, + "39": 36246.0, + "40": 37009.0, + "41": 36324.0, + "42": 35830.0, + "43": 37365.0, + "44": 36616.0, + "45": 39294.0, + "46": 38011.0, + "47": 38023.0, + "48": 38849.0, + "49": 42175.0, + "50": 39256.0, + "51": 39023.0, + "52": 41404.0, + "53": 39891.0, + "54": 41452.0, + "55": 39372.0, + "56": 40729.0, + "57": 36640.0, + "58": 45195.0, + "59": 42384.0, + "60": 42241.0, + "61": 41611.0, + "62": 44445.0, + "63": 44011.0, + "64": 50825.0, + "65": 42425.0, + "66": 45407.0, + "67": 50074.0, + "68": 47147.0, + "69": 45615.0, + "70": 47545.0, + "71": 45786.0, + "72": 44756.0, + "73": 47249.0, + "74": 46468.0, + "75": 48679.0, + "76": 48214.0, + "77": 48135.0, + "78": 49631.0, + "79": 47439.0, + "80": 45773.0, + "81": 54184.0, + "82": 41700.0, + "83": 46683.0, + "84": 45345.0, + "85": 46248.0, + "86": 45653.0, + "87": 35991.0, + "88": 45712.0, + "89": 47699.0, + "90": 58939.0, + "91": 39655.0, + "92": 49405.0, + "93": 45527.0, + "94": 42503.0, + "95": 44707.0, + "96": 50480.0, + "97": 46166.0, + "98": 49395.0, + "99": 44894.0, + "100": 47544.0 + } + }, + "mem-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 1362933248.0, + "2": 1362933248.0, + "3": 1362932224.0, + "4": 1362932736.0, + "5": 1363366400.0, + "6": 1362934272.0, + "7": 1362933248.0, + "8": 1362931712.0, + "9": 1362933248.0, + "10": 1362933248.0, + "11": 1362933760.0, + "12": 1362934272.0, + "13": 1362933760.0, + "14": 1362934272.0, + "15": 1362933760.0, + "16": 1362932736.0, + "17": 1362934272.0, + "18": 1362933248.0, + "19": 1362932736.0, + "20": 1362933248.0, + "21": 1362932736.0, + "22": 1362932736.0, + "23": 1362931712.0, + "24": 1362932736.0, + "25": 1362932736.0, + "26": 1362931712.0, + "27": 1362932736.0, + "28": 1362933248.0, + "29": 1362932224.0, + "30": 1362932736.0, + "31": 1362932736.0, + "32": 1362934272.0, + "33": 1362930688.0, + "34": 1362932224.0, + "35": 1362931200.0, + "36": 1362932224.0, + "37": 1362933760.0, + "38": 1362933248.0, + "39": 1362932736.0, + "40": 1362931200.0, + "41": 1362930176.0, + "42": 1362933760.0, + "43": 1362932224.0, + "44": 1362931200.0, + "45": 1362933760.0, + "46": 1362957312.0, + "47": 1362932736.0, + "48": 1363489792.0, + "49": 1362931712.0, + "50": 1362932736.0, + "51": 1362932224.0, + "52": 1362931200.0, + "53": 1362934784.0, + "54": 1362932736.0, + "55": 1362931200.0, + "56": 1362932224.0, + "57": 1362932736.0, + "58": 1362932224.0, + "59": 1362932736.0, + "60": 1362931712.0, + "61": 1362931200.0, + "62": 1362933760.0, + "63": 1362932736.0, + "64": 1362931200.0, + "65": 1362932224.0, + "66": 1362932224.0, + "67": 1362930688.0, + "68": 1362932224.0, + "69": 1362931712.0, + "70": 1362932224.0, + "71": 1362932736.0, + "72": 1362931200.0, + "73": 1362927616.0, + "74": 1362929152.0, + "75": 1362929152.0, + "76": 1362930688.0, + "77": 1362930176.0, + "78": 1362931200.0, + "79": 1362930688.0, + "80": 1362930688.0, + "81": 1362926080.0, + "82": 1362926592.0, + "83": 1362932224.0, + "84": 1362927104.0, + "85": 1362926080.0, + "86": 1362924544.0, + "87": 1362926592.0, + "88": 1362930176.0, + "89": 1362925056.0, + "90": 1362924544.0, + "91": 1362927616.0, + "92": 1362918400.0, + "93": 1362928640.0, + "94": 1362924544.0, + "95": 1362920448.0, + "96": 1362930176.0, + "97": 1362914816.0, + "98": 1362916864.0, + "99": 1362922496.0, + "100": 1362912256.0 + } + }, + "mem-max-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 2641302016.0, + "2": 3174163968.0, + "3": 3174571008.0, + "4": 3174571008.0, + "5": 3175993856.0, + "6": 3175993856.0, + "7": 3175993856.0, + "8": 3175993856.0, + "9": 3175993856.0, + "10": 3175993856.0, + "11": 3175993856.0, + "12": 3175993856.0, + "13": 3175993856.0, + "14": 3175993856.0, + "15": 3175993856.0, + "16": 3175993856.0, + "17": 3175993856.0, + "18": 3175993856.0, + "19": 3175993856.0, + "20": 3175993856.0, + "21": 3175993856.0, + "22": 3175993856.0, + "23": 3175993856.0, + "24": 3175993856.0, + "25": 3175993856.0, + "26": 3175993856.0, + "27": 3175993856.0, + "28": 3175993856.0, + "29": 3175993856.0, + "30": 3175993856.0, + "31": 3175993856.0, + "32": 3175993856.0, + "33": 3175993856.0, + "34": 3175993856.0, + "35": 3175993856.0, + "36": 3175993856.0, + "37": 3175993856.0, + "38": 3175993856.0, + "39": 3175993856.0, + "40": 3175993856.0, + "41": 3175993856.0, + "42": 3175993856.0, + "43": 3175993856.0, + "44": 3175993856.0, + "45": 3175993856.0, + "46": 3175993856.0, + "47": 3175993856.0, + "48": 3175993856.0, + "49": 3175993856.0, + "50": 3175993856.0, + "51": 3175993856.0, + "52": 3175993856.0, + "53": 3175993856.0, + "54": 3175993856.0, + "55": 3175993856.0, + "56": 3175993856.0, + "57": 3175993856.0, + "58": 3175993856.0, + "59": 3175993856.0, + "60": 3175993856.0, + "61": 3175993856.0, + "62": 3175993856.0, + "63": 3175993856.0, + "64": 3175993856.0, + "65": 3175993856.0, + "66": 3175993856.0, + "67": 3175993856.0, + "68": 3175993856.0, + "69": 3175993856.0, + "70": 3175993856.0, + "71": 3175993856.0, + "72": 3175993856.0, + "73": 3175993856.0, + "74": 3175993856.0, + "75": 3175993856.0, + "76": 3175993856.0, + "77": 3175993856.0, + "78": 3175993856.0, + "79": 3175993856.0, + "80": 3175993856.0, + "81": 3175993856.0, + "82": 3175993856.0, + "83": 3175993856.0, + "84": 3175993856.0, + "85": 3175993856.0, + "86": 3175993856.0, + "87": 3175993856.0, + "88": 3175993856.0, + "89": 3175993856.0, + "90": 3175993856.0, + "91": 3175993856.0, + "92": 3175993856.0, + "93": 3175993856.0, + "94": 3175993856.0, + "95": 3175993856.0, + "96": 3175993856.0, + "97": 3175993856.0, + "98": 3175993856.0, + "99": 3175993856.0, + "100": 3175993856.0 + } + }, + "iteration-time": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": "nan", + "2": 6.00764, + "3": 0.91593, + "4": 0.77258, + "5": 0.63926, + "6": 0.59762, + "7": 0.6259, + "8": 0.95144, + "9": 1.0899, + "10": 0.46567, + "11": 0.80809, + "12": 0.8101, + "13": 0.5784, + "14": 1.04331, + "15": 0.51567, + "16": 0.6965, + "17": 0.76452, + "18": 1.17028, + "19": 0.71874, + "20": 0.94745, + "21": 1.02203, + "22": 0.86641, + "23": 0.86574, + "24": 1.23525, + "25": 0.43536, + "26": 0.94802, + "27": 0.56087, + "28": 0.71514, + "29": 0.6509, + "30": 0.39906, + "31": 0.60536, + "32": 1.15376, + "33": 0.81469, + "34": 0.38265, + "35": 1.01275, + "36": 1.00443, + "37": 0.96089, + "38": 0.4899, + "39": 1.06058, + "40": 1.14981, + "41": 0.67688, + "42": 0.97758, + "43": 0.87837, + "44": 1.01837, + "45": 0.45167, + "46": 0.68988, + "47": 0.92773, + "48": 0.87664, + "49": 0.76627, + "50": 0.74642, + "51": 0.38777, + "52": 0.84552, + "53": 0.92674, + "54": 1.08595, + "55": 0.95417, + "56": 0.81791, + "57": 0.62862, + "58": 1.14503, + "59": 0.65185, + "60": 0.81633, + "61": 0.84425, + "62": 0.92289, + "63": 0.62856, + "64": 0.97346, + "65": 0.64756, + "66": 0.73636, + "67": 0.62105, + "68": 0.69781, + "69": 1.0752, + "70": 0.57613, + "71": 0.9188, + "72": 0.86161, + "73": 0.91826, + "74": 0.57973, + "75": 0.88134, + "76": 0.68602, + "77": 1.00098, + "78": 1.03681, + "79": 1.33275, + "80": 0.81549, + "81": 0.90241, + "82": 0.69762, + "83": 0.78768, + "84": 0.63313, + "85": 0.65063, + "86": 0.66983, + "87": 1.15982, + "88": 0.72206, + "89": 0.64648, + "90": 1.22293, + "91": 0.74744, + "92": 1.24048, + "93": 0.6893, + "94": 0.946, + "95": 0.56594, + "96": 0.92383, + "97": 0.72069, + "98": 1.06283, + "99": 0.9351, + "100": 0.79317 + } + } +} \ No newline at end of file diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp1_pp2_resume_torch_dist_reshard_2x1x4_te_8experts2parallel_dist_optimizer_1node/model_config.yaml b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp1_pp2_resume_torch_dist_reshard_2x1x4_te_8experts2parallel_dist_optimizer_1node/model_config.yaml new file mode 100644 index 00000000000..e885bc255aa --- /dev/null +++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp1_pp2_resume_torch_dist_reshard_2x1x4_te_8experts2parallel_dist_optimizer_1node/model_config.yaml @@ -0,0 +1,60 @@ +ENV_VARS: + CUDA_DEVICE_MAX_CONNECTIONS: 1 + NVTE_ALLOW_NONDETERMINISTIC_ALGO: 1 +MODEL_ARGS: + --num-layers: 12 + --hidden-size: 512 + --num-attention-heads: 8 + --log-params-norm: true + --log-num-zeros-in-grad: true + --log-validation-ppl-to-tensorboard: true + --log-timers-to-tensorboard: true + --tensorboard-dir: ${TENSORBOARD_PATH} + --micro-batch-size: 4 + --global-batch-size: 32 + --seq-length: 1024 + --max-position-embeddings: 1024 + --train-iters: 100 + --timing-log-level: 0 + --lr-decay-iters: 320000 + --save: ${CHECKPOINT_SAVE_PATH} + --load: ${CHECKPOINT_LOAD_PATH} + --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document + --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json + --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt + --split: 949,50,1 + --distributed-backend: nccl + --lr: 0.00015 + --lr-decay-style: cosine + --min-lr: 1.0e-5 + --weight-decay: 1e-2 + --clip-grad: 1.0 + --lr-warmup-fraction: .01 + --log-interval: 1 + --save-interval: 50 + --eval-interval: 1000 + --eval-iters: 10 + --transformer-impl: transformer_engine + --tensor-model-parallel-size: 1 + --pipeline-model-parallel-size: 2 + --expert-model-parallel-size: 2 + --sequence-parallel: true + --num-experts: 8 + --use-distributed-optimizer: true + --moe-router-load-balancing-type: sinkhorn + --moe-router-topk: 1 + --ckpt-fully-parallel-save: true + --ckpt-fully-parallel-load: true + --no-gradient-accumulation-fusion: true + --attention-softmax-in-fp32: true + --use-checkpoint-opt_param-scheduler: true + --use-mcore-models: true + --ckpt-format: torch_dist + --dist-ckpt-strictness: log_all # backward compatibility for TE changes + --data-cache-path: ${DATA_CACHE_PATH} + --bf16: true + --disable-bias-linear: true + --no-bias-gelu-fusion: true + --log-memory-to-tensorboard: true +TEST_TYPE: ckpt-resume +LAUNCHER: ft_launcher diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_resume_torch_dist_te_8experts2parallel_dist_optimizer_1node/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_resume_torch_dist_te_8experts2parallel_dist_optimizer_1node/golden_values_dev_dgx_gb200.json new file mode 100644 index 00000000000..0b34c84f14b --- /dev/null +++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_resume_torch_dist_te_8experts2parallel_dist_optimizer_1node/golden_values_dev_dgx_gb200.json @@ -0,0 +1,537 @@ +{ + "lm loss": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 10.91066, + "2": 10.89646, + "3": 10.90792, + "4": 10.90307, + "5": 10.91195, + "6": 10.90646, + "7": 10.89838, + "8": 10.90122, + "9": 10.90026, + "10": 10.90331, + "11": 10.89166, + "12": 10.89988, + "13": 10.8849, + "14": 10.87998, + "15": 10.8728, + "16": 10.86504, + "17": 10.86869, + "18": 10.8571, + "19": 10.86119, + "20": 10.8188, + "21": 10.80335, + "22": 10.78966, + "23": 10.77604, + "24": 10.74885, + "25": 10.75645, + "26": 10.74017, + "27": 10.72496, + "28": 10.67041, + "29": 10.63542, + "30": 10.61272, + "31": 10.60548, + "32": 10.58806, + "33": 10.56518, + "34": 10.53433, + "35": 10.53075, + "36": 10.51437, + "37": 10.47627, + "38": 10.48915, + "39": 10.44952, + "40": 10.43444, + "41": 10.41455, + "42": 10.40199, + "43": 10.38762, + "44": 10.34257, + "45": 10.36188, + "46": 10.32335, + "47": 10.3067, + "48": 10.27172, + "49": 10.26262, + "50": 10.2613, + "51": 10.25781, + "52": 10.21493, + "53": 10.22583, + "54": 10.17883, + "55": 10.16505, + "56": 10.17871, + "57": 10.1692, + "58": 10.17718, + "59": 10.1302, + "60": 10.14386, + "61": 10.09356, + "62": 10.07265, + "63": 10.12749, + "64": 10.09284, + "65": 10.07125, + "66": 10.08262, + "67": 10.06175, + "68": 10.02605, + "69": 10.04908, + "70": 10.02759, + "71": 10.04607, + "72": 10.03034, + "73": 10.02624, + "74": 10.00701, + "75": 9.98506, + "76": 10.00662, + "77": 10.00431, + "78": 9.95602, + "79": 9.96789, + "80": 9.98225, + "81": 10.01113, + "82": 9.94293, + "83": 9.92061, + "84": 9.84992, + "85": 9.84827, + "86": 9.9353, + "87": 9.96845, + "88": 9.93373, + "89": 9.87283, + "90": 9.87687, + "91": 9.89287, + "92": 9.875, + "93": 9.81107, + "94": 9.89072, + "95": 9.87368, + "96": 9.85278, + "97": 9.7965, + "98": 9.83453, + "99": 9.87169, + "100": 9.76389 + } + }, + "num-zeros": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 37001.0, + "2": 36732.0, + "3": 36500.0, + "4": 36497.0, + "5": 36308.0, + "6": 35804.0, + "7": 37128.0, + "8": 35982.0, + "9": 36339.0, + "10": 37420.0, + "11": 36825.0, + "12": 35918.0, + "13": 36509.0, + "14": 36718.0, + "15": 36144.0, + "16": 35879.0, + "17": 36557.0, + "18": 36406.0, + "19": 36689.0, + "20": 36332.0, + "21": 36165.0, + "22": 36186.0, + "23": 36662.0, + "24": 36628.0, + "25": 35283.0, + "26": 36188.0, + "27": 36438.0, + "28": 35557.0, + "29": 35655.0, + "30": 35347.0, + "31": 36045.0, + "32": 36682.0, + "33": 36030.0, + "34": 35613.0, + "35": 36767.0, + "36": 35693.0, + "37": 36422.0, + "38": 35167.0, + "39": 36773.0, + "40": 37229.0, + "41": 36203.0, + "42": 35749.0, + "43": 37587.0, + "44": 36237.0, + "45": 39444.0, + "46": 37852.0, + "47": 38394.0, + "48": 38946.0, + "49": 42609.0, + "50": 39507.0, + "51": 39090.0, + "52": 41821.0, + "53": 40567.0, + "54": 41610.0, + "55": 39318.0, + "56": 41334.0, + "57": 36899.0, + "58": 46231.0, + "59": 42246.0, + "60": 42784.0, + "61": 42495.0, + "62": 45631.0, + "63": 44361.0, + "64": 49155.0, + "65": 42460.0, + "66": 45659.0, + "67": 50043.0, + "68": 46333.0, + "69": 45862.0, + "70": 46950.0, + "71": 46684.0, + "72": 45346.0, + "73": 47664.0, + "74": 45501.0, + "75": 48985.0, + "76": 47531.0, + "77": 48431.0, + "78": 48651.0, + "79": 48300.0, + "80": 45717.0, + "81": 54912.0, + "82": 42473.0, + "83": 46906.0, + "84": 45830.0, + "85": 46189.0, + "86": 46183.0, + "87": 36768.0, + "88": 45219.0, + "89": 48522.0, + "90": 58149.0, + "91": 39290.0, + "92": 49092.0, + "93": 45067.0, + "94": 42178.0, + "95": 45548.0, + "96": 50098.0, + "97": 46773.0, + "98": 49620.0, + "99": 46056.0, + "100": 46551.0 + } + }, + "mem-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 1254509056.0, + "2": 1254505984.0, + "3": 1254509056.0, + "4": 1254507520.0, + "5": 1254503936.0, + "6": 1254505472.0, + "7": 1254505472.0, + "8": 1254505984.0, + "9": 1254509056.0, + "10": 1254509056.0, + "11": 1254504960.0, + "12": 1254507520.0, + "13": 1254508544.0, + "14": 1254507008.0, + "15": 1254505472.0, + "16": 1254508032.0, + "17": 1254507008.0, + "18": 1254507520.0, + "19": 1254504960.0, + "20": 1254509056.0, + "21": 1254507008.0, + "22": 1254505984.0, + "23": 1254505472.0, + "24": 1254503936.0, + "25": 1254507520.0, + "26": 1254508032.0, + "27": 1254508544.0, + "28": 1254507008.0, + "29": 1254508544.0, + "30": 1254508032.0, + "31": 1254508544.0, + "32": 1254504960.0, + "33": 1254506496.0, + "34": 1254504960.0, + "35": 1254504960.0, + "36": 1254504960.0, + "37": 1254508032.0, + "38": 1254509056.0, + "39": 1254505984.0, + "40": 1254506496.0, + "41": 1254505472.0, + "42": 1254504960.0, + "43": 1254507008.0, + "44": 1254503936.0, + "45": 1254504960.0, + "46": 1254505984.0, + "47": 1254506496.0, + "48": 1254506496.0, + "49": 1254508032.0, + "50": 1254504448.0, + "51": 1254501376.0, + "52": 1254506496.0, + "53": 1254503424.0, + "54": 1254503424.0, + "55": 1254505472.0, + "56": 1254504448.0, + "57": 1254507008.0, + "58": 1254508544.0, + "59": 1254505472.0, + "60": 1254505984.0, + "61": 1254503424.0, + "62": 1254505984.0, + "63": 1254506496.0, + "64": 1254505472.0, + "65": 1254507008.0, + "66": 1254507008.0, + "67": 1254504448.0, + "68": 1254505984.0, + "69": 1254504960.0, + "70": 1254505472.0, + "71": 1254502400.0, + "72": 1254504448.0, + "73": 1254502912.0, + "74": 1254505984.0, + "75": 1254505984.0, + "76": 1254505472.0, + "77": 1254505984.0, + "78": 1254502912.0, + "79": 1254506496.0, + "80": 1254505984.0, + "81": 1254505472.0, + "82": 1254505472.0, + "83": 1254503424.0, + "84": 1254505984.0, + "85": 1254504960.0, + "86": 1254506496.0, + "87": 1254504448.0, + "88": 1254502400.0, + "89": 1254509568.0, + "90": 1254507008.0, + "91": 1254503936.0, + "92": 1254505984.0, + "93": 1254507008.0, + "94": 1254507520.0, + "95": 1254511104.0, + "96": 1254504448.0, + "97": 1254509568.0, + "98": 1254507520.0, + "99": 1254508032.0, + "100": 1254509056.0 + } + }, + "mem-max-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 2065569280.0, + "2": 2546124800.0, + "3": 2546124800.0, + "4": 2546124800.0, + "5": 2546257920.0, + "6": 2546257920.0, + "7": 2546452480.0, + "8": 2546452480.0, + "9": 2546452480.0, + "10": 2546452480.0, + "11": 2546452480.0, + "12": 2546452480.0, + "13": 2546452480.0, + "14": 2546452480.0, + "15": 2546452480.0, + "16": 2546452480.0, + "17": 2546452480.0, + "18": 2546452480.0, + "19": 2546452480.0, + "20": 2546452480.0, + "21": 2546452480.0, + "22": 2546452480.0, + "23": 2546452480.0, + "24": 2546452480.0, + "25": 2546452480.0, + "26": 2546452480.0, + "27": 2546452480.0, + "28": 2546452480.0, + "29": 2546452480.0, + "30": 2546452480.0, + "31": 2546757632.0, + "32": 2546757632.0, + "33": 2546757632.0, + "34": 2546757632.0, + "35": 2546757632.0, + "36": 2546757632.0, + "37": 2546757632.0, + "38": 2546757632.0, + "39": 2546757632.0, + "40": 2546757632.0, + "41": 2546757632.0, + "42": 2546757632.0, + "43": 2546757632.0, + "44": 2546757632.0, + "45": 2546757632.0, + "46": 2546757632.0, + "47": 2546757632.0, + "48": 2546757632.0, + "49": 2546757632.0, + "50": 2546757632.0, + "51": 2546757632.0, + "52": 2546757632.0, + "53": 2546757632.0, + "54": 2546757632.0, + "55": 2546757632.0, + "56": 2546757632.0, + "57": 2546757632.0, + "58": 2546757632.0, + "59": 2546757632.0, + "60": 2546757632.0, + "61": 2546757632.0, + "62": 2546757632.0, + "63": 2546757632.0, + "64": 2546757632.0, + "65": 2546757632.0, + "66": 2546757632.0, + "67": 2546757632.0, + "68": 2546757632.0, + "69": 2546757632.0, + "70": 2546757632.0, + "71": 2546757632.0, + "72": 2546757632.0, + "73": 2546757632.0, + "74": 2546757632.0, + "75": 2546757632.0, + "76": 2546757632.0, + "77": 2546757632.0, + "78": 2546757632.0, + "79": 2546757632.0, + "80": 2546757632.0, + "81": 2546757632.0, + "82": 2546757632.0, + "83": 2546757632.0, + "84": 2546757632.0, + "85": 2546757632.0, + "86": 2546757632.0, + "87": 2546757632.0, + "88": 2546757632.0, + "89": 2546757632.0, + "90": 2546757632.0, + "91": 2546757632.0, + "92": 2546757632.0, + "93": 2548838912.0, + "94": 2548838912.0, + "95": 2549586432.0, + "96": 2549586432.0, + "97": 2549698048.0, + "98": 2549698048.0, + "99": 2549698048.0, + "100": 2553959936.0 + } + }, + "iteration-time": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": "nan", + "2": 5.15436, + "3": 1.14805, + "4": 1.06806, + "5": 0.72669, + "6": 0.94546, + "7": 0.97122, + "8": 1.38628, + "9": 1.36685, + "10": 0.94507, + "11": 1.07108, + "12": 1.1283, + "13": 0.70706, + "14": 1.40285, + "15": 0.72368, + "16": 1.09995, + "17": 0.72743, + "18": 1.07922, + "19": 0.93969, + "20": 1.06804, + "21": 0.895, + "22": 1.03574, + "23": 1.04172, + "24": 1.55617, + "25": 0.72585, + "26": 0.99485, + "27": 1.03451, + "28": 1.00833, + "29": 0.97832, + "30": 0.92092, + "31": 0.78222, + "32": 1.73564, + "33": 1.14641, + "34": 0.7197, + "35": 1.15235, + "36": 1.17339, + "37": 0.99023, + "38": 0.74024, + "39": 0.86518, + "40": 1.35936, + "41": 0.97563, + "42": 1.03828, + "43": 0.93328, + "44": 1.29096, + "45": 0.71549, + "46": 1.0425, + "47": 1.07722, + "48": 1.03663, + "49": 1.08392, + "50": 1.05194, + "51": 0.85302, + "52": 0.84565, + "53": 1.20031, + "54": 1.25822, + "55": 1.05616, + "56": 1.07658, + "57": 0.97891, + "58": 1.26136, + "59": 0.71979, + "60": 0.88178, + "61": 1.14579, + "62": 1.0823, + "63": 0.89143, + "64": 1.18271, + "65": 0.73177, + "66": 0.98411, + "67": 0.999, + "68": 0.91118, + "69": 1.11892, + "70": 0.77223, + "71": 1.19686, + "72": 0.97903, + "73": 1.00633, + "74": 0.74667, + "75": 1.08324, + "76": 1.07002, + "77": 1.26542, + "78": 1.20996, + "79": 1.87681, + "80": 0.98333, + "81": 1.15641, + "82": 1.35391, + "83": 1.00817, + "84": 0.84928, + "85": 0.81655, + "86": 0.96952, + "87": 1.54588, + "88": 0.9453, + "89": 1.06515, + "90": 1.29204, + "91": 0.91247, + "92": 1.56539, + "93": 0.94082, + "94": 0.98214, + "95": 1.02289, + "96": 1.26733, + "97": 0.7751, + "98": 1.2711, + "99": 1.38344, + "100": 1.12745 + } + } +} \ No newline at end of file diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_resume_torch_dist_te_8experts2parallel_dist_optimizer_1node/model_config.yaml b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_resume_torch_dist_te_8experts2parallel_dist_optimizer_1node/model_config.yaml new file mode 100644 index 00000000000..764c576645e --- /dev/null +++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_resume_torch_dist_te_8experts2parallel_dist_optimizer_1node/model_config.yaml @@ -0,0 +1,65 @@ +ENV_VARS: + CUDA_DEVICE_MAX_CONNECTIONS: 1 + NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0 + NCCL_ALGO: Ring + CUBLAS_WORKSPACE_CONFIG: :4096:8 +MODEL_ARGS: + --num-layers: 12 + --hidden-size: 512 + --num-attention-heads: 8 + --log-params-norm: true + --log-num-zeros-in-grad: true + --log-validation-ppl-to-tensorboard: true + --log-timers-to-tensorboard: true + --tensorboard-dir: ${TENSORBOARD_PATH} + --micro-batch-size: 4 + --global-batch-size: 32 + --seq-length: 1024 + --max-position-embeddings: 1024 + --train-iters: 100 + --timing-log-level: 0 + --lr-decay-iters: 320000 + --save: ${CHECKPOINT_SAVE_PATH} + --load: ${CHECKPOINT_LOAD_PATH} + --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document + --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json + --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt + --split: 949,50,1 + --distributed-backend: nccl + --lr: 0.00015 + --lr-decay-style: cosine + --min-lr: 1.0e-5 + --weight-decay: 1e-2 + --clip-grad: 1.0 + --lr-warmup-fraction: .01 + --log-interval: 1 + --save-interval: 50 + --eval-interval: 1000 + --eval-iters: 10 + --transformer-impl: transformer_engine + --tensor-model-parallel-size: 2 + --pipeline-model-parallel-size: 1 + --expert-model-parallel-size: 2 + --sequence-parallel: true + --num-experts: 8 + --use-distributed-optimizer: true + --moe-router-load-balancing-type: sinkhorn + --moe-router-topk: 1 + --ckpt-fully-parallel-load: true + --deterministic-mode: true + --no-gradient-accumulation-fusion: true + --attention-softmax-in-fp32: true + --use-checkpoint-opt_param-scheduler: true + --use-mcore-models: true + --ckpt-format: torch_dist + --dist-ckpt-optim-fully-reshardable: true + --dist-ckpt-strictness: log_all # backward compatibility for TE changes + --ckpt-assume-constant-structure: true + --data-cache-path: ${DATA_CACHE_PATH} + --bf16: true + --disable-bias-linear: true + --no-bias-gelu-fusion: true + --log-memory-to-tensorboard: true + --async-save: true + --use-persistent-ckpt-worker: true +TEST_TYPE: ckpt-resume diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_resume_torch_dist_te_8experts2parallel_multi_dist_optimizer_instances_1node/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_resume_torch_dist_te_8experts2parallel_multi_dist_optimizer_instances_1node/golden_values_dev_dgx_gb200.json new file mode 100644 index 00000000000..9e26dfeeb6e --- /dev/null +++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_resume_torch_dist_te_8experts2parallel_multi_dist_optimizer_instances_1node/golden_values_dev_dgx_gb200.json @@ -0,0 +1 @@ +{} \ No newline at end of file diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_resume_torch_dist_te_8experts2parallel_multi_dist_optimizer_instances_1node/model_config.yaml b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_resume_torch_dist_te_8experts2parallel_multi_dist_optimizer_instances_1node/model_config.yaml new file mode 100644 index 00000000000..19b6015967b --- /dev/null +++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_resume_torch_dist_te_8experts2parallel_multi_dist_optimizer_instances_1node/model_config.yaml @@ -0,0 +1,66 @@ +ENV_VARS: + CUDA_DEVICE_MAX_CONNECTIONS: 1 + NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0 + NCCL_ALGO: Ring + CUBLAS_WORKSPACE_CONFIG: :4096:8 +MODEL_ARGS: + --num-layers: 12 + --hidden-size: 512 + --num-attention-heads: 8 + --log-params-norm: true + --log-num-zeros-in-grad: true + --log-validation-ppl-to-tensorboard: true + --log-timers-to-tensorboard: true + --tensorboard-dir: ${TENSORBOARD_PATH} + --micro-batch-size: 4 + --global-batch-size: 32 + --seq-length: 1024 + --max-position-embeddings: 1024 + --train-iters: 100 + --timing-log-level: 0 + --lr-decay-iters: 320000 + --save: ${CHECKPOINT_SAVE_PATH} + --load: ${CHECKPOINT_LOAD_PATH} + --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document + --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json + --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt + --split: 949,50,1 + --distributed-backend: nccl + --lr: 0.00015 + --lr-decay-style: cosine + --min-lr: 1.0e-5 + --weight-decay: 1e-2 + --clip-grad: 1.0 + --lr-warmup-fraction: .01 + --log-interval: 1 + --save-interval: 50 + --eval-interval: 1000 + --eval-iters: 10 + --transformer-impl: transformer_engine + --tensor-model-parallel-size: 2 + --pipeline-model-parallel-size: 1 + --expert-model-parallel-size: 1 + --sequence-parallel: true + --num-experts: 8 + --use-distributed-optimizer: true + --moe-router-load-balancing-type: sinkhorn + --moe-router-topk: 1 + --ckpt-fully-parallel-load: true + --deterministic-mode: true + --no-gradient-accumulation-fusion: true + --attention-softmax-in-fp32: true + --use-checkpoint-opt_param-scheduler: true + --use-mcore-models: true + --ckpt-format: torch_dist + --dist-ckpt-strictness: log_all # backward compatibility for TE changes + --ckpt-assume-constant-structure: true + --data-cache-path: ${DATA_CACHE_PATH} + --bf16: true + --disable-bias-linear: true + --no-bias-gelu-fusion: true + --log-memory-to-tensorboard: true + --num-distributed-optimizer-instances: 2 + --async-save: true + --use-persistent-ckpt-worker: true +TEST_TYPE: ckpt-resume +LAUNCHER: ft_launcher diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_8experts2parallel_ddp_average_in_collective_1node/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_8experts2parallel_ddp_average_in_collective_1node/golden_values_dev_dgx_gb200.json new file mode 100644 index 00000000000..4e880a431f4 --- /dev/null +++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_8experts2parallel_ddp_average_in_collective_1node/golden_values_dev_dgx_gb200.json @@ -0,0 +1,287 @@ +{ + "lm loss": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 10.92456, + "2": 10.91122, + "3": 10.92478, + "4": 10.91392, + "5": 10.91847, + "6": 10.90864, + "7": 10.90181, + "8": 10.90918, + "9": 10.92285, + "10": 10.91171, + "11": 10.90188, + "12": 10.91191, + "13": 10.88926, + "14": 10.8908, + "15": 10.88438, + "16": 10.87546, + "17": 10.87471, + "18": 10.8628, + "19": 10.8724, + "20": 10.8201, + "21": 10.8072, + "22": 10.79503, + "23": 10.79079, + "24": 10.75741, + "25": 10.76214, + "26": 10.75151, + "27": 10.73656, + "28": 10.67545, + "29": 10.63996, + "30": 10.62263, + "31": 10.61218, + "32": 10.59655, + "33": 10.5765, + "34": 10.53881, + "35": 10.53788, + "36": 10.5216, + "37": 10.48667, + "38": 10.49899, + "39": 10.45883, + "40": 10.44007, + "41": 10.42695, + "42": 10.41344, + "43": 10.38989, + "44": 10.35254, + "45": 10.36981, + "46": 10.33007, + "47": 10.31289, + "48": 10.27798, + "49": 10.27103, + "50": 10.27187 + } + }, + "num-zeros": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 18865.0, + "2": 18932.0, + "3": 18832.0, + "4": 18774.0, + "5": 18548.0, + "6": 18607.0, + "7": 18950.0, + "8": 18828.0, + "9": 19101.0, + "10": 19425.0, + "11": 19225.0, + "12": 18243.0, + "13": 19160.0, + "14": 19145.0, + "15": 18772.0, + "16": 18768.0, + "17": 18807.0, + "18": 19111.0, + "19": 18890.0, + "20": 19015.0, + "21": 18479.0, + "22": 18922.0, + "23": 18880.0, + "24": 18812.0, + "25": 18185.0, + "26": 18922.0, + "27": 18466.0, + "28": 18402.0, + "29": 18510.0, + "30": 18253.0, + "31": 18896.0, + "32": 18852.0, + "33": 18338.0, + "34": 18553.0, + "35": 18775.0, + "36": 18344.0, + "37": 19084.0, + "38": 18477.0, + "39": 18845.0, + "40": 19477.0, + "41": 18606.0, + "42": 18524.0, + "43": 19412.0, + "44": 18862.0, + "45": 20342.0, + "46": 19670.0, + "47": 19622.0, + "48": 19999.0, + "49": 21560.0, + "50": 20111.0 + } + }, + "mem-allocated-bytes": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 1388730880.0, + "2": 1388731392.0, + "3": 1388729856.0, + "4": 1388727296.0, + "5": 1388729856.0, + "6": 1388730368.0, + "7": 1388729856.0, + "8": 1388729856.0, + "9": 1388731392.0, + "10": 1388731392.0, + "11": 1388729344.0, + "12": 1388728832.0, + "13": 1388730880.0, + "14": 1388728320.0, + "15": 1388730880.0, + "16": 1388729344.0, + "17": 1388731904.0, + "18": 1388730880.0, + "19": 1388730368.0, + "20": 1388730368.0, + "21": 1388731392.0, + "22": 1388730880.0, + "23": 1388731392.0, + "24": 1388730880.0, + "25": 1388731904.0, + "26": 1388732928.0, + "27": 1388731392.0, + "28": 1388729856.0, + "29": 1388731392.0, + "30": 1388733440.0, + "31": 1388732416.0, + "32": 1388730368.0, + "33": 1388732416.0, + "34": 1388734464.0, + "35": 1388731904.0, + "36": 1388730368.0, + "37": 1388731904.0, + "38": 1388729856.0, + "39": 1388732928.0, + "40": 1388733952.0, + "41": 1388729856.0, + "42": 1388732928.0, + "43": 1388731392.0, + "44": 1388729856.0, + "45": 1388733952.0, + "46": 1388731392.0, + "47": 1388732928.0, + "48": 1388731904.0, + "49": 1388736000.0, + "50": 1388731904.0 + } + }, + "mem-max-allocated-bytes": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 3179614208.0, + "2": 3661343744.0, + "3": 3663287296.0, + "4": 3663287296.0, + "5": 3663287296.0, + "6": 3663287296.0, + "7": 3663287296.0, + "8": 3663287296.0, + "9": 3663287296.0, + "10": 3663287296.0, + "11": 3663287296.0, + "12": 3663287296.0, + "13": 3663287296.0, + "14": 3663287296.0, + "15": 3663287296.0, + "16": 3663287296.0, + "17": 3663287296.0, + "18": 3663287296.0, + "19": 3663287296.0, + "20": 3663287296.0, + "21": 3663287296.0, + "22": 3663287296.0, + "23": 3663287296.0, + "24": 3663287296.0, + "25": 3663287296.0, + "26": 3663287296.0, + "27": 3663287296.0, + "28": 3663287296.0, + "29": 3663287296.0, + "30": 3663287296.0, + "31": 3663287296.0, + "32": 3663287296.0, + "33": 3663287296.0, + "34": 3664704512.0, + "35": 3664704512.0, + "36": 3664704512.0, + "37": 3664704512.0, + "38": 3664704512.0, + "39": 3664704512.0, + "40": 3664704512.0, + "41": 3664704512.0, + "42": 3664704512.0, + "43": 3664704512.0, + "44": 3664704512.0, + "45": 3664704512.0, + "46": 3665018368.0, + "47": 3665018368.0, + "48": 3665595904.0, + "49": 3665595904.0, + "50": 3665595904.0 + } + }, + "iteration-time": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": "nan", + "2": 5.16531, + "3": 1.04123, + "4": 1.04608, + "5": 0.6702, + "6": 0.97028, + "7": 1.05473, + "8": 1.41719, + "9": 1.33604, + "10": 0.93357, + "11": 0.99617, + "12": 1.15157, + "13": 0.68494, + "14": 1.44824, + "15": 0.74568, + "16": 1.16937, + "17": 1.03022, + "18": 1.19997, + "19": 1.11058, + "20": 1.15049, + "21": 1.03221, + "22": 0.98188, + "23": 1.18487, + "24": 1.57733, + "25": 0.67963, + "26": 0.98667, + "27": 1.06481, + "28": 0.97271, + "29": 1.08609, + "30": 0.92966, + "31": 0.86714, + "32": 1.76431, + "33": 1.00875, + "34": 0.67917, + "35": 1.17957, + "36": 1.22633, + "37": 1.04519, + "38": 0.68646, + "39": 1.06128, + "40": 1.43564, + "41": 1.07329, + "42": 1.23873, + "43": 0.93227, + "44": 1.2455, + "45": 0.90321, + "46": 0.7656, + "47": 1.14639, + "48": 1.09713, + "49": 0.98182, + "50": 1.11118 + } + } +} \ No newline at end of file diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_8experts2parallel_ddp_average_in_collective_1node/model_config.yaml b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_8experts2parallel_ddp_average_in_collective_1node/model_config.yaml new file mode 100644 index 00000000000..8ced6e37a52 --- /dev/null +++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_8experts2parallel_ddp_average_in_collective_1node/model_config.yaml @@ -0,0 +1,66 @@ +ENV_VARS: + CUDA_DEVICE_MAX_CONNECTIONS: 1 + NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0 + NCCL_ALGO: Ring + CUBLAS_WORKSPACE_CONFIG: :4096:8 +MODEL_ARGS: + --num-layers: 12 + --hidden-size: 512 + --num-attention-heads: 8 + --log-params-norm: true + --log-num-zeros-in-grad: true + --log-validation-ppl-to-tensorboard: true + --log-timers-to-tensorboard: true + --tensorboard-dir: ${TENSORBOARD_PATH} + --micro-batch-size: 4 + --global-batch-size: 32 + --seq-length: 1024 + --max-position-embeddings: 1024 + --train-iters: 50 + --timing-log-level: 0 + --lr-decay-iters: 320000 + --save: ${CHECKPOINT_SAVE_PATH} + --load: ${CHECKPOINT_LOAD_PATH} + --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document + --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json + --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt + --split: 949,50,1 + --distributed-backend: nccl + --lr: 0.00015 + --lr-decay-style: cosine + --min-lr: 1.0e-5 + --weight-decay: 1e-2 + --clip-grad: 1.0 + --lr-warmup-fraction: .01 + --log-interval: 1 + --save-interval: 10000 + --eval-interval: 1000 + --eval-iters: 10 + --transformer-impl: transformer_engine + --tensor-model-parallel-size: 2 + --pipeline-model-parallel-size: 1 + --expert-model-parallel-size: 2 + --no-ckpt-fully-parallel-save: true + --moe-grouped-gemm: true + --disable-bias-linear: true + --sequence-parallel: true + --num-experts: 8 + --use-distributed-optimizer: true + --moe-router-load-balancing-type: sinkhorn + --moe-router-topk: 1 + --overlap-grad-reduce: true + --overlap-param-gather: true + --deterministic-mode: true + --no-gradient-accumulation-fusion: true + --attention-softmax-in-fp32: true + --use-mcore-models: true + --ckpt-format: torch_dist + --dist-ckpt-strictness: log_all # backward compatibility for TE changes + --data-cache-path: ${DATA_CACHE_PATH} + --bf16: true + --attention-backend: unfused + --no-bias-gelu-fusion: true + --log-memory-to-tensorboard: true + --async-save: true + --use-persistent-ckpt-worker: true +TEST_TYPE: regular diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_8experts2parallel_overlap_grad_reduce_param_gather_groupedGEMM_1node/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_8experts2parallel_overlap_grad_reduce_param_gather_groupedGEMM_1node/golden_values_dev_dgx_gb200.json new file mode 100644 index 00000000000..5da1e7de276 --- /dev/null +++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_8experts2parallel_overlap_grad_reduce_param_gather_groupedGEMM_1node/golden_values_dev_dgx_gb200.json @@ -0,0 +1,287 @@ +{ + "lm loss": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 10.92456, + "2": 10.91122, + "3": 10.92478, + "4": 10.91392, + "5": 10.91847, + "6": 10.90864, + "7": 10.90181, + "8": 10.90918, + "9": 10.92285, + "10": 10.91171, + "11": 10.90188, + "12": 10.91191, + "13": 10.88926, + "14": 10.8908, + "15": 10.88438, + "16": 10.87546, + "17": 10.87471, + "18": 10.8628, + "19": 10.8724, + "20": 10.8201, + "21": 10.8072, + "22": 10.79503, + "23": 10.79079, + "24": 10.75741, + "25": 10.76214, + "26": 10.75151, + "27": 10.73656, + "28": 10.67545, + "29": 10.63996, + "30": 10.62263, + "31": 10.61218, + "32": 10.59655, + "33": 10.5765, + "34": 10.53881, + "35": 10.53788, + "36": 10.5216, + "37": 10.48667, + "38": 10.49899, + "39": 10.45883, + "40": 10.44007, + "41": 10.42695, + "42": 10.41344, + "43": 10.38989, + "44": 10.35254, + "45": 10.36981, + "46": 10.33007, + "47": 10.31289, + "48": 10.27798, + "49": 10.27103, + "50": 10.27187 + } + }, + "num-zeros": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 18865.0, + "2": 18932.0, + "3": 18832.0, + "4": 18774.0, + "5": 18548.0, + "6": 18607.0, + "7": 18950.0, + "8": 18828.0, + "9": 19101.0, + "10": 19425.0, + "11": 19225.0, + "12": 18243.0, + "13": 19160.0, + "14": 19145.0, + "15": 18772.0, + "16": 18768.0, + "17": 18807.0, + "18": 19111.0, + "19": 18890.0, + "20": 19015.0, + "21": 18479.0, + "22": 18922.0, + "23": 18880.0, + "24": 18812.0, + "25": 18185.0, + "26": 18922.0, + "27": 18466.0, + "28": 18402.0, + "29": 18510.0, + "30": 18253.0, + "31": 18896.0, + "32": 18852.0, + "33": 18338.0, + "34": 18553.0, + "35": 18775.0, + "36": 18344.0, + "37": 19084.0, + "38": 18477.0, + "39": 18845.0, + "40": 19477.0, + "41": 18606.0, + "42": 18524.0, + "43": 19412.0, + "44": 18862.0, + "45": 20342.0, + "46": 19670.0, + "47": 19622.0, + "48": 19999.0, + "49": 21560.0, + "50": 20111.0 + } + }, + "mem-allocated-bytes": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 1388730880.0, + "2": 1388731392.0, + "3": 1388729856.0, + "4": 1388727296.0, + "5": 1388729856.0, + "6": 1388730368.0, + "7": 1388729856.0, + "8": 1388729856.0, + "9": 1388731392.0, + "10": 1388731392.0, + "11": 1388729344.0, + "12": 1388728832.0, + "13": 1388730880.0, + "14": 1388728320.0, + "15": 1388730880.0, + "16": 1388729344.0, + "17": 1388731904.0, + "18": 1388730880.0, + "19": 1388730368.0, + "20": 1388730368.0, + "21": 1388731392.0, + "22": 1388730880.0, + "23": 1388731392.0, + "24": 1388730880.0, + "25": 1388731904.0, + "26": 1388732928.0, + "27": 1388731392.0, + "28": 1388729856.0, + "29": 1388731392.0, + "30": 1388733440.0, + "31": 1388732416.0, + "32": 1388730368.0, + "33": 1388732416.0, + "34": 1388734464.0, + "35": 1388731904.0, + "36": 1388730368.0, + "37": 1388731904.0, + "38": 1388729856.0, + "39": 1388732928.0, + "40": 1388733952.0, + "41": 1388729856.0, + "42": 1388732928.0, + "43": 1388731392.0, + "44": 1388729856.0, + "45": 1388733952.0, + "46": 1388731392.0, + "47": 1388732928.0, + "48": 1388731904.0, + "49": 1388736000.0, + "50": 1388731904.0 + } + }, + "mem-max-allocated-bytes": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 3179614208.0, + "2": 3661343744.0, + "3": 3663287296.0, + "4": 3663287296.0, + "5": 3663287296.0, + "6": 3663287296.0, + "7": 3663287296.0, + "8": 3663287296.0, + "9": 3663287296.0, + "10": 3663287296.0, + "11": 3663287296.0, + "12": 3663287296.0, + "13": 3663287296.0, + "14": 3663287296.0, + "15": 3663287296.0, + "16": 3663287296.0, + "17": 3663287296.0, + "18": 3663287296.0, + "19": 3663287296.0, + "20": 3663287296.0, + "21": 3663287296.0, + "22": 3663287296.0, + "23": 3663287296.0, + "24": 3663287296.0, + "25": 3663287296.0, + "26": 3663287296.0, + "27": 3663287296.0, + "28": 3663287296.0, + "29": 3663287296.0, + "30": 3663287296.0, + "31": 3663287296.0, + "32": 3663287296.0, + "33": 3663287296.0, + "34": 3664704512.0, + "35": 3664704512.0, + "36": 3664704512.0, + "37": 3664704512.0, + "38": 3664704512.0, + "39": 3664704512.0, + "40": 3664704512.0, + "41": 3664704512.0, + "42": 3664704512.0, + "43": 3664704512.0, + "44": 3664704512.0, + "45": 3664704512.0, + "46": 3665018368.0, + "47": 3665018368.0, + "48": 3665595904.0, + "49": 3665595904.0, + "50": 3665595904.0 + } + }, + "iteration-time": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": "nan", + "2": 5.10431, + "3": 1.0895, + "4": 1.17142, + "5": 0.68918, + "6": 1.02487, + "7": 0.87181, + "8": 1.2196, + "9": 1.24209, + "10": 0.81565, + "11": 0.99071, + "12": 1.23287, + "13": 0.67714, + "14": 1.50122, + "15": 0.78165, + "16": 1.10532, + "17": 0.88779, + "18": 1.00749, + "19": 1.07727, + "20": 0.93733, + "21": 0.93154, + "22": 0.8995, + "23": 1.02523, + "24": 1.43664, + "25": 0.68482, + "26": 0.88827, + "27": 0.99047, + "28": 0.83785, + "29": 0.91289, + "30": 0.83794, + "31": 0.73258, + "32": 1.57266, + "33": 0.89346, + "34": 0.67579, + "35": 0.99531, + "36": 1.04733, + "37": 0.96334, + "38": 0.68428, + "39": 0.90409, + "40": 1.2649, + "41": 0.88602, + "42": 1.09446, + "43": 0.9862, + "44": 1.19048, + "45": 0.68556, + "46": 0.9026, + "47": 1.01866, + "48": 0.97401, + "49": 0.97492, + "50": 0.94695 + } + } +} \ No newline at end of file diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_8experts2parallel_overlap_grad_reduce_param_gather_groupedGEMM_1node/model_config.yaml b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_8experts2parallel_overlap_grad_reduce_param_gather_groupedGEMM_1node/model_config.yaml new file mode 100644 index 00000000000..8ced6e37a52 --- /dev/null +++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_8experts2parallel_overlap_grad_reduce_param_gather_groupedGEMM_1node/model_config.yaml @@ -0,0 +1,66 @@ +ENV_VARS: + CUDA_DEVICE_MAX_CONNECTIONS: 1 + NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0 + NCCL_ALGO: Ring + CUBLAS_WORKSPACE_CONFIG: :4096:8 +MODEL_ARGS: + --num-layers: 12 + --hidden-size: 512 + --num-attention-heads: 8 + --log-params-norm: true + --log-num-zeros-in-grad: true + --log-validation-ppl-to-tensorboard: true + --log-timers-to-tensorboard: true + --tensorboard-dir: ${TENSORBOARD_PATH} + --micro-batch-size: 4 + --global-batch-size: 32 + --seq-length: 1024 + --max-position-embeddings: 1024 + --train-iters: 50 + --timing-log-level: 0 + --lr-decay-iters: 320000 + --save: ${CHECKPOINT_SAVE_PATH} + --load: ${CHECKPOINT_LOAD_PATH} + --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document + --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json + --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt + --split: 949,50,1 + --distributed-backend: nccl + --lr: 0.00015 + --lr-decay-style: cosine + --min-lr: 1.0e-5 + --weight-decay: 1e-2 + --clip-grad: 1.0 + --lr-warmup-fraction: .01 + --log-interval: 1 + --save-interval: 10000 + --eval-interval: 1000 + --eval-iters: 10 + --transformer-impl: transformer_engine + --tensor-model-parallel-size: 2 + --pipeline-model-parallel-size: 1 + --expert-model-parallel-size: 2 + --no-ckpt-fully-parallel-save: true + --moe-grouped-gemm: true + --disable-bias-linear: true + --sequence-parallel: true + --num-experts: 8 + --use-distributed-optimizer: true + --moe-router-load-balancing-type: sinkhorn + --moe-router-topk: 1 + --overlap-grad-reduce: true + --overlap-param-gather: true + --deterministic-mode: true + --no-gradient-accumulation-fusion: true + --attention-softmax-in-fp32: true + --use-mcore-models: true + --ckpt-format: torch_dist + --dist-ckpt-strictness: log_all # backward compatibility for TE changes + --data-cache-path: ${DATA_CACHE_PATH} + --bf16: true + --attention-backend: unfused + --no-bias-gelu-fusion: true + --log-memory-to-tensorboard: true + --async-save: true + --use-persistent-ckpt-worker: true +TEST_TYPE: regular diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_a2a_ovlp_8experts_etp1_ep4_1node/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_a2a_ovlp_8experts_etp1_ep4_1node/golden_values_dev_dgx_gb200.json new file mode 100644 index 00000000000..7e76648b5a2 --- /dev/null +++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_a2a_ovlp_8experts_etp1_ep4_1node/golden_values_dev_dgx_gb200.json @@ -0,0 +1,287 @@ +{ + "lm loss": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 10.91962, + "2": 10.9168, + "3": 10.9211, + "4": 10.9202, + "5": 10.92774, + "6": 10.91636, + "7": 10.91825, + "8": 10.91825, + "9": 10.91448, + "10": 10.91644, + "11": 10.90375, + "12": 10.91532, + "13": 10.89542, + "14": 10.89837, + "15": 10.88246, + "16": 10.86562, + "17": 10.86327, + "18": 10.87059, + "19": 10.86132, + "20": 10.79564, + "21": 10.78115, + "22": 10.76223, + "23": 10.75418, + "24": 10.71745, + "25": 10.71507, + "26": 10.70251, + "27": 10.68134, + "28": 10.61348, + "29": 10.5756, + "30": 10.55196, + "31": 10.54825, + "32": 10.51901, + "33": 10.48575, + "34": 10.45833, + "35": 10.46344, + "36": 10.44142, + "37": 10.39933, + "38": 10.40604, + "39": 10.36502, + "40": 10.35425, + "41": 10.33201, + "42": 10.32538, + "43": 10.30184, + "44": 10.26353, + "45": 10.28155, + "46": 10.24406, + "47": 10.23118, + "48": 10.19372, + "49": 10.18577, + "50": 10.1917 + } + }, + "num-zeros": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 16504.0, + "2": 16572.0, + "3": 16848.0, + "4": 16537.0, + "5": 16489.0, + "6": 16124.0, + "7": 16810.0, + "8": 16150.0, + "9": 16548.0, + "10": 17006.0, + "11": 16733.0, + "12": 16149.0, + "13": 16785.0, + "14": 16595.0, + "15": 15935.0, + "16": 16182.0, + "17": 16600.0, + "18": 16851.0, + "19": 16582.0, + "20": 16488.0, + "21": 16536.0, + "22": 16707.0, + "23": 16806.0, + "24": 17135.0, + "25": 16344.0, + "26": 16797.0, + "27": 16755.0, + "28": 16541.0, + "29": 16348.0, + "30": 16485.0, + "31": 17033.0, + "32": 17299.0, + "33": 17333.0, + "34": 17176.0, + "35": 17567.0, + "36": 17050.0, + "37": 17719.0, + "38": 17231.0, + "39": 18134.0, + "40": 18748.0, + "41": 17574.0, + "42": 17132.0, + "43": 18791.0, + "44": 18164.0, + "45": 20261.0, + "46": 19601.0, + "47": 19299.0, + "48": 19756.0, + "49": 22082.0, + "50": 20481.0 + } + }, + "mem-allocated-bytes": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 1559053824.0, + "2": 1559856128.0, + "3": 1558879232.0, + "4": 1559959040.0, + "5": 1559031296.0, + "6": 1559183360.0, + "7": 1559374848.0, + "8": 1559565824.0, + "9": 1559610368.0, + "10": 1559703040.0, + "11": 1559183360.0, + "12": 1559448064.0, + "13": 1560306688.0, + "14": 1559711744.0, + "15": 1559081984.0, + "16": 1559234048.0, + "17": 1559234048.0, + "18": 1559914496.0, + "19": 1559424000.0, + "20": 1560553984.0, + "21": 1559284736.0, + "22": 1559234048.0, + "23": 1559386112.0, + "24": 1559863808.0, + "25": 1560294400.0, + "26": 1560320000.0, + "27": 1559489536.0, + "28": 1559887360.0, + "29": 1559600640.0, + "30": 1559588864.0, + "31": 1559835648.0, + "32": 1559639552.0, + "33": 1560015872.0, + "34": 1559436800.0, + "35": 1560263680.0, + "36": 1561096704.0, + "37": 1559690240.0, + "38": 1559588864.0, + "39": 1559979008.0, + "40": 1560567296.0, + "41": 1559785984.0, + "42": 1560777728.0, + "43": 1559698432.0, + "44": 1559842304.0, + "45": 1559690240.0, + "46": 1561019904.0, + "47": 1560320000.0, + "48": 1560912384.0, + "49": 1561067520.0, + "50": 1561448448.0 + } + }, + "mem-max-allocated-bytes": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": 3486019584.0, + "2": 4058942976.0, + "3": 4058942976.0, + "4": 4058942976.0, + "5": 4058942976.0, + "6": 4071244800.0, + "7": 4071244800.0, + "8": 4078498816.0, + "9": 4078498816.0, + "10": 4078498816.0, + "11": 4078498816.0, + "12": 4078498816.0, + "13": 4078498816.0, + "14": 4078498816.0, + "15": 4078498816.0, + "16": 4078498816.0, + "17": 4078498816.0, + "18": 4078498816.0, + "19": 4078498816.0, + "20": 4078498816.0, + "21": 4078498816.0, + "22": 4078498816.0, + "23": 4078498816.0, + "24": 4078498816.0, + "25": 4078498816.0, + "26": 4078498816.0, + "27": 4078498816.0, + "28": 4078498816.0, + "29": 4078498816.0, + "30": 4078498816.0, + "31": 4078498816.0, + "32": 4078498816.0, + "33": 4078498816.0, + "34": 4078498816.0, + "35": 4078498816.0, + "36": 4078498816.0, + "37": 4078498816.0, + "38": 4078498816.0, + "39": 4078498816.0, + "40": 4078498816.0, + "41": 4078498816.0, + "42": 4078498816.0, + "43": 4078498816.0, + "44": 4078498816.0, + "45": 4078498816.0, + "46": 4078498816.0, + "47": 4078498816.0, + "48": 4078498816.0, + "49": 4078498816.0, + "50": 4078498816.0 + } + }, + "iteration-time": { + "start_step": 1, + "end_step": 50, + "step_interval": 1, + "values": { + "1": "nan", + "2": 6.71344, + "3": 0.91312, + "4": 1.04131, + "5": 0.86382, + "6": 0.87909, + "7": 0.87762, + "8": 1.01283, + "9": 1.22894, + "10": 0.95315, + "11": 1.10276, + "12": 1.44194, + "13": 0.8686, + "14": 1.36291, + "15": 0.87247, + "16": 0.95306, + "17": 0.92075, + "18": 1.27791, + "19": 0.90913, + "20": 1.03807, + "21": 1.09635, + "22": 1.12487, + "23": 0.98485, + "24": 1.57898, + "25": 0.85737, + "26": 0.96274, + "27": 1.06847, + "28": 0.93211, + "29": 1.01486, + "30": 0.89997, + "31": 0.88497, + "32": 1.57666, + "33": 1.00332, + "34": 0.87395, + "35": 1.11181, + "36": 1.10799, + "37": 0.99699, + "38": 0.84601, + "39": 0.98925, + "40": 1.36782, + "41": 1.00209, + "42": 1.10394, + "43": 0.88403, + "44": 1.20166, + "45": 0.87336, + "46": 0.95282, + "47": 0.90049, + "48": 1.18114, + "49": 0.92763, + "50": 1.10248 + } + } +} \ No newline at end of file diff --git a/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_a2a_ovlp_8experts_etp1_ep4_1node/model_config.yaml b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_a2a_ovlp_8experts_etp1_ep4_1node/model_config.yaml new file mode 100644 index 00000000000..5423ee39527 --- /dev/null +++ b/tests/functional_tests/test_cases/moe/gpt3_mcore_te_tp2_pp1_te_a2a_ovlp_8experts_etp1_ep4_1node/model_config.yaml @@ -0,0 +1,68 @@ +ENV_VARS: + CUDA_DEVICE_MAX_CONNECTIONS: 1 + NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0 + NCCL_ALGO: Ring + CUBLAS_WORKSPACE_CONFIG: :4096:8 + N_REPEATS: 5 +MODEL_ARGS: + --num-layers: 12 + --hidden-size: 512 + --num-attention-heads: 8 + --log-params-norm: true + --log-num-zeros-in-grad: true + --log-validation-ppl-to-tensorboard: true + --log-timers-to-tensorboard: true + --tensorboard-dir: ${TENSORBOARD_PATH} + --micro-batch-size: 4 + --global-batch-size: 32 + --seq-length: 1024 + --max-position-embeddings: 1024 + --train-iters: 50 + --timing-log-level: 0 + --lr-decay-iters: 320000 + --save: ${CHECKPOINT_SAVE_PATH} + --load: ${CHECKPOINT_LOAD_PATH} + --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document + --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json + --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt + --split: 949,50,1 + --distributed-backend: nccl + --lr: 0.00015 + --lr-decay-style: cosine + --min-lr: 1.0e-5 + --weight-decay: 1e-2 + --clip-grad: 1.0 + --lr-warmup-fraction: .01 + --log-interval: 1 + --save-interval: 10000 + --eval-interval: 1000 + --eval-iters: 10 + --transformer-impl: transformer_engine + --tensor-model-parallel-size: 2 + --pipeline-model-parallel-size: 1 + --expert-model-parallel-size: 4 + --expert-tensor-parallel-size: 1 + --disable-bias-linear: true + --sequence-parallel: true + --num-experts: 8 + --moe-router-load-balancing-type: aux_loss + --moe-router-topk: 2 + --moe-aux-loss-coeff: 1e-2 + --deterministic-mode: true + --no-gradient-accumulation-fusion: true + --moe-grouped-gemm: true + --moe-token-dispatcher-type: alltoall + --overlap-moe-expert-parallel-comm: true + --attention-softmax-in-fp32: true + --use-mcore-models: true + --ckpt-format: torch_dist + --dist-ckpt-strictness: log_all # backward compatibility for TE changes + --data-cache-path: ${DATA_CACHE_PATH} + --bf16: true + --attention-backend: unfused + --no-bias-gelu-fusion: true + --log-memory-to-tensorboard: true + --exit-interval: 50 + --async-save: true + --use-persistent-ckpt-worker: true +TEST_TYPE: regular diff --git a/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_dist_muon_1node/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_dist_muon_1node/golden_values_dev_dgx_gb200.json new file mode 100644 index 00000000000..be51d3c11ca --- /dev/null +++ b/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_dist_muon_1node/golden_values_dev_dgx_gb200.json @@ -0,0 +1,537 @@ +{ + "lm loss": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 10.93025, + "2": 10.92269, + "3": 10.9252, + "4": 10.91732, + "5": 10.92716, + "6": 10.92029, + "7": 10.91669, + "8": 10.91796, + "9": 10.93887, + "10": 10.92403, + "11": 10.91598, + "12": 10.92558, + "13": 10.93454, + "14": 10.9158, + "15": 10.92838, + "16": 10.91826, + "17": 10.9274, + "18": 10.91269, + "19": 10.93069, + "20": 10.90057, + "21": 10.93975, + "22": 10.92531, + "23": 10.92976, + "24": 10.90567, + "25": 10.91494, + "26": 10.92515, + "27": 10.92786, + "28": 10.92017, + "29": 10.92534, + "30": 10.91409, + "31": 10.91023, + "32": 10.91855, + "33": 10.92026, + "34": 10.90054, + "35": 10.91897, + "36": 10.90741, + "37": 10.91267, + "38": 10.92379, + "39": 10.91251, + "40": 10.91043, + "41": 10.91309, + "42": 10.90891, + "43": 10.90503, + "44": 10.90748, + "45": 10.89756, + "46": 10.90681, + "47": 10.90215, + "48": 10.88414, + "49": 10.90022, + "50": 10.89863, + "51": 10.89746, + "52": 10.90395, + "53": 10.90118, + "54": 10.88728, + "55": 10.8869, + "56": 10.88881, + "57": 10.88731, + "58": 10.8888, + "59": 10.88231, + "60": 10.87378, + "61": 10.8731, + "62": 10.8684, + "63": 10.87569, + "64": 10.85934, + "65": 10.86089, + "66": 10.85671, + "67": 10.86152, + "68": 10.86019, + "69": 10.85018, + "70": 10.86781, + "71": 10.84867, + "72": 10.8405, + "73": 10.848, + "74": 10.83857, + "75": 10.83617, + "76": 10.83457, + "77": 10.82953, + "78": 10.82324, + "79": 10.82899, + "80": 10.82879, + "81": 10.81798, + "82": 10.81714, + "83": 10.80413, + "84": 10.78245, + "85": 10.78302, + "86": 10.80172, + "87": 10.79579, + "88": 10.79542, + "89": 10.77659, + "90": 10.78278, + "91": 10.79029, + "92": 10.78291, + "93": 10.75496, + "94": 10.76061, + "95": 10.77271, + "96": 10.73624, + "97": 10.74137, + "98": 10.74745, + "99": 10.76246, + "100": 10.74181 + } + }, + "num-zeros": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 48601.0, + "2": 50656.0, + "3": 49241.0, + "4": 48921.0, + "5": 50418.0, + "6": 49904.0, + "7": 51727.0, + "8": 49428.0, + "9": 50948.0, + "10": 52087.0, + "11": 50232.0, + "12": 48851.0, + "13": 52086.0, + "14": 50015.0, + "15": 47551.0, + "16": 49640.0, + "17": 49732.0, + "18": 51675.0, + "19": 50978.0, + "20": 52800.0, + "21": 49547.0, + "22": 49732.0, + "23": 53396.0, + "24": 51250.0, + "25": 48900.0, + "26": 52786.0, + "27": 51903.0, + "28": 48333.0, + "29": 49997.0, + "30": 51256.0, + "31": 52920.0, + "32": 49183.0, + "33": 52496.0, + "34": 50444.0, + "35": 50976.0, + "36": 50903.0, + "37": 49604.0, + "38": 51329.0, + "39": 50700.0, + "40": 50997.0, + "41": 48307.0, + "42": 50284.0, + "43": 51039.0, + "44": 47888.0, + "45": 54059.0, + "46": 47726.0, + "47": 50619.0, + "48": 50261.0, + "49": 55116.0, + "50": 50618.0, + "51": 49136.0, + "52": 50826.0, + "53": 49526.0, + "54": 49051.0, + "55": 48764.0, + "56": 48419.0, + "57": 48283.0, + "58": 52488.0, + "59": 50348.0, + "60": 49754.0, + "61": 47059.0, + "62": 50826.0, + "63": 48461.0, + "64": 55874.0, + "65": 51256.0, + "66": 48951.0, + "67": 51893.0, + "68": 49259.0, + "69": 54945.0, + "70": 50024.0, + "71": 48846.0, + "72": 51735.0, + "73": 52873.0, + "74": 49777.0, + "75": 50232.0, + "76": 51309.0, + "77": 50257.0, + "78": 49979.0, + "79": 48610.0, + "80": 51996.0, + "81": 50042.0, + "82": 47216.0, + "83": 52389.0, + "84": 48624.0, + "85": 50422.0, + "86": 49534.0, + "87": 46782.0, + "88": 48381.0, + "89": 49533.0, + "90": 52461.0, + "91": 50240.0, + "92": 50070.0, + "93": 50832.0, + "94": 51690.0, + "95": 48408.0, + "96": 49903.0, + "97": 49691.0, + "98": 48279.0, + "99": 48860.0, + "100": 47315.0 + } + }, + "mem-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 870734336.0, + "2": 870755840.0, + "3": 870738944.0, + "4": 870736384.0, + "5": 870749184.0, + "6": 870733824.0, + "7": 870728704.0, + "8": 870757376.0, + "9": 870707712.0, + "10": 870706176.0, + "11": 870695424.0, + "12": 870721536.0, + "13": 870721536.0, + "14": 870710272.0, + "15": 870740992.0, + "16": 870730240.0, + "17": 870743040.0, + "18": 870720000.0, + "19": 870715392.0, + "20": 870729216.0, + "21": 870750720.0, + "22": 870684672.0, + "23": 870728192.0, + "24": 870740992.0, + "25": 870703616.0, + "26": 870743040.0, + "27": 870746112.0, + "28": 870710272.0, + "29": 870728192.0, + "30": 870737920.0, + "31": 870701568.0, + "32": 870721536.0, + "33": 870708736.0, + "34": 870750208.0, + "35": 870679552.0, + "36": 870751744.0, + "37": 870719488.0, + "38": 870747648.0, + "39": 870707712.0, + "40": 870759424.0, + "41": 870723584.0, + "42": 870704128.0, + "43": 870708736.0, + "44": 870715392.0, + "45": 870730752.0, + "46": 870681088.0, + "47": 870702592.0, + "48": 870694400.0, + "49": 870727680.0, + "50": 870706176.0, + "51": 870675968.0, + "52": 870722560.0, + "53": 870680576.0, + "54": 870742528.0, + "55": 870687232.0, + "56": 870735872.0, + "57": 870709248.0, + "58": 870713344.0, + "59": 870715904.0, + "60": 870678528.0, + "61": 870660608.0, + "62": 870694912.0, + "63": 870702592.0, + "64": 870717952.0, + "65": 870720000.0, + "66": 870642176.0, + "67": 870725632.0, + "68": 870674944.0, + "69": 870699008.0, + "70": 870664704.0, + "71": 870688768.0, + "72": 870688256.0, + "73": 870685184.0, + "74": 870651904.0, + "75": 870676480.0, + "76": 870678528.0, + "77": 870715392.0, + "78": 870711296.0, + "79": 870681088.0, + "80": 870627328.0, + "81": 870669824.0, + "82": 870642688.0, + "83": 870636032.0, + "84": 870680576.0, + "85": 870650880.0, + "86": 870646784.0, + "87": 870637056.0, + "88": 870647808.0, + "89": 870635520.0, + "90": 870673920.0, + "91": 870638592.0, + "92": 870637056.0, + "93": 870653952.0, + "94": 870636032.0, + "95": 870631936.0, + "96": 870638080.0, + "97": 870631936.0, + "98": 870680064.0, + "99": 870618112.0, + "100": 870622720.0 + } + }, + "mem-max-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 3339315712.0, + "2": 3489557504.0, + "3": 3489557504.0, + "4": 3489557504.0, + "5": 3489557504.0, + "6": 3489557504.0, + "7": 3489557504.0, + "8": 3489557504.0, + "9": 3489557504.0, + "10": 3489557504.0, + "11": 3489557504.0, + "12": 3489557504.0, + "13": 3489557504.0, + "14": 3489610752.0, + "15": 3489610752.0, + "16": 3489610752.0, + "17": 3489610752.0, + "18": 3489610752.0, + "19": 3489610752.0, + "20": 3489610752.0, + "21": 3489610752.0, + "22": 3489610752.0, + "23": 3489610752.0, + "24": 3489610752.0, + "25": 3489610752.0, + "26": 3489610752.0, + "27": 3489610752.0, + "28": 3489610752.0, + "29": 3489610752.0, + "30": 3489610752.0, + "31": 3489610752.0, + "32": 3489610752.0, + "33": 3489610752.0, + "34": 3489610752.0, + "35": 3489610752.0, + "36": 3489610752.0, + "37": 3489610752.0, + "38": 3489610752.0, + "39": 3489610752.0, + "40": 3489610752.0, + "41": 3489610752.0, + "42": 3489610752.0, + "43": 3489610752.0, + "44": 3489610752.0, + "45": 3489610752.0, + "46": 3489610752.0, + "47": 3489610752.0, + "48": 3489610752.0, + "49": 3489610752.0, + "50": 3489610752.0, + "51": 3489610752.0, + "52": 3489610752.0, + "53": 3489610752.0, + "54": 3489610752.0, + "55": 3489610752.0, + "56": 3489610752.0, + "57": 3489610752.0, + "58": 3489610752.0, + "59": 3489610752.0, + "60": 3489610752.0, + "61": 3489610752.0, + "62": 3489610752.0, + "63": 3489610752.0, + "64": 3489610752.0, + "65": 3489610752.0, + "66": 3489610752.0, + "67": 3489610752.0, + "68": 3489610752.0, + "69": 3489610752.0, + "70": 3489610752.0, + "71": 3489610752.0, + "72": 3489610752.0, + "73": 3489610752.0, + "74": 3489610752.0, + "75": 3489610752.0, + "76": 3489610752.0, + "77": 3489610752.0, + "78": 3489610752.0, + "79": 3489610752.0, + "80": 3489610752.0, + "81": 3489610752.0, + "82": 3489610752.0, + "83": 3489610752.0, + "84": 3489610752.0, + "85": 3489610752.0, + "86": 3489610752.0, + "87": 3489610752.0, + "88": 3489610752.0, + "89": 3489610752.0, + "90": 3489610752.0, + "91": 3489610752.0, + "92": 3489610752.0, + "93": 3489610752.0, + "94": 3489610752.0, + "95": 3489610752.0, + "96": 3489610752.0, + "97": 3489610752.0, + "98": 3489610752.0, + "99": 3489610752.0, + "100": 3489610752.0 + } + }, + "iteration-time": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": "nan", + "2": 5.14086, + "3": 0.47233, + "4": 0.45712, + "5": 0.44412, + "6": 0.42853, + "7": 0.49106, + "8": 0.59362, + "9": 0.54446, + "10": 0.54165, + "11": 1.02569, + "12": 0.43367, + "13": 0.42601, + "14": 0.62362, + "15": 0.43253, + "16": 0.62994, + "17": 0.44215, + "18": 0.83992, + "19": 0.43915, + "20": 0.43247, + "21": 0.66028, + "22": 0.43116, + "23": 0.55737, + "24": 0.44612, + "25": 0.74027, + "26": 0.43872, + "27": 0.45606, + "28": 0.43992, + "29": 0.55458, + "30": 0.43527, + "31": 0.43318, + "32": 0.75422, + "33": 0.7022, + "34": 0.43471, + "35": 0.44304, + "36": 0.57147, + "37": 0.49801, + "38": 0.43717, + "39": 0.43783, + "40": 0.53625, + "41": 0.55379, + "42": 0.56541, + "43": 0.43329, + "44": 0.8147, + "45": 0.44041, + "46": 0.48828, + "47": 0.53121, + "48": 0.71316, + "49": 0.43013, + "50": 0.44299, + "51": 0.51877, + "52": 0.47177, + "53": 0.44337, + "54": 0.43564, + "55": 0.53808, + "56": 0.54492, + "57": 0.44137, + "58": 0.48745, + "59": 0.65341, + "60": 0.44519, + "61": 0.43692, + "62": 0.42973, + "63": 0.54217, + "64": 0.78646, + "65": 0.42968, + "66": 0.44258, + "67": 0.5025, + "68": 0.43471, + "69": 0.44767, + "70": 0.81252, + "71": 0.43128, + "72": 0.42873, + "73": 0.4366, + "74": 0.43332, + "75": 0.43919, + "76": 0.43802, + "77": 0.43252, + "78": 1.0049, + "79": 0.4342, + "80": 0.43523, + "81": 0.43417, + "82": 0.50204, + "83": 0.65213, + "84": 0.46634, + "85": 0.45183, + "86": 0.43167, + "87": 0.58782, + "88": 0.6618, + "89": 0.49046, + "90": 0.45459, + "91": 0.46901, + "92": 0.69901, + "93": 0.43449, + "94": 0.49848, + "95": 0.48448, + "96": 0.65215, + "97": 0.42996, + "98": 0.44019, + "99": 1.0558, + "100": 0.44015 + } + } +} \ No newline at end of file diff --git a/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_dist_muon_1node/model_config.yaml b/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_dist_muon_1node/model_config.yaml new file mode 100644 index 00000000000..40ac94eed9b --- /dev/null +++ b/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_dist_muon_1node/model_config.yaml @@ -0,0 +1,69 @@ +ENV_VARS: + CUDA_DEVICE_MAX_CONNECTIONS: 1 + NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0 + NCCL_ALGO: Ring + CUBLAS_WORKSPACE_CONFIG: :4096:8 +MODEL_ARGS: + --num-layers: 12 + --hidden-size: 512 + --num-attention-heads: 8 + --log-params-norm: true + --log-num-zeros-in-grad: true + --log-validation-ppl-to-tensorboard: true + --log-timers-to-tensorboard: true + --tensorboard-dir: ${TENSORBOARD_PATH} + --micro-batch-size: 4 + --global-batch-size: 32 + --seq-length: 1024 + --max-position-embeddings: 1024 + --disable-bias-linear: true + --train-iters: 100 + --timing-log-level: 0 + --lr-decay-iters: 320000 + --save: ${CHECKPOINT_SAVE_PATH} + --load: ${CHECKPOINT_LOAD_PATH} + --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document + --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json + --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt + --split: 949,50,1 + --distributed-backend: nccl + --lr: 0.00015 + --lr-decay-style: cosine + --min-lr: 1.0e-5 + --weight-decay: 1e-2 + --clip-grad: 1.0 + --lr-warmup-fraction: .01 + --log-interval: 1 + --save-interval: 50 + --eval-interval: 1000 + --eval-iters: 10 + --transformer-impl: transformer_engine + --tensor-model-parallel-size: 1 + --pipeline-model-parallel-size: 1 + --expert-model-parallel-size: 4 + --num-experts: 8 + --moe-token-dispatcher-type: allgather + --moe-router-load-balancing-type: aux_loss + --moe-router-topk: 2 + --moe-router-dtype: fp32 + --moe-ffn-hidden-size: 1024 + --moe-grouped-gemm: true + --ckpt-fully-parallel-load: true + --deterministic-mode: true + --no-gradient-accumulation-fusion: true + --attention-softmax-in-fp32: true + --use-checkpoint-opt_param-scheduler: true + --use-mcore-models: true + --ckpt-format: torch_dist + --data-cache-path: ${DATA_CACHE_PATH} + --bf16: true + --no-bias-gelu-fusion: true + --log-memory-to-tensorboard: true + --optimizer: muon + --muon-momentum: 0.9 + --muon-extra-scale-factor: 0.2 + --muon-scale-mode: spectral + --async-save: true + --use-persistent-ckpt-worker: true + --use-distributed-optimizer: true +TEST_TYPE: ckpt-resume diff --git a/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_dist_optimizer_1node/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_dist_optimizer_1node/golden_values_dev_dgx_gb200.json new file mode 100644 index 00000000000..9f64d216de0 --- /dev/null +++ b/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_dist_optimizer_1node/golden_values_dev_dgx_gb200.json @@ -0,0 +1,537 @@ +{ + "lm loss": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 10.93025, + "2": 10.92269, + "3": 10.92513, + "4": 10.91739, + "5": 10.9269, + "6": 10.92031, + "7": 10.91598, + "8": 10.91649, + "9": 10.93519, + "10": 10.9206, + "11": 10.90717, + "12": 10.91306, + "13": 10.92203, + "14": 10.90168, + "15": 10.89219, + "16": 10.87582, + "17": 10.88267, + "18": 10.85858, + "19": 10.87882, + "20": 10.80364, + "21": 10.81054, + "22": 10.78654, + "23": 10.77822, + "24": 10.74148, + "25": 10.73996, + "26": 10.73202, + "27": 10.71123, + "28": 10.65525, + "29": 10.62359, + "30": 10.59735, + "31": 10.58089, + "32": 10.57395, + "33": 10.54555, + "34": 10.50091, + "35": 10.51208, + "36": 10.48236, + "37": 10.45341, + "38": 10.46938, + "39": 10.42943, + "40": 10.41557, + "41": 10.3926, + "42": 10.38242, + "43": 10.34965, + "44": 10.33277, + "45": 10.33406, + "46": 10.30638, + "47": 10.28401, + "48": 10.2469, + "49": 10.2361, + "50": 10.24807, + "51": 10.24514, + "52": 10.19574, + "53": 10.19181, + "54": 10.16588, + "55": 10.12867, + "56": 10.16541, + "57": 10.13627, + "58": 10.15663, + "59": 10.10169, + "60": 10.1085, + "61": 10.07239, + "62": 10.03691, + "63": 10.10933, + "64": 10.05782, + "65": 10.02658, + "66": 10.05862, + "67": 10.02843, + "68": 9.98898, + "69": 10.00162, + "70": 9.99443, + "71": 10.02546, + "72": 9.98142, + "73": 9.9673, + "74": 9.95642, + "75": 9.92751, + "76": 9.97435, + "77": 9.96221, + "78": 9.9049, + "79": 9.91517, + "80": 9.92522, + "81": 9.94341, + "82": 9.90375, + "83": 9.85642, + "84": 9.79325, + "85": 9.77579, + "86": 9.89058, + "87": 9.9164, + "88": 9.89322, + "89": 9.82868, + "90": 9.82075, + "91": 9.83171, + "92": 9.82545, + "93": 9.75863, + "94": 9.83386, + "95": 9.83104, + "96": 9.81382, + "97": 9.75302, + "98": 9.78874, + "99": 9.8339, + "100": 9.72866 + } + }, + "num-zeros": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 48601.0, + "2": 50656.0, + "3": 48952.0, + "4": 48834.0, + "5": 50218.0, + "6": 50021.0, + "7": 51375.0, + "8": 49745.0, + "9": 51326.0, + "10": 52144.0, + "11": 49913.0, + "12": 48458.0, + "13": 51932.0, + "14": 50123.0, + "15": 47949.0, + "16": 49843.0, + "17": 49882.0, + "18": 52196.0, + "19": 51092.0, + "20": 52823.0, + "21": 49874.0, + "22": 49685.0, + "23": 53705.0, + "24": 52981.0, + "25": 49745.0, + "26": 53405.0, + "27": 53055.0, + "28": 49700.0, + "29": 52803.0, + "30": 54243.0, + "31": 56717.0, + "32": 53359.0, + "33": 56818.0, + "34": 53753.0, + "35": 55935.0, + "36": 57567.0, + "37": 54585.0, + "38": 55711.0, + "39": 58747.0, + "40": 58084.0, + "41": 55456.0, + "42": 57851.0, + "43": 61712.0, + "44": 58241.0, + "45": 67706.0, + "46": 57622.0, + "47": 105431.0, + "48": 64577.0, + "49": 71094.0, + "50": 67123.0, + "51": 63642.0, + "52": 71395.0, + "53": 110439.0, + "54": 72921.0, + "55": 61905.0, + "56": 1115059.0, + "57": 105088.0, + "58": 1127666.0, + "59": 80324.0, + "60": 123288.0, + "61": 1114978.0, + "62": 2170919.0, + "63": 70744.0, + "64": 91071.0, + "65": 173088.0, + "66": 1120234.0, + "67": 139759.0, + "68": 1123171.0, + "69": 1163608.0, + "70": 193228.0, + "71": 1118079.0, + "72": 2177823.0, + "73": 1182410.0, + "74": 2166520.0, + "75": 2166182.0, + "76": 1129053.0, + "77": 2168343.0, + "78": 1177493.0, + "79": 1175756.0, + "80": 1176743.0, + "81": 2170930.0, + "82": 2170395.0, + "83": 2170477.0, + "84": 1122782.0, + "85": 144101.0, + "86": 1124182.0, + "87": 1171244.0, + "88": 1168022.0, + "89": 1184028.0, + "90": 1183941.0, + "91": 1164459.0, + "92": 1131381.0, + "93": 1120131.0, + "94": 2175177.0, + "95": 2225361.0, + "96": 2172323.0, + "97": 2176261.0, + "98": 2175520.0, + "99": 1172004.0, + "100": 1197756.0 + } + }, + "mem-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 1059404800.0, + "2": 1059426304.0, + "3": 1059409408.0, + "4": 1059407360.0, + "5": 1059418624.0, + "6": 1059404288.0, + "7": 1059398656.0, + "8": 1059425280.0, + "9": 1059375104.0, + "10": 1059374080.0, + "11": 1059355648.0, + "12": 1059381248.0, + "13": 1059381248.0, + "14": 1059368448.0, + "15": 1059378688.0, + "16": 1059364864.0, + "17": 1059371520.0, + "18": 1059343872.0, + "19": 1059335680.0, + "20": 1059316224.0, + "21": 1059302912.0, + "22": 1059234304.0, + "23": 1059273216.0, + "24": 1059272704.0, + "25": 1059232768.0, + "26": 1059252736.0, + "27": 1059254272.0, + "28": 1059177984.0, + "29": 1059168256.0, + "30": 1059190272.0, + "31": 1059146752.0, + "32": 1059154944.0, + "33": 1059131392.0, + "34": 1059156992.0, + "35": 1059128320.0, + "36": 1059153920.0, + "37": 1059128320.0, + "38": 1059129344.0, + "39": 1059120128.0, + "40": 1059153408.0, + "41": 1059121152.0, + "42": 1059118592.0, + "43": 1059116032.0, + "44": 1059122176.0, + "45": 1059144192.0, + "46": 1059119104.0, + "47": 1059133440.0, + "48": 1059131392.0, + "49": 1059138560.0, + "50": 1059123712.0, + "51": 1059121664.0, + "52": 1059122688.0, + "53": 1059100160.0, + "54": 1059146752.0, + "55": 1059116544.0, + "56": 1059119104.0, + "57": 1059092992.0, + "58": 1059114496.0, + "59": 1059089920.0, + "60": 1059072000.0, + "61": 1059088384.0, + "62": 1059093504.0, + "63": 1059085312.0, + "64": 1059065856.0, + "65": 1059073024.0, + "66": 1059051520.0, + "67": 1059049984.0, + "68": 1059025408.0, + "69": 1059041280.0, + "70": 1059020288.0, + "71": 1059035136.0, + "72": 1059006464.0, + "73": 1059009536.0, + "74": 1059009024.0, + "75": 1059028480.0, + "76": 1059019264.0, + "77": 1059038208.0, + "78": 1059022848.0, + "79": 1059030016.0, + "80": 1059014656.0, + "81": 1059033600.0, + "82": 1059005952.0, + "83": 1059012096.0, + "84": 1059024384.0, + "85": 1059003904.0, + "86": 1059016192.0, + "87": 1059000320.0, + "88": 1058989056.0, + "89": 1059011584.0, + "90": 1059018752.0, + "91": 1058996736.0, + "92": 1059015680.0, + "93": 1058995712.0, + "94": 1058995200.0, + "95": 1059003392.0, + "96": 1059005440.0, + "97": 1059012096.0, + "98": 1059019264.0, + "99": 1058999808.0, + "100": 1058989568.0 + } + }, + "mem-max-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 3362050048.0, + "2": 3674120704.0, + "3": 3674120704.0, + "4": 3674120704.0, + "5": 3674120704.0, + "6": 3674120704.0, + "7": 3674120704.0, + "8": 3674120704.0, + "9": 3674120704.0, + "10": 3674120704.0, + "11": 3674120704.0, + "12": 3674120704.0, + "13": 3674120704.0, + "14": 3674120704.0, + "15": 3674120704.0, + "16": 3674120704.0, + "17": 3674120704.0, + "18": 3674120704.0, + "19": 3674120704.0, + "20": 3674120704.0, + "21": 3674120704.0, + "22": 3674120704.0, + "23": 3674120704.0, + "24": 3674120704.0, + "25": 3674120704.0, + "26": 3674120704.0, + "27": 3674120704.0, + "28": 3674120704.0, + "29": 3674120704.0, + "30": 3674120704.0, + "31": 3674120704.0, + "32": 3674120704.0, + "33": 3674120704.0, + "34": 3674120704.0, + "35": 3674120704.0, + "36": 3674120704.0, + "37": 3674120704.0, + "38": 3674120704.0, + "39": 3674120704.0, + "40": 3674120704.0, + "41": 3674120704.0, + "42": 3674120704.0, + "43": 3674120704.0, + "44": 3674120704.0, + "45": 3674120704.0, + "46": 3674120704.0, + "47": 3674120704.0, + "48": 3674120704.0, + "49": 3674120704.0, + "50": 3674120704.0, + "51": 3674120704.0, + "52": 3674120704.0, + "53": 3674120704.0, + "54": 3674120704.0, + "55": 3674120704.0, + "56": 3674120704.0, + "57": 3674120704.0, + "58": 3674120704.0, + "59": 3674120704.0, + "60": 3674120704.0, + "61": 3674120704.0, + "62": 3674120704.0, + "63": 3674120704.0, + "64": 3674120704.0, + "65": 3674120704.0, + "66": 3674120704.0, + "67": 3674120704.0, + "68": 3674120704.0, + "69": 3674120704.0, + "70": 3674120704.0, + "71": 3674120704.0, + "72": 3674120704.0, + "73": 3674120704.0, + "74": 3674120704.0, + "75": 3674120704.0, + "76": 3674120704.0, + "77": 3674120704.0, + "78": 3674120704.0, + "79": 3674120704.0, + "80": 3674120704.0, + "81": 3674120704.0, + "82": 3674120704.0, + "83": 3674120704.0, + "84": 3674120704.0, + "85": 3674120704.0, + "86": 3674120704.0, + "87": 3674120704.0, + "88": 3674120704.0, + "89": 3674120704.0, + "90": 3674120704.0, + "91": 3674120704.0, + "92": 3674120704.0, + "93": 3674120704.0, + "94": 3674120704.0, + "95": 3674120704.0, + "96": 3674120704.0, + "97": 3674120704.0, + "98": 3674120704.0, + "99": 3674120704.0, + "100": 3674120704.0 + } + }, + "iteration-time": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": "nan", + "2": 5.52163, + "3": 0.36689, + "4": 0.35504, + "5": 0.34056, + "6": 0.49682, + "7": 0.77331, + "8": 0.89355, + "9": 0.59397, + "10": 0.34961, + "11": 0.99811, + "12": 0.32714, + "13": 0.36966, + "14": 0.63202, + "15": 0.33393, + "16": 0.73874, + "17": 0.46619, + "18": 0.87849, + "19": 0.32443, + "20": 0.39295, + "21": 0.71931, + "22": 0.60297, + "23": 0.54723, + "24": 0.32986, + "25": 0.37182, + "26": 0.78659, + "27": 0.33267, + "28": 0.43626, + "29": 0.59793, + "30": 0.32969, + "31": 0.32437, + "32": 0.79649, + "33": 0.55708, + "34": 0.54273, + "35": 0.35997, + "36": 0.48818, + "37": 0.50813, + "38": 0.36341, + "39": 0.50853, + "40": 0.48538, + "41": 0.48648, + "42": 0.61299, + "43": 0.33761, + "44": 0.83386, + "45": 0.32743, + "46": 0.49262, + "47": 0.46349, + "48": 0.61229, + "49": 0.3602, + "50": 0.56504, + "51": 0.38618, + "52": 0.36741, + "53": 0.35641, + "54": 0.50231, + "55": 0.4516, + "56": 0.69186, + "57": 0.36689, + "58": 0.33529, + "59": 0.61356, + "60": 0.53446, + "61": 0.32501, + "62": 0.44349, + "63": 0.48405, + "64": 0.77754, + "65": 0.34885, + "66": 0.57108, + "67": 0.48156, + "68": 0.32859, + "69": 0.49511, + "70": 0.8307, + "71": 0.32534, + "72": 0.33486, + "73": 0.47976, + "74": 0.338, + "75": 0.7855, + "76": 0.34926, + "77": 0.34724, + "78": 1.09379, + "79": 0.3295, + "80": 0.46209, + "81": 0.50691, + "82": 0.44397, + "83": 0.62128, + "84": 0.59958, + "85": 0.33003, + "86": 0.33041, + "87": 0.72406, + "88": 0.76686, + "89": 0.51293, + "90": 0.4376, + "91": 0.49002, + "92": 0.63985, + "93": 0.39129, + "94": 0.50277, + "95": 0.77256, + "96": 0.34588, + "97": 0.61078, + "98": 0.50905, + "99": 1.19718, + "100": 0.33814 + } + } +} \ No newline at end of file diff --git a/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_dist_optimizer_1node/model_config.yaml b/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_dist_optimizer_1node/model_config.yaml new file mode 100644 index 00000000000..85b4ea629df --- /dev/null +++ b/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_dist_optimizer_1node/model_config.yaml @@ -0,0 +1,67 @@ +ENV_VARS: + CUDA_DEVICE_MAX_CONNECTIONS: 1 + NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0 + NCCL_ALGO: Ring + CUBLAS_WORKSPACE_CONFIG: :4096:8 +MODEL_ARGS: + --num-layers: 12 + --hidden-size: 512 + --num-attention-heads: 8 + --log-params-norm: true + --log-num-zeros-in-grad: true + --log-validation-ppl-to-tensorboard: true + --log-timers-to-tensorboard: true + --tensorboard-dir: ${TENSORBOARD_PATH} + --micro-batch-size: 4 + --global-batch-size: 32 + --seq-length: 1024 + --max-position-embeddings: 1024 + --disable-bias-linear: true + --train-iters: 100 + --timing-log-level: 0 + --lr-decay-iters: 320000 + --save: ${CHECKPOINT_SAVE_PATH} + --load: ${CHECKPOINT_LOAD_PATH} + --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document + --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json + --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt + --split: 949,50,1 + --distributed-backend: nccl + --lr: 0.00015 + --lr-decay-style: cosine + --min-lr: 1.0e-5 + --weight-decay: 1e-2 + --clip-grad: 1.0 + --lr-warmup-fraction: .01 + --log-interval: 1 + --save-interval: 50 + --eval-interval: 1000 + --eval-iters: 10 + --transformer-impl: transformer_engine + --tensor-model-parallel-size: 1 + --pipeline-model-parallel-size: 1 + --expert-model-parallel-size: 4 + --num-experts: 8 + --use-distributed-optimizer: true + --moe-token-dispatcher-type: allgather + --moe-router-load-balancing-type: aux_loss + --moe-router-topk: 2 + --moe-router-dtype: fp32 + --moe-ffn-hidden-size: 1024 + --moe-grouped-gemm: true + --ckpt-fully-parallel-load: true + --deterministic-mode: true + --no-gradient-accumulation-fusion: true + --attention-softmax-in-fp32: true + --use-checkpoint-opt_param-scheduler: true + --use-mcore-models: true + --ckpt-format: torch_dist + --ckpt-assume-constant-structure: true + --data-cache-path: ${DATA_CACHE_PATH} + --bf16: true + --no-bias-gelu-fusion: true + --log-memory-to-tensorboard: true + --async-save: true + --async-strategy: mcore + --use-persistent-ckpt-worker: true +TEST_TYPE: ckpt-resume diff --git a/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_muon_1node/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_muon_1node/golden_values_dev_dgx_gb200.json new file mode 100644 index 00000000000..cdfed84f204 --- /dev/null +++ b/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_muon_1node/golden_values_dev_dgx_gb200.json @@ -0,0 +1,537 @@ +{ + "lm loss": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 10.93025, + "2": 10.92269, + "3": 10.9252, + "4": 10.91732, + "5": 10.92716, + "6": 10.92029, + "7": 10.91689, + "8": 10.91796, + "9": 10.93832, + "10": 10.9236, + "11": 10.91613, + "12": 10.92602, + "13": 10.93472, + "14": 10.91527, + "15": 10.92824, + "16": 10.91861, + "17": 10.9271, + "18": 10.91296, + "19": 10.93023, + "20": 10.90016, + "21": 10.93966, + "22": 10.92562, + "23": 10.93028, + "24": 10.90543, + "25": 10.91525, + "26": 10.9253, + "27": 10.92741, + "28": 10.92037, + "29": 10.92588, + "30": 10.91378, + "31": 10.91042, + "32": 10.91855, + "33": 10.91979, + "34": 10.90066, + "35": 10.91944, + "36": 10.90776, + "37": 10.91233, + "38": 10.92343, + "39": 10.91274, + "40": 10.91036, + "41": 10.91294, + "42": 10.90908, + "43": 10.90588, + "44": 10.90751, + "45": 10.89747, + "46": 10.90732, + "47": 10.90263, + "48": 10.88416, + "49": 10.90034, + "50": 10.89852, + "51": 10.89775, + "52": 10.9041, + "53": 10.90148, + "54": 10.88746, + "55": 10.88695, + "56": 10.88936, + "57": 10.88718, + "58": 10.88857, + "59": 10.88171, + "60": 10.87361, + "61": 10.8735, + "62": 10.86885, + "63": 10.87575, + "64": 10.85989, + "65": 10.86121, + "66": 10.85651, + "67": 10.86113, + "68": 10.86021, + "69": 10.85001, + "70": 10.86693, + "71": 10.84848, + "72": 10.84069, + "73": 10.84796, + "74": 10.83901, + "75": 10.83614, + "76": 10.83454, + "77": 10.82879, + "78": 10.82267, + "79": 10.82851, + "80": 10.82921, + "81": 10.81793, + "82": 10.8169, + "83": 10.80474, + "84": 10.78276, + "85": 10.78362, + "86": 10.80088, + "87": 10.7953, + "88": 10.79578, + "89": 10.77732, + "90": 10.78292, + "91": 10.78991, + "92": 10.7824, + "93": 10.75549, + "94": 10.76106, + "95": 10.7731, + "96": 10.73578, + "97": 10.74142, + "98": 10.74708, + "99": 10.76219, + "100": 10.74211 + } + }, + "num-zeros": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 48601.0, + "2": 50656.0, + "3": 49241.0, + "4": 48921.0, + "5": 50418.0, + "6": 49904.0, + "7": 51873.0, + "8": 50118.0, + "9": 50874.0, + "10": 52037.0, + "11": 50176.0, + "12": 48637.0, + "13": 52081.0, + "14": 50045.0, + "15": 48440.0, + "16": 49755.0, + "17": 49654.0, + "18": 51941.0, + "19": 51305.0, + "20": 52452.0, + "21": 50160.0, + "22": 49759.0, + "23": 53668.0, + "24": 51479.0, + "25": 48721.0, + "26": 52060.0, + "27": 51901.0, + "28": 48375.0, + "29": 50453.0, + "30": 51324.0, + "31": 52343.0, + "32": 49490.0, + "33": 51963.0, + "34": 50282.0, + "35": 51151.0, + "36": 51128.0, + "37": 49489.0, + "38": 50866.0, + "39": 50922.0, + "40": 50972.0, + "41": 48289.0, + "42": 50130.0, + "43": 51143.0, + "44": 47593.0, + "45": 54555.0, + "46": 48428.0, + "47": 50067.0, + "48": 50925.0, + "49": 54379.0, + "50": 51185.0, + "51": 48440.0, + "52": 50898.0, + "53": 49761.0, + "54": 48987.0, + "55": 49733.0, + "56": 48541.0, + "57": 48439.0, + "58": 52836.0, + "59": 50186.0, + "60": 49645.0, + "61": 47579.0, + "62": 50843.0, + "63": 48799.0, + "64": 56194.0, + "65": 51032.0, + "66": 49352.0, + "67": 52039.0, + "68": 49477.0, + "69": 54578.0, + "70": 49750.0, + "71": 49318.0, + "72": 52184.0, + "73": 53016.0, + "74": 49953.0, + "75": 50635.0, + "76": 51334.0, + "77": 50040.0, + "78": 49918.0, + "79": 48849.0, + "80": 51988.0, + "81": 50001.0, + "82": 46662.0, + "83": 52909.0, + "84": 48924.0, + "85": 50721.0, + "86": 49065.0, + "87": 46136.0, + "88": 48331.0, + "89": 48705.0, + "90": 52429.0, + "91": 50491.0, + "92": 50216.0, + "93": 50541.0, + "94": 51146.0, + "95": 48468.0, + "96": 49648.0, + "97": 49767.0, + "98": 48502.0, + "99": 49428.0, + "100": 47072.0 + } + }, + "mem-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 1257367552.0, + "2": 1257389056.0, + "3": 1257372160.0, + "4": 1257369600.0, + "5": 1257382400.0, + "6": 1257367040.0, + "7": 1257362944.0, + "8": 1257389056.0, + "9": 1257339904.0, + "10": 1257339392.0, + "11": 1257327616.0, + "12": 1257354240.0, + "13": 1257356800.0, + "14": 1257344000.0, + "15": 1257376256.0, + "16": 1257363968.0, + "17": 1257374208.0, + "18": 1257352704.0, + "19": 1257348096.0, + "20": 1257361408.0, + "21": 1257383424.0, + "22": 1257317376.0, + "23": 1257361408.0, + "24": 1257375232.0, + "25": 1257339904.0, + "26": 1257375232.0, + "27": 1257379328.0, + "28": 1257344512.0, + "29": 1257361920.0, + "30": 1257371136.0, + "31": 1257334272.0, + "32": 1257356288.0, + "33": 1257341440.0, + "34": 1257382400.0, + "35": 1257312256.0, + "36": 1257382912.0, + "37": 1257352704.0, + "38": 1257382400.0, + "39": 1257344000.0, + "40": 1257394176.0, + "41": 1257356800.0, + "42": 1257335808.0, + "43": 1257340416.0, + "44": 1257348608.0, + "45": 1257361920.0, + "46": 1257313792.0, + "47": 1257336832.0, + "48": 1257327616.0, + "49": 1257357824.0, + "50": 1257340928.0, + "51": 1257310208.0, + "52": 1257357824.0, + "53": 1257312768.0, + "54": 1257373696.0, + "55": 1257321472.0, + "56": 1257371136.0, + "57": 1257341952.0, + "58": 1257348096.0, + "59": 1257347584.0, + "60": 1257311744.0, + "61": 1257295360.0, + "62": 1257329152.0, + "63": 1257332224.0, + "64": 1257348608.0, + "65": 1257352192.0, + "66": 1257273856.0, + "67": 1257360384.0, + "68": 1257308160.0, + "69": 1257333248.0, + "70": 1257301504.0, + "71": 1257320960.0, + "72": 1257318400.0, + "73": 1257319936.0, + "74": 1257283584.0, + "75": 1257308160.0, + "76": 1257313280.0, + "77": 1257349632.0, + "78": 1257342976.0, + "79": 1257314304.0, + "80": 1257260032.0, + "81": 1257303552.0, + "82": 1257275392.0, + "83": 1257269248.0, + "84": 1257310720.0, + "85": 1257284096.0, + "86": 1257277952.0, + "87": 1257269248.0, + "88": 1257281024.0, + "89": 1257268224.0, + "90": 1257305600.0, + "91": 1257270784.0, + "92": 1257271808.0, + "93": 1257287680.0, + "94": 1257270272.0, + "95": 1257265152.0, + "96": 1257272320.0, + "97": 1257265152.0, + "98": 1257313792.0, + "99": 1257251328.0, + "100": 1257255936.0 + } + }, + "mem-max-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 3479111680.0, + "2": 3875777024.0, + "3": 3875777024.0, + "4": 3875777024.0, + "5": 3875777024.0, + "6": 3875777024.0, + "7": 3875777024.0, + "8": 3875777024.0, + "9": 3875777024.0, + "10": 3875777024.0, + "11": 3875777024.0, + "12": 3875777024.0, + "13": 3875777024.0, + "14": 3875777024.0, + "15": 3875777024.0, + "16": 3875777024.0, + "17": 3875777024.0, + "18": 3875777024.0, + "19": 3875777024.0, + "20": 3875777024.0, + "21": 3875777024.0, + "22": 3875777024.0, + "23": 3875777024.0, + "24": 3875777024.0, + "25": 3875777024.0, + "26": 3875777024.0, + "27": 3875777024.0, + "28": 3875777024.0, + "29": 3875777024.0, + "30": 3875777024.0, + "31": 3875777024.0, + "32": 3875777024.0, + "33": 3875777024.0, + "34": 3875777024.0, + "35": 3875777024.0, + "36": 3875777024.0, + "37": 3875777024.0, + "38": 3875777024.0, + "39": 3875777024.0, + "40": 3875777024.0, + "41": 3875777024.0, + "42": 3875777024.0, + "43": 3875777024.0, + "44": 3875777024.0, + "45": 3875777024.0, + "46": 3875777024.0, + "47": 3875777024.0, + "48": 3875777024.0, + "49": 3875777024.0, + "50": 3875777024.0, + "51": 3875777024.0, + "52": 3875777024.0, + "53": 3875777024.0, + "54": 3875777024.0, + "55": 3875777024.0, + "56": 3875777024.0, + "57": 3875777024.0, + "58": 3875777024.0, + "59": 3875777024.0, + "60": 3875777024.0, + "61": 3875777024.0, + "62": 3875777024.0, + "63": 3875777024.0, + "64": 3875777024.0, + "65": 3875777024.0, + "66": 3875777024.0, + "67": 3875777024.0, + "68": 3875777024.0, + "69": 3875777024.0, + "70": 3875777024.0, + "71": 3875777024.0, + "72": 3875777024.0, + "73": 3875777024.0, + "74": 3875777024.0, + "75": 3875777024.0, + "76": 3875777024.0, + "77": 3875777024.0, + "78": 3875777024.0, + "79": 3875777024.0, + "80": 3875777024.0, + "81": 3875777024.0, + "82": 3875777024.0, + "83": 3875777024.0, + "84": 3875777024.0, + "85": 3875777024.0, + "86": 3875777024.0, + "87": 3875777024.0, + "88": 3875777024.0, + "89": 3875777024.0, + "90": 3875777024.0, + "91": 3875777024.0, + "92": 3875777024.0, + "93": 3875777024.0, + "94": 3875777024.0, + "95": 3875777024.0, + "96": 3875777024.0, + "97": 3875777024.0, + "98": 3875777024.0, + "99": 3875777024.0, + "100": 3875777024.0 + } + }, + "iteration-time": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": "nan", + "2": 4.90903, + "3": 0.72987, + "4": 0.70519, + "5": 0.69147, + "6": 0.66775, + "7": 0.68374, + "8": 0.67766, + "9": 0.67333, + "10": 0.69935, + "11": 0.67646, + "12": 0.66659, + "13": 0.68781, + "14": 0.67233, + "15": 0.67774, + "16": 0.67467, + "17": 0.66907, + "18": 0.66449, + "19": 0.68006, + "20": 0.66355, + "21": 0.67472, + "22": 0.66709, + "23": 0.66598, + "24": 0.66738, + "25": 0.65573, + "26": 0.65754, + "27": 0.66009, + "28": 0.66367, + "29": 0.6509, + "30": 0.65101, + "31": 0.6674, + "32": 0.6727, + "33": 0.66404, + "34": 0.66244, + "35": 0.66492, + "36": 0.66368, + "37": 0.67258, + "38": 0.65887, + "39": 0.65308, + "40": 0.66256, + "41": 0.66257, + "42": 0.66037, + "43": 0.66807, + "44": 0.65994, + "45": 0.66633, + "46": 0.65686, + "47": 0.66158, + "48": 0.66441, + "49": 0.66691, + "50": 0.6666, + "51": 0.74717, + "52": 0.68959, + "53": 0.68197, + "54": 0.65359, + "55": 0.67803, + "56": 0.66091, + "57": 0.67686, + "58": 0.66096, + "59": 0.66567, + "60": 0.67938, + "61": 0.66497, + "62": 0.67231, + "63": 0.67488, + "64": 0.66956, + "65": 0.66887, + "66": 0.6562, + "67": 0.67334, + "68": 0.67364, + "69": 0.67741, + "70": 0.66866, + "71": 0.66849, + "72": 0.67614, + "73": 0.67758, + "74": 0.66573, + "75": 0.66271, + "76": 0.66488, + "77": 0.65585, + "78": 0.66728, + "79": 0.67574, + "80": 0.66834, + "81": 0.66631, + "82": 0.67168, + "83": 0.67853, + "84": 0.6634, + "85": 0.66574, + "86": 0.67974, + "87": 0.67622, + "88": 0.66866, + "89": 0.67442, + "90": 0.67448, + "91": 0.68094, + "92": 0.66755, + "93": 0.67661, + "94": 0.67926, + "95": 0.66852, + "96": 0.6691, + "97": 0.66801, + "98": 0.66945, + "99": 0.67927, + "100": 0.67769 + } + } +} \ No newline at end of file diff --git a/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_muon_1node/model_config.yaml b/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_muon_1node/model_config.yaml new file mode 100644 index 00000000000..e1653ce7ed4 --- /dev/null +++ b/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_ep8_resume_torch_dist_muon_1node/model_config.yaml @@ -0,0 +1,70 @@ +ENV_VARS: + CUDA_DEVICE_MAX_CONNECTIONS: 1 + NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0 + NCCL_ALGO: Ring + CUBLAS_WORKSPACE_CONFIG: :4096:8 +MODEL_ARGS: + --num-layers: 12 + --hidden-size: 512 + --num-attention-heads: 8 + --log-params-norm: true + --log-num-zeros-in-grad: true + --log-validation-ppl-to-tensorboard: true + --log-timers-to-tensorboard: true + --tensorboard-dir: ${TENSORBOARD_PATH} + --micro-batch-size: 4 + --global-batch-size: 32 + --seq-length: 1024 + --max-position-embeddings: 1024 + --disable-bias-linear: true + --train-iters: 100 + --timing-log-level: 0 + --lr-decay-iters: 320000 + --save: ${CHECKPOINT_SAVE_PATH} + --load: ${CHECKPOINT_LOAD_PATH} + --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document + --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json + --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt + --split: 949,50,1 + --distributed-backend: nccl + --lr: 0.00015 + --lr-decay-style: cosine + --min-lr: 1.0e-5 + --weight-decay: 1e-2 + --clip-grad: 1.0 + --lr-warmup-fraction: .01 + --log-interval: 1 + --save-interval: 50 + --eval-interval: 1000 + --eval-iters: 10 + --transformer-impl: transformer_engine + --tensor-model-parallel-size: 1 + --pipeline-model-parallel-size: 1 + --expert-model-parallel-size: 4 + --num-experts: 8 + --moe-token-dispatcher-type: allgather + --moe-router-load-balancing-type: aux_loss + --moe-router-topk: 2 + --moe-router-dtype: fp32 + --moe-ffn-hidden-size: 1024 + --moe-grouped-gemm: true + --ckpt-fully-parallel-load: true + --deterministic-mode: true + --no-gradient-accumulation-fusion: true + --attention-softmax-in-fp32: true + --use-checkpoint-opt_param-scheduler: true + --use-mcore-models: true + --ckpt-format: torch_dist + --ckpt-assume-constant-structure: true + --data-cache-path: ${DATA_CACHE_PATH} + --bf16: true + --no-bias-gelu-fusion: true + --log-memory-to-tensorboard: true + --optimizer: muon + --muon-momentum: 0.9 + --muon-extra-scale-factor: 0.2 + --muon-scale-mode: spectral + --check-weight-hash-across-dp-replicas-interval: 1 + --async-save: true + --use-persistent-ckpt-worker: true +TEST_TYPE: ckpt-resume diff --git a/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_resume_torch_dist_dist_optimizer_1node/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_resume_torch_dist_dist_optimizer_1node/golden_values_dev_dgx_gb200.json new file mode 100644 index 00000000000..414923c1924 --- /dev/null +++ b/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_resume_torch_dist_dist_optimizer_1node/golden_values_dev_dgx_gb200.json @@ -0,0 +1,537 @@ +{ + "lm loss": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 10.92183, + "2": 10.9077, + "3": 10.90928, + "4": 10.9197, + "5": 10.90609, + "6": 10.91532, + "7": 10.92252, + "8": 10.90995, + "9": 10.91161, + "10": 10.90689, + "11": 10.88851, + "12": 10.90503, + "13": 10.89514, + "14": 10.89598, + "15": 10.87477, + "16": 10.86137, + "17": 10.86454, + "18": 10.85029, + "19": 10.85483, + "20": 10.78398, + "21": 10.77763, + "22": 10.75638, + "23": 10.74498, + "24": 10.71724, + "25": 10.71549, + "26": 10.71018, + "27": 10.67229, + "28": 10.60467, + "29": 10.57062, + "30": 10.54185, + "31": 10.54427, + "32": 10.53161, + "33": 10.49774, + "34": 10.47495, + "35": 10.47083, + "36": 10.44074, + "37": 10.41615, + "38": 10.41757, + "39": 10.38481, + "40": 10.36819, + "41": 10.34676, + "42": 10.33402, + "43": 10.31355, + "44": 10.28912, + "45": 10.29322, + "46": 10.26555, + "47": 10.24535, + "48": 10.19867, + "49": 10.19508, + "50": 10.1961, + "51": 10.20234, + "52": 10.15458, + "53": 10.16274, + "54": 10.12749, + "55": 10.09557, + "56": 10.1248, + "57": 10.11363, + "58": 10.12436, + "59": 10.07391, + "60": 10.0937, + "61": 10.04301, + "62": 10.0162, + "63": 10.08729, + "64": 10.03767, + "65": 10.01755, + "66": 10.04075, + "67": 10.01819, + "68": 9.97999, + "69": 10.00344, + "70": 9.98385, + "71": 10.01034, + "72": 9.99152, + "73": 9.98086, + "74": 9.96234, + "75": 9.94256, + "76": 9.97138, + "77": 9.96086, + "78": 9.91565, + "79": 9.92236, + "80": 9.93093, + "81": 9.9588, + "82": 9.89549, + "83": 9.8631, + "84": 9.79593, + "85": 9.78814, + "86": 9.88181, + "87": 9.90776, + "88": 9.88258, + "89": 9.82994, + "90": 9.82197, + "91": 9.82939, + "92": 9.82025, + "93": 9.75994, + "94": 9.83142, + "95": 9.82809, + "96": 9.80747, + "97": 9.75381, + "98": 9.78418, + "99": 9.82603, + "100": 9.71824 + } + }, + "num-zeros": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 2443.0, + "2": 2426.0, + "3": 2525.0, + "4": 2507.0, + "5": 2396.0, + "6": 2390.0, + "7": 2713.0, + "8": 2450.0, + "9": 2503.0, + "10": 2522.0, + "11": 2501.0, + "12": 2415.0, + "13": 2653.0, + "14": 2579.0, + "15": 2431.0, + "16": 2402.0, + "17": 2493.0, + "18": 2552.0, + "19": 2604.0, + "20": 2483.0, + "21": 2598.0, + "22": 2493.0, + "23": 2503.0, + "24": 2584.0, + "25": 2435.0, + "26": 2468.0, + "27": 2494.0, + "28": 2636.0, + "29": 2684.0, + "30": 2519.0, + "31": 2839.0, + "32": 2808.0, + "33": 6456.0, + "34": 2874.0, + "35": 527564.0, + "36": 527663.0, + "37": 26089.0, + "38": 527761.0, + "39": 527729.0, + "40": 527902.0, + "41": 31694.0, + "42": 527567.0, + "43": 528026.0, + "44": 527859.0, + "45": 30402.0, + "46": 528061.0, + "47": 528003.0, + "48": 528169.0, + "49": 528568.0, + "50": 550768.0, + "51": 531413.0, + "52": 533217.0, + "53": 1059324.0, + "54": 554963.0, + "55": 554219.0, + "56": 558038.0, + "57": 1052795.0, + "58": 1078748.0, + "59": 558355.0, + "60": 582120.0, + "61": 587284.0, + "62": 1578182.0, + "63": 557832.0, + "64": 557805.0, + "65": 1578376.0, + "66": 1055033.0, + "67": 1085742.0, + "68": 1053875.0, + "69": 561563.0, + "70": 531055.0, + "71": 1054821.0, + "72": 550583.0, + "73": 1578398.0, + "74": 1075460.0, + "75": 557948.0, + "76": 1578104.0, + "77": 550589.0, + "78": 556345.0, + "79": 1077900.0, + "80": 578114.0, + "81": 1059016.0, + "82": 1055102.0, + "83": 1577764.0, + "84": 575310.0, + "85": 556803.0, + "86": 554852.0, + "87": 575521.0, + "88": 559536.0, + "89": 1053598.0, + "90": 1578544.0, + "91": 1053540.0, + "92": 1053497.0, + "93": 35286.0, + "94": 1053482.0, + "95": 530157.0, + "96": 1055299.0, + "97": 1054056.0, + "98": 1054267.0, + "99": 577154.0, + "100": 1578785.0 + } + }, + "mem-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 906272768.0, + "2": 906273792.0, + "3": 906273792.0, + "4": 906273792.0, + "5": 906273792.0, + "6": 906273792.0, + "7": 906273792.0, + "8": 906273792.0, + "9": 906273792.0, + "10": 906273792.0, + "11": 906273792.0, + "12": 906273792.0, + "13": 906273792.0, + "14": 906273792.0, + "15": 906273792.0, + "16": 906273792.0, + "17": 906273792.0, + "18": 906273792.0, + "19": 906273792.0, + "20": 906273792.0, + "21": 906273792.0, + "22": 906273792.0, + "23": 906273792.0, + "24": 906273792.0, + "25": 906273792.0, + "26": 906273792.0, + "27": 906273792.0, + "28": 906273792.0, + "29": 906273792.0, + "30": 906273792.0, + "31": 906273792.0, + "32": 906273792.0, + "33": 906273792.0, + "34": 906273792.0, + "35": 906273792.0, + "36": 906273792.0, + "37": 906273792.0, + "38": 906273792.0, + "39": 906273792.0, + "40": 906273792.0, + "41": 906273792.0, + "42": 906273792.0, + "43": 906273792.0, + "44": 906273792.0, + "45": 906273792.0, + "46": 906273792.0, + "47": 906273792.0, + "48": 906273792.0, + "49": 906273792.0, + "50": 906273792.0, + "51": 906273792.0, + "52": 906273792.0, + "53": 906273792.0, + "54": 906273792.0, + "55": 906273792.0, + "56": 906273792.0, + "57": 906273792.0, + "58": 906273792.0, + "59": 906273792.0, + "60": 906273792.0, + "61": 906273792.0, + "62": 906273792.0, + "63": 906273792.0, + "64": 906273792.0, + "65": 906273792.0, + "66": 906273792.0, + "67": 906273792.0, + "68": 906273792.0, + "69": 906273792.0, + "70": 906273792.0, + "71": 906273792.0, + "72": 906273792.0, + "73": 906273792.0, + "74": 906273792.0, + "75": 906273792.0, + "76": 906273792.0, + "77": 906273792.0, + "78": 906273792.0, + "79": 906273792.0, + "80": 906273792.0, + "81": 906273792.0, + "82": 906273792.0, + "83": 906273792.0, + "84": 906273792.0, + "85": 906273792.0, + "86": 906273792.0, + "87": 906273792.0, + "88": 906273792.0, + "89": 906273792.0, + "90": 906273792.0, + "91": 906273792.0, + "92": 906273792.0, + "93": 906273792.0, + "94": 906273792.0, + "95": 906273792.0, + "96": 906273792.0, + "97": 906273792.0, + "98": 906273792.0, + "99": 906273792.0, + "100": 906273792.0 + } + }, + "mem-max-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 1322251776.0, + "2": 1614032384.0, + "3": 1617393664.0, + "4": 1618344960.0, + "5": 1618880512.0, + "6": 1618880512.0, + "7": 1618880512.0, + "8": 1618880512.0, + "9": 1618880512.0, + "10": 1618880512.0, + "11": 1618880512.0, + "12": 1618880512.0, + "13": 1618880512.0, + "14": 1618880512.0, + "15": 1618880512.0, + "16": 1618880512.0, + "17": 1618880512.0, + "18": 1618880512.0, + "19": 1618880512.0, + "20": 1618880512.0, + "21": 1618880512.0, + "22": 1618880512.0, + "23": 1618880512.0, + "24": 1618880512.0, + "25": 1618880512.0, + "26": 1618880512.0, + "27": 1618880512.0, + "28": 1618880512.0, + "29": 1618880512.0, + "30": 1618880512.0, + "31": 1618880512.0, + "32": 1618880512.0, + "33": 1618880512.0, + "34": 1618880512.0, + "35": 1618880512.0, + "36": 1618880512.0, + "37": 1618880512.0, + "38": 1618880512.0, + "39": 1618880512.0, + "40": 1618880512.0, + "41": 1618880512.0, + "42": 1618880512.0, + "43": 1618880512.0, + "44": 1618880512.0, + "45": 1618880512.0, + "46": 1618880512.0, + "47": 1618880512.0, + "48": 1618880512.0, + "49": 1618880512.0, + "50": 1618880512.0, + "51": 1618880512.0, + "52": 1618880512.0, + "53": 1618880512.0, + "54": 1618880512.0, + "55": 1618880512.0, + "56": 1618880512.0, + "57": 1618880512.0, + "58": 1618880512.0, + "59": 1618880512.0, + "60": 1618880512.0, + "61": 1618880512.0, + "62": 1618880512.0, + "63": 1618880512.0, + "64": 1618880512.0, + "65": 1618880512.0, + "66": 1618880512.0, + "67": 1618880512.0, + "68": 1618880512.0, + "69": 1618880512.0, + "70": 1618880512.0, + "71": 1618880512.0, + "72": 1618880512.0, + "73": 1618880512.0, + "74": 1618880512.0, + "75": 1618880512.0, + "76": 1618880512.0, + "77": 1618880512.0, + "78": 1618880512.0, + "79": 1618880512.0, + "80": 1618880512.0, + "81": 1618880512.0, + "82": 1618880512.0, + "83": 1618880512.0, + "84": 1618880512.0, + "85": 1618880512.0, + "86": 1618880512.0, + "87": 1618880512.0, + "88": 1618880512.0, + "89": 1618880512.0, + "90": 1618880512.0, + "91": 1618880512.0, + "92": 1618880512.0, + "93": 1618880512.0, + "94": 1618880512.0, + "95": 1618880512.0, + "96": 1618880512.0, + "97": 1618880512.0, + "98": 1618880512.0, + "99": 1618880512.0, + "100": 1618880512.0 + } + }, + "iteration-time": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": "nan", + "2": 7.17367, + "3": 2.36997, + "4": 1.88234, + "5": 1.60485, + "6": 2.01268, + "7": 1.6239, + "8": 2.24846, + "9": 2.05937, + "10": 2.07514, + "11": 1.79596, + "12": 2.07647, + "13": 1.84504, + "14": 1.93246, + "15": 1.92602, + "16": 1.8095, + "17": 2.19946, + "18": 1.85941, + "19": 1.79313, + "20": 1.94342, + "21": 2.16326, + "22": 1.73336, + "23": 2.14566, + "24": 1.63949, + "25": 2.00506, + "26": 2.2635, + "27": 1.61718, + "28": 2.19724, + "29": 1.64472, + "30": 1.63178, + "31": 1.63652, + "32": 2.58541, + "33": 1.7057, + "34": 1.63187, + "35": 2.16804, + "36": 1.85951, + "37": 1.81326, + "38": 2.12646, + "39": 1.89071, + "40": 1.76913, + "41": 2.14446, + "42": 2.61166, + "43": 1.70492, + "44": 2.16013, + "45": 1.97522, + "46": 1.80839, + "47": 1.83702, + "48": 2.04579, + "49": 1.91283, + "50": 1.65676, + "51": 1.77303, + "52": 1.82896, + "53": 1.92652, + "54": 2.02397, + "55": 2.00098, + "56": 2.32737, + "57": 1.63052, + "58": 1.9368, + "59": 1.65491, + "60": 1.67199, + "61": 1.96779, + "62": 2.25377, + "63": 1.63514, + "64": 1.9377, + "65": 1.75827, + "66": 1.87479, + "67": 2.11588, + "68": 1.65166, + "69": 1.63955, + "70": 2.46109, + "71": 3.08416, + "72": 2.01424, + "73": 2.11351, + "74": 1.62562, + "75": 1.67179, + "76": 2.29454, + "77": 1.78459, + "78": 1.63262, + "79": 2.32882, + "80": 2.08221, + "81": 1.97846, + "82": 2.23858, + "83": 1.80306, + "84": 1.64936, + "85": 2.41125, + "86": 1.71738, + "87": 2.64579, + "88": 1.9064, + "89": 1.93713, + "90": 2.045, + "91": 1.95994, + "92": 2.26963, + "93": 1.67215, + "94": 2.17149, + "95": 1.79179, + "96": 2.21942, + "97": 1.66088, + "98": 1.6611, + "99": 2.60608, + "100": 1.66523 + } + } +} \ No newline at end of file diff --git a/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_resume_torch_dist_dist_optimizer_1node/model_config.yaml b/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_resume_torch_dist_dist_optimizer_1node/model_config.yaml new file mode 100644 index 00000000000..650a234ecc0 --- /dev/null +++ b/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_resume_torch_dist_dist_optimizer_1node/model_config.yaml @@ -0,0 +1,70 @@ +ENV_VARS: + CUDA_DEVICE_MAX_CONNECTIONS: 1 + NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0 + NCCL_ALGO: Ring + CUBLAS_WORKSPACE_CONFIG: :4096:8 +MODEL_ARGS: + --num-layers: 12 + --hidden-size: 512 + --num-attention-heads: 8 + --log-params-norm: true + --log-num-zeros-in-grad: true + --log-validation-ppl-to-tensorboard: true + --log-timers-to-tensorboard: true + --tensorboard-dir: ${TENSORBOARD_PATH} + --micro-batch-size: 4 + --global-batch-size: 32 + --seq-length: 1024 + --max-position-embeddings: 1024 + --disable-bias-linear: true + --train-iters: 100 + --timing-log-level: 0 + --lr-decay-iters: 320000 + --save: ${CHECKPOINT_SAVE_PATH} + --load: ${CHECKPOINT_LOAD_PATH} + --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document + --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json + --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt + --split: 949,50,1 + --distributed-backend: nccl + --lr: 0.00015 + --lr-decay-style: cosine + --min-lr: 1.0e-5 + --weight-decay: 1e-2 + --clip-grad: 1.0 + --lr-warmup-fraction: .01 + --log-interval: 1 + --save-interval: 50 + --eval-interval: 1000 + --eval-iters: 10 + --transformer-impl: transformer_engine + --tensor-model-parallel-size: 4 + --pipeline-model-parallel-size: 1 + --expert-model-parallel-size: 2 + --expert-tensor-parallel-size: 2 + --sequence-parallel: true + --num-experts: 8 + --use-distributed-optimizer: true + --moe-token-dispatcher-type: alltoall + --moe-router-load-balancing-type: aux_loss + --moe-router-topk: 2 + --moe-router-dtype: fp32 + --moe-ffn-hidden-size: 1024 + --moe-shared-expert-intermediate-size: 512 + --moe-grouped-gemm: true + --ckpt-fully-parallel-load: true + --deterministic-mode: true + --no-gradient-accumulation-fusion: true + --attention-softmax-in-fp32: true + --use-checkpoint-opt_param-scheduler: true + --use-mcore-models: true + --ckpt-format: torch_dist + --ckpt-assume-constant-structure: true + --data-cache-path: ${DATA_CACHE_PATH} + --bf16: true + --no-bias-gelu-fusion: true + --log-memory-to-tensorboard: true + --async-save: true + --use-persistent-ckpt-worker: true +TEST_TYPE: ckpt-resume +LAUNCHER: ft_launcher diff --git a/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_scoped_cudagraph_1node/golden_values_dev_dgx_gb200.json b/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_scoped_cudagraph_1node/golden_values_dev_dgx_gb200.json new file mode 100644 index 00000000000..d82c4eb4512 --- /dev/null +++ b/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_scoped_cudagraph_1node/golden_values_dev_dgx_gb200.json @@ -0,0 +1,644 @@ +{ + "lm loss": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 10.91588, + "2": 10.90184, + "3": 10.90036, + "4": 10.8985, + "5": 10.91061, + "6": 10.90083, + "7": 10.89149, + "8": 10.89639, + "9": 10.90451, + "10": 10.91489, + "11": 10.89661, + "12": 10.8917, + "13": 10.88954, + "14": 10.88614, + "15": 10.85108, + "16": 10.85211, + "17": 10.84991, + "18": 10.82547, + "19": 10.82709, + "20": 10.78487, + "21": 10.72652, + "22": 10.72776, + "23": 10.70191, + "24": 10.68491, + "25": 10.66936, + "26": 10.67853, + "27": 10.6252, + "28": 10.56185, + "29": 10.52939, + "30": 10.50384, + "31": 10.48555, + "32": 10.45505, + "33": 10.40871, + "34": 10.41317, + "35": 10.3906, + "36": 10.35863, + "37": 10.33986, + "38": 10.3391, + "39": 10.3013, + "40": 10.29549, + "41": 10.26326, + "42": 10.21451, + "43": 10.20958, + "44": 10.16925, + "45": 10.19889, + "46": 10.13447, + "47": 10.12936, + "48": 10.08452, + "49": 10.09798, + "50": 10.0824, + "51": 10.09868, + "52": 10.04035, + "53": 10.02775, + "54": 10.01433, + "55": 9.97183, + "56": 10.00445, + "57": 9.99802, + "58": 10.00295, + "59": 9.9287, + "60": 9.97012, + "61": 9.91698, + "62": 9.87553, + "63": 9.98345, + "64": 9.91548, + "65": 9.86891, + "66": 9.89926, + "67": 9.87096, + "68": 9.81568, + "69": 9.84759, + "70": 9.82292, + "71": 9.86035, + "72": 9.82942, + "73": 9.79802, + "74": 9.7925, + "75": 9.75336, + "76": 9.80597, + "77": 9.79973, + "78": 9.74622, + "79": 9.73623, + "80": 9.76025, + "81": 9.78589, + "82": 9.71866, + "83": 9.65533, + "84": 9.60421, + "85": 9.56793, + "86": 9.70175, + "87": 9.73397, + "88": 9.70809, + "89": 9.6311, + "90": 9.61504, + "91": 9.66544, + "92": 9.64234, + "93": 9.54639, + "94": 9.65479, + "95": 9.62545, + "96": 9.61898, + "97": 9.52876, + "98": 9.57219, + "99": 9.64071, + "100": 9.5252 + } + }, + "num-zeros": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 22985898.0, + "2": 22867204.0, + "3": 22719338.0, + "4": 22793492.0, + "5": 22800688.0, + "6": 22759160.0, + "7": 22889588.0, + "8": 22617256.0, + "9": 22771052.0, + "10": 22482836.0, + "11": 22768406.0, + "12": 22647152.0, + "13": 23376528.0, + "14": 23021332.0, + "15": 22728996.0, + "16": 22844724.0, + "17": 22956648.0, + "18": 23025800.0, + "19": 23122132.0, + "20": 22738176.0, + "21": 22939332.0, + "22": 22975632.0, + "23": 22636864.0, + "24": 22886054.0, + "25": 22646996.0, + "26": 23036638.0, + "27": 22820628.0, + "28": 23032042.0, + "29": 23008068.0, + "30": 22978312.0, + "31": 22931800.0, + "32": 22672272.0, + "33": 22754260.0, + "34": 23115556.0, + "35": 22764504.0, + "36": 22708706.0, + "37": 23140732.0, + "38": 22991324.0, + "39": 23018016.0, + "40": 22767276.0, + "41": 23101932.0, + "42": 22700208.0, + "43": 23019112.0, + "44": 22716560.0, + "45": 22868708.0, + "46": 22743342.0, + "47": 22872300.0, + "48": 22852860.0, + "49": 22908166.0, + "50": 22654360.0, + "51": 22713880.0, + "52": 22833060.0, + "53": 22987656.0, + "54": 22806978.0, + "55": 22950712.0, + "56": 22670468.0, + "57": 23234272.0, + "58": 22699672.0, + "59": 22862372.0, + "60": 23046518.0, + "61": 22688440.0, + "62": 22743716.0, + "63": 22643988.0, + "64": 23031736.0, + "65": 23243604.0, + "66": 22705288.0, + "67": 22986358.0, + "68": 22949504.0, + "69": 23193436.0, + "70": 22838430.0, + "71": 22749944.0, + "72": 23155156.0, + "73": 23168500.0, + "74": 22969776.0, + "75": 22903244.0, + "76": 22714108.0, + "77": 23011226.0, + "78": 23010276.0, + "79": 22845388.0, + "80": 22958164.0, + "81": 22850072.0, + "82": 22746082.0, + "83": 22741616.0, + "84": 23135482.0, + "85": 22945660.0, + "86": 23108132.0, + "87": 22369162.0, + "88": 22565092.0, + "89": 22737938.0, + "90": 22782088.0, + "91": 22941008.0, + "92": 22680516.0, + "93": 22648188.0, + "94": 23168816.0, + "95": 22701980.0, + "96": 22867212.0, + "97": 22852324.0, + "98": 22897468.0, + "99": 22645688.0, + "100": 23029594.0 + } + }, + "mem-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 1030661632.0, + "2": 1052299264.0, + "3": 1061900288.0, + "4": 1071962624.0, + "5": 1071962624.0, + "6": 1071962624.0, + "7": 1071962624.0, + "8": 1071962624.0, + "9": 1074813440.0, + "10": 1074600448.0, + "11": 1074939392.0, + "12": 1074707968.0, + "13": 1074788864.0, + "14": 1071962624.0, + "15": 1071962624.0, + "16": 1071962624.0, + "17": 1071962624.0, + "18": 1071962624.0, + "19": 1071962624.0, + "20": 1075222016.0, + "21": 1071962624.0, + "22": 1071962624.0, + "23": 1074786816.0, + "24": 1075165696.0, + "25": 1071962624.0, + "26": 1071962624.0, + "27": 1071962624.0, + "28": 1071962624.0, + "29": 1071962624.0, + "30": 1071962624.0, + "31": 1071962624.0, + "32": 1071962624.0, + "33": 1071962624.0, + "34": 1071962624.0, + "35": 1071962624.0, + "36": 1071962624.0, + "37": 1071962624.0, + "38": 1071962624.0, + "39": 1071962624.0, + "40": 1071962624.0, + "41": 1071962624.0, + "42": 1071962624.0, + "43": 1074748928.0, + "44": 1074986496.0, + "45": 1074682368.0, + "46": 1075206656.0, + "47": 1071962624.0, + "48": 1071962624.0, + "49": 1071962624.0, + "50": 1071962624.0, + "51": 1075486208.0, + "52": 1071962624.0, + "53": 1074903552.0, + "54": 1074600448.0, + "55": 1075408384.0, + "56": 1074600448.0, + "57": 1071962624.0, + "58": 1071962624.0, + "59": 1071962624.0, + "60": 1075583488.0, + "61": 1071962624.0, + "62": 1075206656.0, + "63": 1074600448.0, + "64": 1071962624.0, + "65": 1071962624.0, + "66": 1075642880.0, + "67": 1074600448.0, + "68": 1071962624.0, + "69": 1071962624.0, + "70": 1071962624.0, + "71": 1071962624.0, + "72": 1075379712.0, + "73": 1071962624.0, + "74": 1071962624.0, + "75": 1074661888.0, + "76": 1071962624.0, + "77": 1075620352.0, + "78": 1074600448.0, + "79": 1071962624.0, + "80": 1071962624.0, + "81": 1074600448.0, + "82": 1071962624.0, + "83": 1071962624.0, + "84": 1071962624.0, + "85": 1075206656.0, + "86": 1074797056.0, + "87": 1071962624.0, + "88": 1074600448.0, + "89": 1071962624.0, + "90": 1071962624.0, + "91": 1071962624.0, + "92": 1075446272.0, + "93": 1071962624.0, + "94": 1071962624.0, + "95": 1071962624.0, + "96": 1071962624.0, + "97": 1071962624.0, + "98": 1071962624.0, + "99": 1075249664.0, + "100": 1071962624.0 + } + }, + "mem-max-allocated-bytes": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 1097419776.0, + "2": 1426488832.0, + "3": 1426488832.0, + "4": 1426488832.0, + "5": 1426488832.0, + "6": 1426488832.0, + "7": 1426488832.0, + "8": 1426488832.0, + "9": 1426488832.0, + "10": 1426488832.0, + "11": 1426488832.0, + "12": 1426488832.0, + "13": 1426488832.0, + "14": 1426488832.0, + "15": 1426488832.0, + "16": 1426488832.0, + "17": 1426488832.0, + "18": 1426488832.0, + "19": 1426488832.0, + "20": 1426488832.0, + "21": 1426488832.0, + "22": 1426488832.0, + "23": 1426488832.0, + "24": 1426488832.0, + "25": 1426488832.0, + "26": 1426488832.0, + "27": 1426488832.0, + "28": 1426488832.0, + "29": 1426488832.0, + "30": 1426488832.0, + "31": 1426488832.0, + "32": 1426488832.0, + "33": 1426488832.0, + "34": 1426488832.0, + "35": 1426488832.0, + "36": 1426488832.0, + "37": 1426488832.0, + "38": 1426488832.0, + "39": 1426488832.0, + "40": 1426488832.0, + "41": 1426488832.0, + "42": 1426488832.0, + "43": 1426488832.0, + "44": 1426488832.0, + "45": 1426488832.0, + "46": 1426488832.0, + "47": 1426488832.0, + "48": 1426488832.0, + "49": 1426488832.0, + "50": 1426488832.0, + "51": 1426488832.0, + "52": 1426488832.0, + "53": 1426488832.0, + "54": 1426488832.0, + "55": 1426488832.0, + "56": 1426488832.0, + "57": 1426488832.0, + "58": 1426488832.0, + "59": 1426488832.0, + "60": 1426488832.0, + "61": 1426488832.0, + "62": 1426488832.0, + "63": 1426488832.0, + "64": 1426488832.0, + "65": 1426488832.0, + "66": 1426488832.0, + "67": 1426488832.0, + "68": 1426488832.0, + "69": 1426488832.0, + "70": 1426488832.0, + "71": 1426488832.0, + "72": 1426488832.0, + "73": 1426488832.0, + "74": 1426488832.0, + "75": 1426488832.0, + "76": 1426488832.0, + "77": 1426488832.0, + "78": 1426488832.0, + "79": 1426488832.0, + "80": 1426488832.0, + "81": 1426488832.0, + "82": 1426488832.0, + "83": 1426488832.0, + "84": 1426488832.0, + "85": 1426488832.0, + "86": 1426488832.0, + "87": 1426488832.0, + "88": 1426488832.0, + "89": 1426488832.0, + "90": 1426488832.0, + "91": 1426488832.0, + "92": 1426488832.0, + "93": 1426488832.0, + "94": 1426488832.0, + "95": 1426488832.0, + "96": 1426488832.0, + "97": 1426488832.0, + "98": 1426488832.0, + "99": 1426488832.0, + "100": 1426488832.0 + } + }, + "mtp_1 loss": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": 10.92471, + "2": 10.94621, + "3": 10.93864, + "4": 10.93913, + "5": 10.92992, + "6": 10.9275, + "7": 10.93605, + "8": 10.93317, + "9": 10.93661, + "10": 10.93065, + "11": 10.94517, + "12": 10.92353, + "13": 10.94271, + "14": 10.93606, + "15": 10.94086, + "16": 10.93683, + "17": 10.93577, + "18": 10.9352, + "19": 10.93613, + "20": 10.93736, + "21": 10.94508, + "22": 10.93751, + "23": 10.93434, + "24": 10.92963, + "25": 10.94063, + "26": 10.94114, + "27": 10.92887, + "28": 10.93038, + "29": 10.9296, + "30": 10.93961, + "31": 10.91806, + "32": 10.92899, + "33": 10.92159, + "34": 10.92688, + "35": 10.91862, + "36": 10.917, + "37": 10.91536, + "38": 10.91473, + "39": 10.90157, + "40": 10.9197, + "41": 10.90546, + "42": 10.88722, + "43": 10.89763, + "44": 10.87484, + "45": 10.88603, + "46": 10.87926, + "47": 10.87569, + "48": 10.86052, + "49": 10.86102, + "50": 10.84773, + "51": 10.86037, + "52": 10.84549, + "53": 10.85137, + "54": 10.84161, + "55": 10.82734, + "56": 10.83662, + "57": 10.8441, + "58": 10.82265, + "59": 10.81464, + "60": 10.82154, + "61": 10.80216, + "62": 10.79501, + "63": 10.80803, + "64": 10.80094, + "65": 10.78782, + "66": 10.78839, + "67": 10.78222, + "68": 10.76003, + "69": 10.78043, + "70": 10.75405, + "71": 10.75799, + "72": 10.76611, + "73": 10.75116, + "74": 10.73729, + "75": 10.72095, + "76": 10.73456, + "77": 10.72932, + "78": 10.71777, + "79": 10.70884, + "80": 10.70775, + "81": 10.71148, + "82": 10.66765, + "83": 10.66648, + "84": 10.64351, + "85": 10.63273, + "86": 10.6701, + "87": 10.65544, + "88": 10.6544, + "89": 10.62333, + "90": 10.61782, + "91": 10.64542, + "92": 10.62804, + "93": 10.60107, + "94": 10.62753, + "95": 10.61361, + "96": 10.60403, + "97": 10.58649, + "98": 10.60102, + "99": 10.58864, + "100": 10.56583 + } + }, + "iteration-time": { + "start_step": 1, + "end_step": 100, + "step_interval": 1, + "values": { + "1": "nan", + "2": 21.72113, + "3": 4.38181, + "4": 3.75773, + "5": 2.22198, + "6": 2.24816, + "7": 2.21855, + "8": 2.3062, + "9": 2.22068, + "10": 2.21936, + "11": 2.19556, + "12": 2.32476, + "13": 2.20846, + "14": 2.33591, + "15": 2.43288, + "16": 2.21964, + "17": 2.37586, + "18": 2.23756, + "19": 2.22294, + "20": 2.44852, + "21": 2.21899, + "22": 4.27481, + "23": 2.22622, + "24": 2.20254, + "25": 2.19347, + "26": 2.19347, + "27": 2.20091, + "28": 2.21919, + "29": 2.19257, + "30": 2.20443, + "31": 2.20235, + "32": 2.32185, + "33": 2.41, + "34": 2.20384, + "35": 2.28083, + "36": 2.24841, + "37": 2.21647, + "38": 2.22011, + "39": 2.28982, + "40": 2.2143, + "41": 2.22595, + "42": 2.35974, + "43": 2.18075, + "44": 2.32993, + "45": 2.1987, + "46": 2.23006, + "47": 2.23289, + "48": 2.35477, + "49": 2.19673, + "50": 2.20477, + "51": 2.29043, + "52": 2.42878, + "53": 2.17785, + "54": 2.20952, + "55": 2.21678, + "56": 2.59173, + "57": 2.18677, + "58": 2.19748, + "59": 2.17501, + "60": 2.19422, + "61": 2.16902, + "62": 2.44638, + "63": 2.18754, + "64": 2.18608, + "65": 2.1724, + "66": 2.17373, + "67": 2.18153, + "68": 2.19831, + "69": 2.18566, + "70": 2.1811, + "71": 2.69381, + "72": 2.19558, + "73": 2.56698, + "74": 2.16275, + "75": 2.15677, + "76": 2.18512, + "77": 2.15347, + "78": 2.1736, + "79": 2.53832, + "80": 2.15263, + "81": 2.33751, + "82": 2.64826, + "83": 2.31273, + "84": 2.17777, + "85": 2.18388, + "86": 2.1902, + "87": 2.74837, + "88": 2.36972, + "89": 2.17239, + "90": 2.18885, + "91": 2.33106, + "92": 2.42868, + "93": 2.3434, + "94": 2.18941, + "95": 2.51484, + "96": 2.74467, + "97": 2.17694, + "98": 2.22232, + "99": 3.33018, + "100": 2.34624 + } + } +} \ No newline at end of file diff --git a/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_scoped_cudagraph_1node/model_config.yaml b/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_scoped_cudagraph_1node/model_config.yaml new file mode 100644 index 00000000000..a33ffbe1018 --- /dev/null +++ b/tests/functional_tests/test_cases/moe/gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_scoped_cudagraph_1node/model_config.yaml @@ -0,0 +1,97 @@ +ENV_VARS: + CUDA_DEVICE_MAX_CONNECTIONS: 1 + NVTE_ALLOW_NONDETERMINISTIC_ALGO: 0 + NCCL_ALGO: Ring + CUBLAS_WORKSPACE_CONFIG: :4096:8 +MODEL_ARGS: + --num-layers: 13 + --hidden-size: 512 + --num-attention-heads: 8 + --mtp-num-layers: 1 + --micro-batch-size: 2 + --global-batch-size: 32 + --seq-length: 1024 + --max-position-embeddings: 1024 + --position-embedding-type: rope + --rotary-base: 10000 + --untie-embeddings-and-output-weights: true + --disable-bias-linear: true + --attention-dropout: 0.0 + --hidden-dropout: 0.0 + --train-iters: 100 + --lr-decay-iters: 320000 + --split: 949,50,1 + --distributed-backend: nccl + --lr: 0.00015 + --lr-decay-style: cosine + --min-lr: 1.0e-5 + --weight-decay: 1e-2 + --clip-grad: 1.0 + --lr-warmup-fraction: .01 + --transformer-impl: transformer_engine + --tensor-model-parallel-size: 4 + --pipeline-model-parallel-size: 1 + --expert-model-parallel-size: 2 + --expert-tensor-parallel-size: 2 + --sequence-parallel: true + --num-experts: 8 + --use-distributed-optimizer: true + --overlap-grad-reduce: true + --overlap-param-gather: true + --moe-token-dispatcher-type: alltoall + --moe-router-load-balancing-type: global_aux_loss + --moe-router-topk: 2 + --moe-router-dtype: fp32 + --moe-router-fusion: true + --moe-router-enable-expert-bias: true + --moe-router-score-function: sigmoid + --moe-router-pre-softmax: true + --moe-ffn-hidden-size: 1024 + --moe-shared-expert-intermediate-size: 512 + --moe-grouped-gemm: true + --moe-layer-freq: ([0]*4+[1]*9) + --moe-permute-fusion: true + --deterministic-mode: true + --no-gradient-accumulation-fusion: true + --attention-softmax-in-fp32: true + --use-checkpoint-opt_param-scheduler: true + --use-mcore-models: true + --bf16: true + --fp8-format: hybrid + --fp8-recipe: blockwise + --first-last-layers-bf16: true + --no-bias-gelu-fusion: true + --recompute-granularity: selective + --recompute-modules: "[moe_act]" + --cuda-graph-impl: transformer_engine + --cuda-graph-scope: "[attn mlp moe_router moe_preprocess]" + --log-memory-to-tensorboard: true + --log-params-norm: true + --log-num-zeros-in-grad: true + --log-validation-ppl-to-tensorboard: true + --log-timers-to-tensorboard: true + --tensorboard-dir: ${TENSORBOARD_PATH} + --log-interval: 1 + --timing-log-level: 0 + --save-interval: 50 + --eval-interval: 1000 + --eval-iters: 10 + --data-path: ${DATA_PATH}/text/common_pile/v01_filtered_data/my-gpt3_00_text_document + --data-cache-path: ${DATA_CACHE_PATH} + --vocab-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/vocab.json + --merge-file: ${DATA_PATH}/text/common_pile/v01_filtered_data/bpe/merges.txt + --save: ${CHECKPOINT_SAVE_PATH} + --load: ${CHECKPOINT_LOAD_PATH} + --ckpt-fully-parallel-load: true + --ckpt-format: torch_dist + --ckpt-assume-constant-structure: true + --async-save: true + --use-persistent-ckpt-worker: true +TEST_TYPE: ckpt-resume +METRICS: + # - "iteration-time" + - "lm loss" + - "num-zeros" + # - "mem-allocated-bytes" + # - "mem-max-allocated-bytes" + - "mtp_1 loss" diff --git a/tests/test_utils/recipes/gb200/gpt-1node.yaml b/tests/test_utils/recipes/gb200/gpt-1node.yaml new file mode 100644 index 00000000000..67375f6a97b --- /dev/null +++ b/tests/test_utils/recipes/gb200/gpt-1node.yaml @@ -0,0 +1,320 @@ +type: basic +format_version: 1 +maintainers: [mcore] +loggers: [stdout] +spec: + name: "{test_case}_{environment}_{platforms}" + model: gpt + build: mcore-pyt-{environment} + nodes: 1 + gpus: 4 + n_repeat: 5 + platforms: dgx_gb200 + script_setup: | + unset https_proxy + echo "machine gitlab-master.nvidia.com login okoenig password $RO_API_TOKEN" | tee -a /root/.netrc + + # Checkout latest + cd /opt + rm -rf /opt/megatron-lm; mkdir megatron-lm; cd megatron-lm + git init + git remote add origin $MCORE_REPO + git fetch origin '+refs/merge-requests/*:refs/remotes/merge-requests/*' + git fetch origin $MCORE_MR_COMMIT + git checkout $MCORE_MR_COMMIT + git rev-parse HEAD + + # Checkout backwards-ref + cd /opt + rm -rf /opt/megatron-lm-legacy; mkdir megatron-lm-legacy; cd megatron-lm-legacy + git init + git remote add origin $MCORE_REPO + git fetch origin $MCORE_BACKWARDS_COMMIT + git checkout $MCORE_BACKWARDS_COMMIT + git rev-parse HEAD + rm -rf megatron; cp -a /opt/megatron-lm/megatron ./ + script: |- + ls + cd /opt/megatron-lm + + NAME=$(echo {test_case}_{environment} | sed 's/dgx_gb200/dgx_a100/g') + export GPUS_PER_NODE={gpus} + + ARGUMENTS=( + "DATA_PATH=/mnt/artifacts" + "DATA_CACHE_PATH=/lustre/fsw/coreai_dlalgo_mcore/mcore_ci/data/$RUN_ID/cache/" + "OUTPUT_PATH={assets_dir}" + "TENSORBOARD_PATH={assets_dir}/tensorboard" + "CHECKPOINT_SAVE_PATH={artifacts_dir}/checkpoints" + "CHECKPOINT_LOAD_PATH=/mnt/artifacts/" + "TRAINING_SCRIPT_PATH=pretrain_gpt.py" + "TRAINING_PARAMS_PATH=./tests/functional_tests/test_cases/{model}/{test_case}/model_config.yaml" + "GOLDEN_VALUES_PATH=./tests/functional_tests/test_cases/{model}/{test_case}/golden_values_{environment}_{platforms}.json" + "N_REPEAT={n_repeat}" + "ENABLE_LIGHTWEIGHT_MODE=${{ENABLE_LIGHTWEIGHT_MODE}}" + "RECORD_CHECKPOINTS=${{RECORD_CHECKPOINTS}}" + ) + + set +x + bash ./tests/functional_tests/shell_test_utils/run_ci_test.sh ${{ARGUMENTS[@]}} + exit_code=$? + echo "Exit code: $exit_code" + rm -rf /lustre/fsw/coreai_dlalgo_mcore/mcore_ci/data/$RUN_ID || true + set -x + exit $exit_code + +products: + - test_case: [gpt3_mcore_te_tp1_pp1_resume_torch_dist_dist_optimizer_1node] + products: + - environment: [dev] + scope: [mr-github, mr-github-slim] + platforms: [dgx_gb200] + - test_case: [gpt3_mcore_te_tp2_pp1_gdn_1node] + products: + - environment: [dev] + scope: [mr-github] + platforms: [dgx_gb200] + - test_case: [gpt3_mcore_reruns_persistent_1_1node] + products: + - environment: [dev] + scope: [mr-github] + platforms: [dgx_gb200] + - test_case: [gpt3_mcore_te_tp1_pp4_vp1_dist_optimizer_overlap_grad_reduce_param_gather_overlap_optimizer_1node] + products: + - environment: [dev] + scope: [mr-github] + platforms: [dgx_gb200] + - test_case: [gpt3_mcore_te_tp4_pp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_param_gather_1node] + products: + - environment: [dev] + scope: [mr-github, mr-github-slim] + platforms: [dgx_gb200] + ####################################################################### + # tp1_pp1 variants # + ####################################################################### + - test_case: [gpt3_mcore_te_tp1_pp1_dist_optimizer_no_mmap_bin_files_1node] + products: + - environment: [dev] + scope: [mr-github] + platforms: [dgx_gb200] + - test_case: [gpt3_mcore_te_tp1_pp1_dist_optimizer_fim_dataset_1node] + products: + - environment: [dev] + scope: [mr-github] + platforms: [dgx_gb200] + - test_case: [gpt3_mcore_te_tp1_pp1_resume_torch_dist_uniform_full_recompute_1node] + products: + - environment: [dev] + scope: [mr-github] + platforms: [dgx_gb200] + ####################################################################### + # tp1_pp2 variants # + ####################################################################### + - test_case: [gpt3_mcore_te_tp1_pp2_resume_torch_dist_rope_embeddings_1node] + products: + - environment: [dev] + scope: [mr-github] + platforms: [dgx_gb200] + - test_case: [gpt3_mcore_te_tp1_pp2_resume_torch_dist_rope_embeddings_interleaved_no_fusion_1node] + products: + - environment: [dev] + scope: [mr-github] + platforms: [dgx_gb200] + ####################################################################### + # tp1_pp4 variants # + ####################################################################### + - test_case: [gpt3_mcore_te_tp1_pp4_resume_torch_dist_disable_bias_linear_1node] + products: + - environment: [dev] + scope: [mr-github] + platforms: [dgx_gb200] + - test_case: [gpt3_mcore_te_tp1_pp4_resume_torch_dist_persistent_disable_bias_linear_1node] + products: + - environment: [dev] + scope: [mr-github] + platforms: [dgx_gb200] + - test_case: [gpt3_mcore_te_tp1_pp4_resume_torch_dist_swiglu_1node] + products: + - environment: [dev] + scope: [mr-github] + platforms: [dgx_gb200] + - test_case: [gpt3_mcore_te_tp1_pp4_resume_torch_dist_untie_embeddings_and_outputs_1node] + products: + - environment: [dev] + scope: [mr-github] + platforms: [dgx_gb200] + - test_case: [gpt3_mcore_te_tp1_pp4_vp1_1node] + products: + - environment: [dev] + scope: [mr-github-broken] + platforms: [dgx_gb200] + - test_case: [gpt3_mcore_te_tp1_pp4_vp1_resume_torch_decoupled_lr_1node] + products: + - environment: [dev] + scope: [mr-github] + platforms: [dgx_gb200] + - test_case: [gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist_calculate_per_token_loss_1node] + products: + - environment: [dev] + scope: [mr-github] + platforms: [dgx_gb200] + - test_case: [gpt3_mcore_te_tp1_pp4_vp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_1node] + products: + - environment: [dev] + scope: [mr-github] + platforms: [dgx_gb200] + - test_case: [gpt3_mcore_te_tp1_pp4_vp1_tunable_overlap_1node] + products: + - environment: [dev] + scope: [mr-github-broken] + platforms: [dgx_gb200] + - test_case: [gpt3_mcore_te_tp1_pp4_vp1_uneven_pipeline_1node] + products: + - environment: [dev] + scope: [mr-github] + platforms: [dgx_gb200] + - test_case: [gpt3_mcore_te_tp1_pp4_vp2_account_for_embedding_loss_in_pipeline_split_1node] + products: + - environment: [dev] + scope: [mr-github] + platforms: [dgx_gb200] + ####################################################################### + # tp2_pp1 variants # + ####################################################################### + - test_case: [gpt3_mcore_te_tp2_pp1_resume_torch_dist_cp2_nondeterministic_1node] + products: + - environment: [dev] + scope: [mr-github] + platforms: [dgx_gb200] + - test_case: [gpt3_mcore_te_tp2_pp1_resume_torch_dist_multi_dist_optimizer_instances_1node] + products: + - environment: [dev] + scope: [mr-github] + platforms: [dgx_gb200] + - test_case: [gpt3_mcore_te_tp2_zp_z3_resume_fsdp_dtensor_1node] + products: + - environment: [dev] + scope: [mr-github] + platforms: [dgx_gb200] + - test_case: [gpt3_mcore_te_tp2_pp1_modelopt_distill_resume_1node] + products: + - environment: [dev] + scope: [mr-github] + platforms: [dgx_gb200] + ####################################################################### + # tp2_pp2 (no cp) variants # + ####################################################################### + - test_case: [gpt3_mcore_te_tp2_pp2_cross_entropy_loss_fusion_1node] + products: + - environment: [dev] + scope: [mr-github] + platforms: [dgx_gb200] + - test_case: [gpt3_mcore_te_tp2_pp2_resume_torch_dist_cross_entropy_loss_fusion_1node] + products: + - environment: [dev] + scope: [mr-github] + platforms: [dgx_gb200] + - test_case: [gpt3_mcore_te_tp2_pp2_mla_1node] + products: + - environment: [dev] + scope: [mr-github] + platforms: [dgx_gb200] + - test_case: [gpt3_mcore_te_tp2_pp2_resume_torch_dist_ddp_average_in_collective_1node] + products: + - environment: [dev] + scope: [mr-github] + platforms: [dgx_gb200] + - test_case: [gpt3_mcore_te_tp2_pp2_resume_torch_dist_defer_embedding_wgrad_compute_1node] + products: + - environment: [dev] + scope: [mr-github] + platforms: [dgx_gb200] + - test_case: [gpt3_mcore_te_tp2_pp2_resume_torch_dist_1node] + products: + - environment: [dev] + scope: [mr-github] + platforms: [dgx_gb200] + - test_case: [gpt3_mcore_te_tp2_pp2_resume_torch_dist_reshard_1x4xNone_1node] + products: + - environment: [dev] + scope: [mr-github] + platforms: [dgx_gb200] + - test_case: [gpt3_mcore_tp2_pp2_uninstall_te_1node] + products: + - environment: [dev] + scope: [mr-github] + platforms: [dgx_gb200] + ####################################################################### + # tp2_pp2_cp2 variants (PP reduced 2→1 for 1-node) # + ####################################################################### + - test_case: [gpt3_mcore_te_tp2_pp2_cp2_1node] + products: + - environment: [dev] + scope: [mr-github] + platforms: [dgx_gb200] + - test_case: [gpt3_mcore_te_tp2_pp2_cp2_etp4_dp_last_1node] + products: + - environment: [dev] + scope: [mr-github] + platforms: [dgx_gb200] + - test_case: [gpt3_mcore_te_tp2_pp2_cp2_calculate_per_token_loss_1node] + products: + - environment: [dev] + scope: [mr-github] + platforms: [dgx_gb200] + - test_case: [gpt3_mcore_te_tp2_pp2_cp2_etp4_calculate_per_token_loss_dp_last_1node] + products: + - environment: [dev] + scope: [mr-github] + platforms: [dgx_gb200] + - test_case: [gpt3_mcore_te_tp2_pp2_cp2_nondeterministic_1node] + products: + - environment: [dev] + scope: [mr-github] + platforms: [dgx_gb200] + - test_case: [gpt3_mcore_te_tp2_pp2_cp2_etp4_nondeterministic_dp_last_1node] + products: + - environment: [dev] + scope: [mr-github] + platforms: [dgx_gb200] + - test_case: [gpt3_mcore_te_tp2_pp2_cp2_calculate_per_token_loss_nondeterministic_1node] + products: + - environment: [dev] + scope: [mr-github] + platforms: [dgx_gb200] + - test_case: [gpt3_mcore_te_tp2_pp2_cp2_etp4_calculate_per_token_loss_nondeterministic_dp_last_1node] + products: + - environment: [dev] + scope: [mr-github] + platforms: [dgx_gb200] + - test_case: [gpt3_mcore_te_tp2_pp2_resume_torch_dist_cp2_nondeterministic_1node] + products: + - environment: [dev] + scope: [mr-github] + platforms: [dgx_gb200] + ####################################################################### + # tp4_pp1 variants # + ####################################################################### + - test_case: [gpt3_mcore_te_tp4_pp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_1node] + products: + - environment: [dev] + scope: [mr-github] + platforms: [dgx_gb200] + - test_case: [gpt3_mcore_te_tp4_pp1_resume_torch_dist_qk_layernorm_test_mode_1node] + products: + - environment: [dev] + scope: [mr-github] + platforms: [dgx_gb200] + - test_case: [gpt3_mcore_te_tp4_pp1_dist_optimizer_overlap_grad_reduce_param_gather_1node] + products: + - environment: [dev] + scope: [mr-github] + platforms: [dgx_gb200] + ####################################################################### + # tp4_pp2 variant (PP reduced 2→1 for 1-node) # + ####################################################################### + - test_case: [gpt3_mcore_te_tp4_pp2_resume_torch_dist_reshard_8x1xNone_1node] + products: + - environment: [dev] + scope: [mr-github] + platforms: [dgx_gb200] diff --git a/tests/test_utils/recipes/gb200/moe-1node.yaml b/tests/test_utils/recipes/gb200/moe-1node.yaml index 10fe3219c28..cd1e8fe0555 100644 --- a/tests/test_utils/recipes/gb200/moe-1node.yaml +++ b/tests/test_utils/recipes/gb200/moe-1node.yaml @@ -63,3 +63,70 @@ products: - environment: [dev] scope: [mr-github, mr-github-slim] platforms: [dgx_gb200] + - test_case: [gpt3_moe_mcore_te_ep8_resume_torch_dist_muon_1node] + products: + - environment: [dev] + scope: [mr-github, mr-github-slim] + platforms: [dgx_gb200] + - test_case: [gpt3_moe_mcore_te_ep8_resume_torch_dist_dist_muon_1node] + products: + - environment: [dev] + scope: [mr-github] + platforms: [dgx_gb200] + - test_case: [gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_resume_torch_dist_dist_optimizer_1node] + products: + - environment: [dev] + scope: [mr-github] + platforms: [dgx_gb200] + ####################################################################### + # tp1_pp2 MoE variants # + ####################################################################### + - test_case: [gpt3_mcore_te_tp1_pp2_resume_torch_dist_reshard_2x1x4_te_8experts2parallel_dist_optimizer_1node] + products: + - environment: [dev] + scope: [mr-github] + platforms: [dgx_gb200] + ####################################################################### + # tp2_pp1 MoE variants # + ####################################################################### + - test_case: [gpt3_mcore_te_tp2_pp1_resume_torch_dist_te_8experts2parallel_dist_optimizer_1node] + products: + - environment: [dev] + scope: [mr-github] + platforms: [dgx_gb200] + - test_case: [gpt3_mcore_te_tp2_pp1_resume_torch_dist_te_8experts2parallel_multi_dist_optimizer_instances_1node] + products: + - environment: [dev] + scope: [mr-github-broken] + platforms: [dgx_gb200] + - test_case: [gpt3_mcore_te_tp2_pp1_te_8experts2parallel_overlap_grad_reduce_param_gather_groupedGEMM_1node] + products: + - environment: [dev] + scope: [mr-github] + platforms: [dgx_gb200] + - test_case: [gpt3_mcore_te_tp2_pp1_te_a2a_ovlp_8experts_etp1_ep4_1node] + products: + - environment: [dev] + scope: [mr-github] + platforms: [dgx_gb200] + - test_case: [gpt3_mcore_te_tp2_pp1_te_8experts2parallel_ddp_average_in_collective_1node] + products: + - environment: [dev] + scope: [mr-github] + platforms: [dgx_gb200] + ####################################################################### + # ep8 variant (EP reduced 8→4 for 1-node) # + ####################################################################### + - test_case: [gpt3_moe_mcore_te_ep8_resume_torch_dist_dist_optimizer_1node] + products: + - environment: [dev] + scope: [mr-github] + platforms: [dgx_gb200] + ####################################################################### + # tp4_ep2_etp2_pp2 variant (PP reduced 2→1 for 1-node) # + ####################################################################### + - test_case: [gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_scoped_cudagraph_1node] + products: + - environment: [dev] + scope: [mr-github] + platforms: [dgx_gb200]