feat(evals): add registry enrichment and validation tooling - #1
Merged
Arist12 merged 3 commits intoMar 6, 2026
Merged
Conversation
Arist12
added a commit
that referenced
this pull request
Apr 18, 2026
Eval instance for Primus issue #1: optimize Qwen3-30B-A3B (MoE, 128 experts, top-8) pretraining throughput on 8x MI355X GPUs. Frozen launch recipe: TP=1, EP=8, PP=1, mbs=1, gbs=8, seq=8192, BF16. Metric: tflops_per_gpu (higher is better). Baseline: ~300 TFLOP/s/GPU.
jhinpan
added a commit
that referenced
this pull request
May 2, 2026
* add primus-qwen3-30b-mfu eval instance for issue #1 8x MI355X Qwen3-30B-A3B MoE pretraining MFU optimization. Built on ghcr.io/amdpilot-org/primus-mi355x-ready:v1. Two task.yaml variants for A/B testing the Phase 1 baseline agent: - task.yaml: phase1_baseline: true - task_nophase1.yaml: phase1_baseline: false Both pin the executor to Kimi-K2.6 at 10.235.24.154:30000. * fix(primus): use primus-mi355x-flat:v1 as base + install uv The previous base (ghcr.io/amdpilot-org/primus-mi355x-ready:v1) was a slimmed copy that did NOT include /workspace/primus_train/Primus. Phase 1 spent its full max_turns budget trying to bootstrap from scratch. Switch to primus-mi355x-flat:v1 (locally available, has Primus + Primus-Turbo pre-installed and patched for triton 3.4.0). Also install uv at /root/.local/bin in our Dockerfile so the kimi-cli runtime's source $HOME/.local/bin/env succeeds — without this, executor trials exit 137 immediately after image switch. Tag the new image primus-qwen3-30b-mfu-base:v1 so amdpilot triggers a build the first time it runs. * fix(primus): add gfx950 env vars + runtime IFNAME detection Two cumulative fixes for n08-09 (8x MI355X): 1. Without PYTORCH_ROCM_ARCH=gfx950 (and AITER_ROCM_ARCH, HSA_NO_SCRATCH_RECLAIM, HIP_FORCE_DEV_KERNARG, etc) the torch HIP runtime can't dispatch kernels and benchmarks die immediately with hipErrorInvalidDeviceFunction. These were set in xiao/baizhou's working containers but missing from amdpilot's docker run line. Add to both Dockerfile ENV and task.yaml container.env so they apply via either path. 2. Replace /workspace/detect_interface.sh with a /proc-based detector (the original needed `ip` from iproute2, unavailable in the slim base). bench_mfu.sh now auto-detects GLOO/NCCL socket IFNAME at runtime if bench_config.env doesn't pin one — without this, Megatron's distributed init fails fast (3-5s) before training starts. * feat(primus): add phase1_publish config to task.yaml Enables tag + push of the Phase 1 baseline image to docker.io/jhinpan/primus-qwen3-30b-mfu-phase1 after a successful phase1 commit. Template: {date}-{metric} (e.g. 20260422-278p80) + :latest. Other nodes can then `docker pull` that tag and skip Phase 1 entirely. * feat(primus): enable phase1_publish to ghcr.io/amdpilot-org Switch repository from docker.io/jhinpan to ghcr.io/amdpilot-org so all nodes in the org can pull the verified phase1-baseline image directly. Bump push timeout_s to 9000 (2.5h) to absorb the one-time 42 GB base-layer seed; subsequent pushes only upload the phase1-commit delta (~500 MB - 1 GB) via GHCR cross-repo mount. * docs: align metadata tooling instructions Remove references to missing metadata schema files and point reviewers at the current registry and validation helper tests. * docs(evals): keep metadata tooling PR scoped Remove the inherited Primus eval bundle from the docs-only branch so the PR only updates metadata tooling guidance.
jhinpan
added a commit
that referenced
this pull request
May 2, 2026
* add primus-qwen3-30b-mfu eval instance for issue #1 8x MI355X Qwen3-30B-A3B MoE pretraining MFU optimization. Built on ghcr.io/amdpilot-org/primus-mi355x-ready:v1. Two task.yaml variants for A/B testing the Phase 1 baseline agent: - task.yaml: phase1_baseline: true - task_nophase1.yaml: phase1_baseline: false Both pin the executor to Kimi-K2.6 at 10.235.24.154:30000. * fix(primus): use primus-mi355x-flat:v1 as base + install uv The previous base (ghcr.io/amdpilot-org/primus-mi355x-ready:v1) was a slimmed copy that did NOT include /workspace/primus_train/Primus. Phase 1 spent its full max_turns budget trying to bootstrap from scratch. Switch to primus-mi355x-flat:v1 (locally available, has Primus + Primus-Turbo pre-installed and patched for triton 3.4.0). Also install uv at /root/.local/bin in our Dockerfile so the kimi-cli runtime's source $HOME/.local/bin/env succeeds — without this, executor trials exit 137 immediately after image switch. Tag the new image primus-qwen3-30b-mfu-base:v1 so amdpilot triggers a build the first time it runs. * fix(primus): add gfx950 env vars + runtime IFNAME detection Two cumulative fixes for n08-09 (8x MI355X): 1. Without PYTORCH_ROCM_ARCH=gfx950 (and AITER_ROCM_ARCH, HSA_NO_SCRATCH_RECLAIM, HIP_FORCE_DEV_KERNARG, etc) the torch HIP runtime can't dispatch kernels and benchmarks die immediately with hipErrorInvalidDeviceFunction. These were set in xiao/baizhou's working containers but missing from amdpilot's docker run line. Add to both Dockerfile ENV and task.yaml container.env so they apply via either path. 2. Replace /workspace/detect_interface.sh with a /proc-based detector (the original needed `ip` from iproute2, unavailable in the slim base). bench_mfu.sh now auto-detects GLOO/NCCL socket IFNAME at runtime if bench_config.env doesn't pin one — without this, Megatron's distributed init fails fast (3-5s) before training starts. * feat(primus): add phase1_publish config to task.yaml Enables tag + push of the Phase 1 baseline image to docker.io/jhinpan/primus-qwen3-30b-mfu-phase1 after a successful phase1 commit. Template: {date}-{metric} (e.g. 20260422-278p80) + :latest. Other nodes can then `docker pull` that tag and skip Phase 1 entirely. * feat(primus): enable phase1_publish to ghcr.io/amdpilot-org Switch repository from docker.io/jhinpan to ghcr.io/amdpilot-org so all nodes in the org can pull the verified phase1-baseline image directly. Bump push timeout_s to 9000 (2.5h) to absorb the one-time 42 GB base-layer seed; subsequent pushes only upload the phase1-commit delta (~500 MB - 1 GB) via GHCR cross-repo mount. * feat(flydsl): sglang-kimi-k26-flydsl-mi355x eval instance Adds a hand-authored eval instance that targets the FlyDSL fused-MoE port from AMD's Kimi-K2.5 MI300X optimization blog to Kimi-K2.6 on 4x MI355X, on top of the baseline established by sgl-project/sglang#23381. Contents: - task.yaml : optimize-type task, phase1_baseline=true, frontier_model=true, TP=4, 4xMI355X, FlyDSL env OFF at start so Phase 1 reproduces PR #23381 numbers (DSL2_ROOT + MLIR_PATH + CK_TILE_FLOAT_TO_BFLOAT16 pre-set so import works as soon as agent flips gates during a trial) - Dockerfile : layers FlyDSL + AITER dev/kimi-K2.5 on top of jhinpan/sglang-k26-mi355x:v0.5.10rc0-rocm720-20260420 - bench_flydsl_k26.sh : single bench script emitting one canonical line: output_throughput_tok_s: <v> | concurrency=40 in=10240 out=512 decode_bs1_in8k=<guard> - task_description.md : full phased plan A-G, env-var reference table, BS=1 decode guard (>= 0.98x PR #23381 ~38.05 tok/s), front-loaded supervisor-visible gating block - test_harness.py : GSM8K accuracy gate (lm_eval limit=50 >= 0.90) - metadata.json : links to GH issue amdpilot-org/sglang#2 + PR #23381 - ISSUE.md : condensed body as filed on amdpilot-org/sglang#2 Tracks GitHub issue amdpilot-org/sglang#2. * fix(evals): align flydsl eval metadata and runtime env Keep FlyDSL's runtime MLIR path consistent with the Dockerfile and make new eval metadata consumable by the registry tooling. Mirror the Primus no-phase1 runtime env so the control variant does not hit known HIP temp path failures.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR brings a focused subset of task-plane tooling from
amd-kernel-sft-pipelineintoamdpilot-evalswithout touching the existinginstances/*runtime path.It adds:
scripts/enrich_registry.pyfor PR registry enrichmentscripts/extract_validation_specs.pyfor task-plane validation-spec generationshared/Why this belongs in
amdpilot-evalsThese additions are task-plane concerns:
They do not move batch scheduling, trajectory selection, or SFT formatting into this repo.
Safety / Compatibility
instances/*filesshared/eval_runner.pybehaviorCommits
feat(registry): add PR enrichment toolingfeat(validation): add validation spec extraction toolingdocs(evals): document registry and validation toolingValidation
Ran:
Also ran a smoke test with a temporary one-row PR batch through:
Both commands completed successfully and produced structured output.