Skip to content

compass(docs): pin 16's first CPU-tier census and 08's Revised paragraph to their commits #349

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jgong5 merged 1 commit into from
Sep 23, 2026
Merged

compass(docs): pin 16's first CPU-tier census and 08's Revised paragraph to their commits #349

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 7 additions & 8 deletions atom/compass/design/08_validation_protocol.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,22 +22,23 @@ property than everything else here.**

**Revised 2026-09-20; the original claim was measured false.** This decision used to read
*"187 test files under `tests/`, and ATOM's own `CLAUDE.md` states they need **no GPU** —
AITER and `torch.cuda` are mocked"*. Neither half survived measurement. There are **189**
test files under `tests/`, and **29 of the 159 outside `tests/plugin/` reach the driver**: 28
AITER and `torch.cuda` are mocked"*. Neither half survived measurement. At `fada7424e` there are **189**
test files under `tests/`, and the **29 of the 159 outside `tests/plugin/`** that
`scripts/compass/cpu_gate_exclude.txt` lists **reach the driver**, each run alone: 28
at *collection* time, via `rocminfo` reached on import, so pytest cannot even build the node
list, plus 1 that collects cleanly and then fails at run time on a pinned-host allocation
(`hipHostMalloc failed: 100`). That numerator is non-plugin, so its denominator must be too —
quoting 29 against 189 is the defect `16` L4 fixes and it is not repeated here. **At the
whole-suite denominator the figure is at least 32 of 189**, because `tests/plugin/` is not
purely a packaging problem: collecting it alone at `236abfd9a` in `xiaobizh_n18_cpu` gives
purely a packaging problem: collecting it alone at `236abfd9a`, and again at `fada7424e`, in `xiaobizh_n18_cpu` gives
`153 tests collected, 7 errors in 1.45s`, rc=2, and those 7 decompose as **3 `rocminfo`**

@jgong5 jgong5 Sep 23, 2026

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking. Principle 8 ("Every claim carries its measurement. A number without a source is a defect").

This sentence attaches one wall time, 1.45s, to two runs: one at 236abfd9a and one at fada7424e. I ran the same collection on node 18, in xiaobizh_n18_cpu, on a git archive of fada7424e, with atom.__file__ = /tmp/pr349r1-gates/fada/ATOM/atom/__init__.py. python -m pytest -q --collect-only tests/plugin gave rc=2 and 153 tests collected, 7 errors in 1.41s. The 7 errors split the same way the text says: 3 rocminfo, 1 No module named 'sglang' and 3 cannot import name 'fused_gdn_gating'.

The counts and the rc reproduce. The time is noise, so it cannot be a reading of both commits. Either drop in 1.45s, or say it is the 236abfd9a reading. For example: "gives 153 tests collected, 7 errors, rc=2 (1.45 s at 236abfd9a)".

(`test_gdn_target_verify_batched_equiv.py`, `test_rtpllm_forward_context_semantics.py`,
`test_vllm_deepseek_v4_proxy_state_arena_layout.py`), **1 `ModuleNotFoundError: No module
named 'sglang'`**, and **3 `ImportError: cannot import name 'fused_gdn_gating' from
'atom.model_ops.attention_gdn' (unknown location)`** — a module left half-initialised by the
first three, not a fourth cause. 29 + 3 = 32 is a floor: the other 23 plugin files collect

@jgong5 jgong5 Sep 23, 2026

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking. Principle 7 ("Never report an aggregate without its decomposition").

This PR narrows the numerator to "the 29 … that cpu_gate_exclude.txt lists". The PR body gives the reason: "two tier files also reach rocminfo when run alone". I measured both at fada7424e on node 18, in xiaobizh_n18_cpu, on a git archive with atom.__file__ under /tmp/pr349r1-gates/fada/ATOM. Each was run alone:

  • tests/test_postprocess_width.py: rc=2, 1 error in 0.76s, Get GPU arch from rocminfo failed.
  • tests/test_v4_checkpoint_slot_copy.py: rc=2, 1 error in 0.94s, the same rocminfo error.

So the per-file figure measured at fada7424e is 29 + 2 + 3 = 34 of 189. "At least 32" is still true. But this sentence explains the floor by pointing at the 23 unrun plugin files alone, and leaves out a part that has been measured. Suggestion: "29 + 3 = 32, plus the 2 tier files the table below names, so at least 34; the other 23 plugin files …". Or keep 32 and name the 2 as a known excess.

here and are never run, so nothing is measured about what they would touch.
`tests/plugin/`'s 30 files are dropped whole because the tier targets ATOM and sglang
`tests/plugin/`'s 30 files at `fada7424e` are dropped whole because the tier targets ATOM and sglang
and vllm are in neither image — the measured reason for 27 of them, not for all 30. The
suite as a whole is **not** GPU-free and never was. What is GPU-free is a *tier* of it, and
that tier is genuinely green.
Expand All @@ -46,12 +47,10 @@ that tier is genuinely green.

| Tier | What runs | Measured result |
|---|---|---|
| **CPU tier** — per task, and green is the bar | Every test file outside `tests/plugin/` and `scripts/compass/cpu_gate_exclude.txt`. Driven by `scripts/compass/gate_cpu.sh`; no driver is touched **as a batch**. That is the weaker and correct claim: `test_postprocess_width.py` and `test_v4_checkpoint_slot_copy.py` are both in the tier, and each reaches `rocminfo` when run *alone* (`no tests collected, 1 error in 0.78s`, rc=2, each, measured at `236abfd9a` in `xiaobizh_n18_cpu`); inside the gate they module-skip on the mock an earlier file installs. Driver-freedom here is a property of the batch, not of every file in it. | Green, `GATE_CPU_RC=0`. The counts are not restated here, because they move with every test the tier gains or loses: `gate_cpu.sh` prints them under the `commit:` line of the tree it ran, and `scripts/compass/README.md` says what the tier runs. |
| **CPU tier** — per task, and green is the bar | Every test file outside `tests/plugin/` and `scripts/compass/cpu_gate_exclude.txt`. Driven by `scripts/compass/gate_cpu.sh`; no driver is touched **as a batch**. That is the weaker and correct claim: `test_postprocess_width.py` and `test_v4_checkpoint_slot_copy.py` are both in the tier, and each reaches `rocminfo` when run *alone* (`no tests collected, 1 error in 0.78s`, rc=2, each, measured at `236abfd9a` in `xiaobizh_n18_cpu`); inside the gate they module-skip on the mock an earlier file installs. Driver-freedom here is a property of the batch, not of every file in it. | Green, `GATE_CPU_RC=0`, read at `bdd244c57` in `xiaobizh_n18_cpu`. The counts are not restated here, because they move with every test the tier gains or loses: `gate_cpu.sh` prints them under the `commit:` line of the tree it ran, and `scripts/compass/README.md` says what the tier runs. |
| **GPU superset** — per wave, judged as a **delta**, never as "green" | `tests/ --ignore=tests/plugin` in the GPU container, driven by `scripts/compass/gate_gpu.sh`. | **4779 passed / 5 failed**, 0 errors, 105 skipped, 3 xfailed, 72.6 s, at `fe9ea043c` on node 18 in `xiaobizh_n18`, `HIP_VISIBLE_DEVICES=1`, 2026-09-20 — torch **2.10.0+rocm7.2.4.git3d3aa833**, `torch.version.hip` **7.2.53211**, ROCm **7.2.4**, AITER **v0.1.21.dev0-49-gf4e7c7509**; two runs, byte-identical failing sets. The five are pre-existing and unrelated to Compass, and they are **four ULP comparisons plus one bitwise check**, not "five bf16 ULP failures": four `allclose` cases in `tests/test_fused_compress_ragged.py` off by one bf16 ULP (`max\|diff\| = 0.001953125`, exactly 2⁻⁹, against `atol=rtol=1e-3`), plus `tests/test_dcp_merge_ops.py::test_row_view_matches_output_slicing_bitwise`, a `torch.equal` with **no tolerance at all** — a tolerance bump would not move it. All five node-ids are on file verbatim in `scripts/compass/gpu_gate_known_failures.txt` and compared by name, so "5 failed" can now be checked against "the *same* 5 failed". |

**A total for this tier is a reading of one tree, not a property of the suite.** Each total
belongs to one exclusion list and one `tests/compass`, both of which change, so two totals
compare only when each names its commit. `test_dp_metadata.py`, `test_dp_sync_layout.py`
`test_dp_metadata.py`, `test_dp_sync_layout.py`
and `test_forward_mode.py` were once excluded and were re-measured CPU-green (`16`); prose
saying those three need the driver is stale.

Expand Down
41 changes: 20 additions & 21 deletions atom/compass/design/16_execution_plan.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,33 +114,32 @@ joined two real numbers wrongly: the 28 counts `rocminfo` across the *whole* sui
- **Per task — the CPU tier, re-measured by P0.1 and superseding the counts above.**
`tests/` minus `tests/plugin/` (30 files) minus the 29 driver-dependent files in
`scripts/compass/cpu_gate_exclude.txt`, driven by `scripts/compass/gate_cpu.sh`:
**130 of this tree's 189 test files, 4030 passed, 0 failed, 149 skipped, 3 xfailed,
rc=0** — identical in every run taken on 2026-09-21 in `xiaobizh_n18_cpu` on
hjbog-srdc-18, where the clock read **25.4-31.7 s of pytest inside 31.1-37.8 s of
wall (`time` real)**, which is a measured spread rather than a bound: it tracks
**130 of the 189 test files at `105ca4197`, 4030 passed, 0 failed, 149 skipped, 3 xfailed,
rc=0** in `xiaobizh_n18_cpu` on hjbog-srdc-18. `105ca4197` is P0.1's head and landed as
`4c16792d9`, the same tree. Six runs at two of that branch's commits before its restack, `d737f15e7` and
`7ff80cc4b`, gave the same counts, and the clock read **25.6-30.8 s of pytest inside
31.6-36.9 s of wall (`time` real)**, which is a measured spread rather than a bound: it tracks
what else is on the node. Run against a `git archive` snapshot of the tree, with
`PYTHONPATH` asserted to resolve `atom` under that root and pytest's own exit status
captured before any pipe. The 4030 is **3956 ATOM tests + 74 `tests/compass/` tests**,
captured before any pipe. The 4030 at `105ca4197` is **3956 ATOM tests + 74 `tests/compass/` tests**,
stated as its parts because a single total cannot show which half moved (principle 7). The
file count moves 128 → 130 and the test count 3956 → 4030 because this tree adds
3956 is the same gate at the base, `042aad97d`, which has no `tests/compass/`. The

@jgong5 jgong5 Sep 23, 2026

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking. Principle 8, and ponytail shrink:.

The citation is correct. #6's gate table has 042aad97d in its control column: 3956 passed, 0 failed, 149 skipped, 3 xfailed, rc=0. That control was not the base tree as it stands, though. #6's body says: "Control is fork/feature/atomcompass_new at 042aad97d with only scripts/compass/ copied in". git ls-tree 042aad97d scripts/ has no scripts/compass/, so a reader who checks out 042aad97d finds no gate to run. The words "the same gate" are only literally true.

shrink: L126-129. This sentence and the one that ends at L129 ("the P0.2 readings above are the same suite without them") make one point twice: the 3956 half has no tests/compass/. One sentence can carry the pin and the control's construction. For example: "The 3956 is #6's control, 042aad97d with #6's scripts/compass/ copied in and no tests/compass/, as in the P0.2 readings above; 105ca4197 adds tests/compass/test_cpu_gate_exclude.py and tests/compass/test_gate_gpu_surplus.py, so the file count moves 128 → 130." That is -1 line.

file count moves 128 → 130 and the test count 3956 → 4030 because `105ca4197` adds
`tests/compass/test_cpu_gate_exclude.py` and `tests/compass/test_gate_gpu_surplus.py`; the
P0.2 readings above are the same suite without them. The other totals in circulation are
the same suite under a different exclusion list or a different `tests/compass`, not
discrepancies: **3925** was 32 exclusions with
`tests/compass` at 35 tests, **3956** is the ATOM-only half at 29 exclusions, and **4005**
was this gate at `3afcb4880` with `tests/compass` at 49, and **4022** was it at 66. The
49 → 66 step is mechanical: `tests/compass/test_cpu_gate_exclude.py` parametrises one case
per entry of `gpu_gate_triggers.txt`, and correcting that file's derivation took it from 13
entries to 30. The 66 → 74 step is `tests/compass/test_gate_gpu_surplus.py`, added here.
- **Green is the bar, but green is not "exercised".** Of the 130 files handed, **22 collect
P0.2 readings above are the same suite without them. **4005** was the same gate at
`3afcb4880`, with `tests/compass` at 49 tests. The 49 → 74
growth is two steps: `tests/compass/test_cpu_gate_exclude.py` parametrises one case per
entry of `gpu_gate_triggers.txt`, and correcting that file's derivation took it from 13
entries to 30 (+17); `tests/compass/test_gate_gpu_surplus.py`, added here, is the other 8.
- **Green is the bar, but green is not "exercised".** Of the 130 files handed at `105ca4197`, **22 collect
no test at all**: 16 declare a device dependency, 3 need PyAV, 2 are dead since ATOM #690
split `kv_transfer_engine` into `moriio` (`test_kv_connector_scheduler.py`,
`test_transfer_engine.py`), and `test_prefix_cache_accuracy.py` has **no test function at
all** — it is an `argparse` script that drives a live server on `localhost:8000`. The 149
skips are 68 distinct reasons: 66 skipped tests name a device, 83 do not. That
decomposition was measured per file by P0.2 against an earlier tree, where the same 22 sat
inside a 128-file gate; the gate's own counts above supersede that file count, and the
split of the 22 stands because nothing since has changed which files hold runnable tests.
all** — it is an `argparse` script that drives a live server on `localhost:8000`. That
decomposition was measured per file by P0.2 at `b963c9411`, where the same 22 sat inside a
128-file gate whose 149 skips were 68 distinct reasons: 66 skipped tests name a device, 83
do not. Both carry to `105ca4197`: its `tests/` and code differ from `b963c9411`'s only by
the two `tests/compass/` files above, which collect tests and skip none.
- **Per wave — the GPU superset, re-measured by P0.1.** `tests/ --ignore=tests/plugin` in the
GPU container, driven by `scripts/compass/gate_gpu.sh`, judged as a **delta**, never as
"green". The baseline is **4779 passed / 5 failed**, 0 errors, 105 skipped, 3 xfailed,
Expand Down Expand Up @@ -193,7 +192,7 @@ areas — EPLB, DP metadata, cudagraph bounds, block tables — which nothing re
named two files (`test_dp_metadata`, `test_dp_sync_layout`) the CPU tier does in fact cover.
It is now `scripts/compass/gpu_gate_triggers.txt`, **30 paths** in the tree committed here,
generated by `regen_gpu_gate_triggers.sh` and never hand-edited — the counts in its own
header included (189 test files, 130 CPU tier, 108 of them collecting, 47 candidates, 99
header included (at `fada7424e`: 189 test files, 130 CPU tier, 108 of them collecting, 47 candidates, 99
covered). The rule it applies: an `atom` module named by an excluded test is a blind spot
**unless a CPU-tier test that actually runs names it too**. A module both tiers import is
covered when the CPU gate runs, so triggering on it would make the gate cry wolf. A trailing
Expand Down