Conversation
3 tasks
Oseltamivir
added a commit
that referenced
this pull request
Jun 23, 2026
Add summarize.py (compact NCCL/DeepEP results table, printed at end of every job) and make it the result gate. Fix review findings: benchmark failures/skipped-deepep now fail the job instead of reporting green (#1); DeepEP nodes from SLURM_NNODES not world_size//8 (#3); apply Buffer.set_num_sms so num_comm_sms is real (#8); nccl-tests -c 1 with a missing check footer is now invalid (#7); use context managers for file reads (#4,#5); launchers export COLLECTIVEX_IMAGE/_DIGEST for provenance (#9); trim workflow_dispatch sku options to launcher-backed pools (#2). Artifact-path finding (#6) already fixed via cx_collect_results.
Oseltamivir
added a commit
that referenced
this pull request
Aug 7, 2026
…on a bad consume mode queries.md #13 and #9. #13: flashinfer-ep declares maturity "production" -- true of the backend, since vLLM exposes --all2all-backend flashinfer_nvlink_one_sided -- while its FP8 dispatch is a precision no engine can select (vLLM takes nvfp4/mxfp8/bf16 on that transport). The only machine-readable marker was an "-offpath" suffix buried inside the dispatch_dtype STRING, which nothing groups or filters on, and summarize.py prints no dispatch_dtype column at all. So in the view most people read, a transport-only measurement was indistinguishable from a deployable result. maturity cannot carry this: it is per-backend, and the whole point is that one backend is deployed at BF16 and off-path at FP8. So path_status is per-case, defaults to "deployed", and flashinfer-ep sets "offpath" under FP8. The harness emits it beside maturity and summarize marks the precision column ("fp8 [offpath]"). Not excluded from the matrix, which was the other option on the table: the row exists precisely to compare FP8 wire cost against deepep-v2/uccl-ep at matching bytes and block size, and dropping it would throw that away to fix a labelling problem. The "-offpath" dispatch_dtype suffix stays for durable-store row continuity and can retire at the next version bump. #9: CX_FP8_CONSUME accepted any string and silently meant "native", so a typo measured the opposite model from the one requested AND tagged the artifact with the typo. It now fails closed. Because the value is read at class-body evaluation this raises at import, before any measurement runs, which is the right moment for it. The rest of #9 -- relocating the dequant hatch to a separate diagnostic script -- is declined. The review conditioned that on the dequant cost being derivable, and it is: dequant roundtrip is roundtrip + stage to within +2.4%/-0.1%, verified in-tree. But the hatch is what re-verifies that bound after a kernel change (302.0 vs 302.5us in run 30177021271), a separate script would need the identical staged-inline branch, and BF16 requires that branch regardless. Moving it deletes the instrument, not the mechanism, and would make the derivability claim unfalsifiable. Also repaired here: a comment I orphaned two commits ago. Deleting combine_input_attr left its comment with no declaration beneath it, so it ran straight into the fp8_consume block and read as one paragraph. It is now marked as what it is -- a handle contract, not an attribute of the class. Tests: the unrecognised-mode guard (verified failing when removed), the deployed default, and two render tests -- an off-path row is marked, a deployed row and a row predating the field are not. 168 tests green.
Oseltamivir
added a commit
that referenced
this pull request
Aug 7, 2026
…alize the FP8 chain divergence The FP8 chained output disagrees with a drained pair by 1000x-2966x tolerance on every SKU while BF16 is bit-identical (probe 31180411148). Two candidate causes remain and they have different fixes: the free-running regime genuinely changes FP8 combine output, or the chain's hoisted staging structure does. CX_FP8_CONSUME=dequant separates them exactly. It is already a documented verification hatch, and it makes stage_excluded_from_roundtrip False, so the chain stages per pair and never hoists -- the one variable that follows the FP8/BF16 split perfectly. Error collapsing to ~0 implicates the hoist; error persisting implicates the regime. It was reachable only by hand before; the container srun exports the submitting environment, so a job-level variable reaches the ranks. The input is a two-option choice, defaulted off, and its description says it changes what is measured -- it is a diagnostic, not a sweep axis. One trap closed while adding it, of the exact class this repo keeps meeting. Passing the input through empty would set CX_FP8_CONSUME="" on every normal run. That is NOT unset, so os.environ.get's default never applies, the harness's fail-closed check (added earlier today for queries.md #9) rejects it, and every leg would die at import before any measurement -- a silent fleet-wide kill introduced by a diagnostic nobody was using. The workflow defaults to 'native' instead, and a test pins that: it fails if the fallback is dropped or an unrecognised literal appears on that line. 173 tests green.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merging to debug workflow scheduler testing workflow