[DeepSeek-V4.1] Optimize DSpark verify and MoE kernels on Blackwell

[BBuf]

Motivation

DSpark target verification and draft steps miss several decode fast paths. MoE routing, input quantization, and finalize also leave work on the critical path. This carries the compatible kernel changes onto dsv4.1.

Modifications

• Use mHC statistics overlap in target verify and all three draft stages, and widen the validated verify shapes.
• Write WO-A output directly in token-major layout and fuse index candidate masking.
• Produce padded, packed router outputs and run routed-input MXFP8 quantization on a separate stream.
• Fuse deferred MoE finalize, shared-expert addition, and custom push all-reduce for supported small TP4 batches. Preserve the BF16 rounding points and synchronize phase-counter reuse.
• Prefer custom all-reduce for the supported single-node Blackwell V4.1 configuration. Other configurations retain their defaults and fallbacks. No new environment flags.

The updated public top-k implementation is retained. This PR contains production changes only; standalone validation harnesses are kept outside the source diff.

Co-authored with @DarkSharpness (Ziyi Xu), who contributed the MoE router and input prequantization, finalize/shared-add/all-reduce fusion, and custom all-reduce backend selection.

Speed Tests

Base: public dsv4.1 at 7bdebdab7db4befb71c64ae0d6f0eb37fe7d8402. Candidate: 1b742acd2a49ebd7acd017032875552099d12391.

4x B300 SXM6 AC, TP4/EP4, PyTorch 2.13.0+cu130, FlashInfer 0.6.18, Triton 3.7.1, sglang-kernel==0.4.6.post1, sgl-deep-gemm==0.1.7, CUTLASS DSL 4.6.2, nvcc 13.0.88, driver 580.126.20. DSpark block size 5 with real acceptance. Both arms use the same checkpoint, package environment, fixed KV pool sizes, and hashed input IDs.

Two independent server launches per arm, with one warmup and three measured repetitions each. Order: baseline A, candidate A, candidate B, baseline B. Medians below use all six measured repetitions.

Workload	Baseline	Candidate	Change
BS1, input 4096/output 1024, post-first-event output tok/s	569.88	764.29	+34.1%
BS64, input 4096/output 2048, stable decode output tok/s	6,555.95	13,472.51	+105.5%
BS64, same workload, full batch output tok/s including prefill	4,114.78	6,434.55	+56.4%

Median BS1 acceptance length: 5.658 → 5.818; BS64: 5.407 → 5.479. No simulated acceptance. BS1 acceptance/TPS cycle proxy: 9.897 → 7.603 ms. This proxy is separate from the GPU trace timings below.

BS64 decode samples require consecutive log intervals with exactly 64 running requests and CUDA graphs active, excluding intervals across prefill. The full-batch metric includes prefill and the draining tail. BS1 excludes the first SSE event and counts only the remaining output tokens. These metrics should not be compared interchangeably.

Per-server measurements

Run	BS1 measured tok/s	BS64 decode tok/s	BS64 full-batch tok/s
base-a	577.07, 570.83, 556.71	6491.16, 6577.62, 6534.28	4229.08, 3910.66, 3969.39
candidate-a	766.36, 763.00, 764.22	13771.38, 13120.99, 13846.63	6634.80, 6262.78, 6643.85
candidate-b	754.66, 764.37, 766.16	12872.41, 13572.70, 13372.32	5847.66, 6405.10, 6464.00
base-b	558.82, 585.27, 568.93	6501.85, 6849.70, 6691.80	4000.47, 4540.29, 4276.04

Accuracy Tests

Test	Baseline	Candidate	Budget truncations (base/candidate)
GSM8K, serial 100	97/100 (97.00%)	98/100 (98.00%)	0/0
GSM8K, five-shot held-out 1314, concurrency 64	1268/1314 (96.50%)	1266/1314 (96.35%)	0/0
AIME 2026, serial 30, temperature 0	25/30 (83.33%)	25/30 (83.33%)	4/5
AIME 2026, 30 questions × 16 samples, concurrency 64, temperature 1	446/480 (92.92%)	451/480 (93.96%)	13/18

All requested samples completed with zero request errors. GSM8K has no empty or truncated responses. AIME budget truncations are counted in the score, not dropped. The repeated AIME score is correct samples / 480, not pass@16.

GSM8K has 14 correct-to-incorrect and 12 incorrect-to-correct changes under the unchanged prompt and scorer. Generated text is not bitwise identical. These results and the kernel checks do not establish exact model equivalence for every input.

GSM8K uses the first five test rows as demonstrations and excludes them from evaluation. AIME uses sgl-eval==0.1.0, the bundled MathArena prompt and NeMo-Skills revision 645cf567ff08c0ae9cc3fc8e1edbb975b3067816, thinking mode, top-p 0.95, and max tokens 65,536. Serial AIME uses seed 0; repeated AIME leaves the request seed unset.

Kernel validation and profiling

• 60 kernel/integration tests and 224 subtests passed.
• Four-GPU all-reduce/finalize suites: 99 + 61 cases passed, including graph replay, mixed token counts, independent unfused/FP32 references, and delayed-reader phase-counter probes.
• Coverage includes router packing/padding, routed-input quantization events, WO-A layout, mHC target/draft overlap, source-stream dependency guards, candidate masking, and verify compressor state/cache writes.
• Repository pre-commit hooks and public lint passed.

GPU-only TP0 traces, BS1 with 4096 input tokens, 20 target graph replays and 20 draft graph replays per arm. Each draft graph contains the three draft stages. Trace timings are reported separately from unprofiled serving throughput.

GPU timeline metric	Baseline	Candidate
Target verify graph median	9.049 ms	6.927 ms
Draft graph median	0.833 ms	0.719 ms
Median target-start to next target-start	10.050 ms	7.810 ms
Target kernels per graph	2209	1965
Draft kernels per graph	196	181

The following launch counts cover the complete 20-cycle trace:

Kernel path	Baseline launches	Candidate launches
Separate router padding mask	860	0
Separate router ID packing	860	0
Standalone MoE finalize	860	0
Fused MoE finalize + shared add + all-reduce	0	860
FlashInfer oneshot all-reduce	1760	0
Custom push all-reduce	40	940

mHC statistics overlap with other kernel families on different CUDA streams increases from 0.18% to 93.50% of its kernel duration. This measures time overlap in the trace; it is not an occupancy metric or a guarantee that the overlapped work has zero cost. Graph spans above measure the net effect.

Server command (both arms)

# Run from the corresponding SGLang checkout.
MODEL_PATH=/path/to/DeepSeek-V4.1
CUDA_VISIBLE_DEVICES=4,5,6,7 PYTHONPATH="$PWD/python" MAX_JOBS=16 \
python -m sglang.launch_server \
  --model-path "$MODEL_PATH" \
  --tp 4 --ep-size 4 --trust-remote-code \
  --mem-fraction-static 0.80 --max-total-tokens 33554432 \
  --chunked-prefill-size 4096 \
  --cuda-graph-bs-decode 1 2 4 8 16 32 64 \
  --max-running-requests 128 \
  --speculative-algorithm DSPARK --speculative-dspark-block-size 5 \
  --skip-server-warmup --reasoning-parser deepseek-v41 \
  --random-seed 42 --decode-log-interval 10 \
  --host 127.0.0.1 --port 30021

After the server is ready, call POST /freeze_gc. Flush the request cache before each measured repetition. Both arms use the identical launch command; the runtime selects the compatible kernel paths.

Checkouts used for the comparison

From a clone of sgl-project/sglang:

git fetch origin dsv4.1
git fetch origin pull/38879/head:pr-38879-validation
git worktree add ../sglang-baseline-38879 7bdebdab7db4befb71c64ae0d6f0eb37fe7d8402
git worktree add ../sglang-candidate-38879 1b742acd2a49ebd7acd017032875552099d12391

Run each arm from its own worktree with the same installed dependency versions. Start with inherited SGLANG_* overrides removed; the measured controllers clear those variables. The same checkpoint directory is used for both arms.

Benchmark and GSM8K client commands and harnesses

Save the three Python blocks below beside gsm8k-test.jsonl. Set MODEL_PATH to the same checkpoint used by the server and SERVER_LOG to its log file. Keep the serving checkout first on PYTHONPATH; the preinstalled editable SGLang package is not the source under test. The benchmark scripts differ from the measured copies only in making the tokenizer path configurable.

export PYTHONPATH="$PWD/python"
python -c "import sglang; print(sglang.__file__)"
export MODEL_PATH=/path/to/DeepSeek-V4.1
SERVER_LOG=/path/to/server.log
mkdir -p results/bs1 results/bs64 results/gsm-serial100 results/gsm-full
curl -f -X POST http://127.0.0.1:30021/freeze_gc
python bench_bs1.py --out results/bs1 --server-log "$SERVER_LOG"
python bench_bs64.py --out results/bs64 --server-log "$SERVER_LOG"
curl -f -X POST 'http://127.0.0.1:30021/flush_cache?timeout=30'
python gsm_eval.py --out results/gsm-serial100 --count 100 --threads 1
curl -f -X POST 'http://127.0.0.1:30021/flush_cache?timeout=30'
python gsm_eval.py --out results/gsm-full --count 1314 --threads 64

GSM8K uses the public openai/grade-school-math grade_school_math/data/test.jsonl file, SHA256 3730d312f6e3440559ace48831e51066acaca737f6eabec99bccb9e4b3c39d14. The first five rows are demonstrations and are excluded from the 1,314 evaluated questions.

bench_bs1.py

"""Fixed-input, real-acceptance BS1 measurements on the owned server only."""
import argparse
import os
import hashlib
import json
import re
import time
from pathlib import Path

import requests
from tokenizers import Tokenizer


def save(path, value):
    path.write_text(json.dumps(value, indent=2, ensure_ascii=False))


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('--out', type=Path, required=True)
    parser.add_argument('--server-log', type=Path, required=True)
    parser.add_argument('--url', default='http://127.0.0.1:30021')
    parser.add_argument('--repeat', type=int, default=3)
    parser.add_argument('--profile', action='store_true')
    args = parser.parse_args()
    tok = Tokenizer.from_file(os.path.join(os.environ['MODEL_PATH'], 'tokenizer.json'))
    prefix = tok.encode('Read these notes and summarize them.\n', add_special_tokens=False).ids
    unit = tok.encode('The observatory records the temperature, wind, and rainfall each day. Researchers compare the measurements across seasons.\n', add_special_tokens=False).ids
    suffix = tok.encode('\nWrite a detailed summary of the notes above.\n', add_special_tokens=False).ids

    def post(route, body=None, timeout=1800):
        r = requests.post(args.url+route, json=body or {}, timeout=timeout)
        r.raise_for_status()
        return r

    def warm(ids, rate):
        # The final SSE response can precede scheduler request cleanup.
        # Use the server's bounded idle wait, outside the measured interval.
        post('/flush_cache?timeout=30', timeout=60)
        if rate:
            r = post('/generate', {'input_ids':ids[:int(len(ids)*rate)], 'sampling_params':{'temperature':0, 'max_new_tokens':1, 'ignore_eos':True}})
            assert r.json()['meta_info']['completion_tokens']==1

    results=[]
    for length, out_len, rate in ((4096,1024,0.),):
        n = length-len(prefix)-len(suffix)
        ids = prefix + (unit*((n+len(unit)-1)//len(unit)))[:n] + suffix
        assert len(ids)==length
        save(args.out/f'input-{length}.json',ids)
        input_hash=hashlib.sha256(json.dumps(ids).encode()).hexdigest()
        for rep in range(args.repeat+1):
            warm(ids,rate)
            log_offset=args.server_log.stat().st_size
            body={'input_ids':ids,'sampling_params':{'temperature':0,'max_new_tokens':out_len,'ignore_eos':True,'stream_interval':1},'stream':True}
            start=time.perf_counter();first=None;first_count=None;last=None
            with requests.post(args.url+'/generate',json=body,stream=True,timeout=(30,1800)) as r:
                r.raise_for_status()
                for line in r.iter_lines():
                    if not line.startswith(b'data: '):continue
                    raw=line[6:]
                    if raw==b'[DONE]':break
                    last=json.loads(raw)
                    if 'error' in last:raise RuntimeError(last)
                    count=last.get('meta_info',{}).get('completion_tokens',0)
                    if first is None and count>0:first=time.perf_counter();first_count=count
            end=time.perf_counter()
            assert last is not None and first is not None,last
            meta=last['meta_info']
            assert meta['completion_tokens']==out_len and meta['prompt_tokens']==length,meta
            # Required evidence of actual speculative execution and acceptance.
            assert meta.get('spec_verify_ct',0)>0 and meta.get('spec_accept_length',0)>0,meta
            with args.server_log.open() as f:f.seek(log_offset);log=f.read()
            samples=[float(v) for v in re.findall(r'gen throughput \(token/s\):\s*([0-9.]+)',log)]
            result=dict(input_tokens=length,output_tokens=out_len,requested_cache_hit_rate=rate,repeat=rep,warmup=(rep==0),input_sha256=input_hash,elapsed_s=end-start,ttft_s=first-start,first_stream_completion_tokens=first_count,post_first_event_tps=(out_len-first_count)/(end-first),full_request_tps=out_len/(end-start),decode_log_samples=samples,response=last)
            results.append(result);save(args.out/'performance.json',results)
            print('PERF',length,rep,result['post_first_event_tps'],'accept',meta['spec_accept_length'],'cached',meta.get('cached_tokens'),flush=True)

        if args.profile:
            for activities in (['GPU'],):
                if length==131072 and activities==['CPU','GPU']:continue
                warm(ids,rate)
                tag=f'{length}-'+('-'.join(activities)).lower()
                folder=args.out/('profile-'+tag);folder.mkdir()
                request=dict(output_dir=str(folder),num_steps=20,activities=activities,profile_by_stage=True,profile_stages=['decode'],with_stack=(activities==['CPU','GPU']),record_shapes=(activities==['CPU','GPU']),profile_id=args.out.name+'-'+tag)
                save(folder/'request.json',request)
                post('/start_profile',request,timeout=60)
                response=post('/generate',{'input_ids':ids,'sampling_params':{'temperature':0,'max_new_tokens':200,'ignore_eos':True}}).json()
                save(folder/'response.json',response)
                # At most six tokens per verify step: 200 output tokens
                # always cover the 20 requested decode steps. Export is automatic.
                traces=list(folder.rglob('*.trace.json*'))
                assert len(traces)>=4,(tag,'Missing TP traces',traces)
                save(folder/'trace-files.json',[str(p) for p in traces])
                print('PROFILE',tag,len(traces),flush=True)


if __name__=='__main__':
    main()

bench_bs64.py

"""BS64 serving regression measurement; raw requests and log intervals retained."""
import argparse
import os
from concurrent.futures import ThreadPoolExecutor
import hashlib
import json
from pathlib import Path
import re
import statistics
import threading
import time
import urllib.request

from tokenizers import Tokenizer


def save(path, value):
    path.write_text(json.dumps(value, indent=2, ensure_ascii=False) + '\n')


def post(url, route, body=None):
    request = urllib.request.Request(url + route, data=json.dumps(body or {}).encode(),
                                     headers={'Content-Type': 'application/json'})
    with urllib.request.urlopen(request, timeout=1800) as response:
        if route.startswith(('/flush_cache', '/start_profile', '/stop_profile')):
            return response.read().decode()
        return json.load(response)


def decode_samples(log):
    rows = []
    previous_bs = None
    for line in log.replace('\r', '\n').splitlines():
        if 'Prefill batch' in line:
            previous_bs = None
        if 'Decode batch' not in line:
            continue
        bs = re.search(r'#running-req: (\d+)', line)
        tps = re.search(r'gen throughput \(token/s\): ([0-9.]+)', line)
        al = re.search(r'accept len: ([0-9.]+)', line)
        graph = re.search(r'(?:CUDA graph|cuda graph|cuda_graph|full graph): (True|False)', line)
        if not bs or not tps:
            continue
        batch = int(bs.group(1))
        rows.append(dict(batch_size=batch, tps=float(tps.group(1)),
                         accept_length=float(al.group(1)) if al else None,
                         graph=graph.group(1) == 'True' if graph else None,
                         eligible=(batch == 64 and previous_bs == 64), line=line))
        previous_bs = batch
    return rows


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('--out', type=Path, required=True)
    parser.add_argument('--server-log', type=Path, required=True)
    parser.add_argument('--url', default='http://127.0.0.1:30021')
    parser.add_argument('--no-spec', action='store_true')
    args = parser.parse_args()
    tok = Tokenizer.from_file(os.path.join(os.environ['MODEL_PATH'], 'tokenizer.json'))
    unit = tok.encode('The observatory records the temperature, wind, and rainfall each day. Researchers compare the measurements across seasons.\n', add_special_tokens=False).ids
    suffix = tok.encode('\nWrite a detailed summary of the notes above.\n', add_special_tokens=False).ids
    inputs = []
    for i in range(64):
        prefix = tok.encode(f'Notebook {i:03d}. Read these notes and summarize them.\n', add_special_tokens=False).ids
        n = 4096 - len(prefix) - len(suffix)
        inputs.append(prefix + (unit * ((n + len(unit) - 1) // len(unit)))[:n] + suffix)
    assert len({tuple(ids) for ids in inputs}) == 64
    assert all(len(ids) == 4096 for ids in inputs)
    save(args.out/'inputs.json', inputs)
    input_hash = hashlib.sha256(json.dumps(inputs).encode()).hexdigest()
    waves = []
    for rep in range(4):
        warmup = rep == 0
        output_length = 256 if warmup else 2048
        post(args.url, '/flush_cache?timeout=30')
        offset = args.server_log.stat().st_size
        barrier = threading.Barrier(65)

        def request_one(i):
            body = {'input_ids': inputs[i], 'sampling_params': {
                'temperature': 0, 'max_new_tokens': output_length, 'ignore_eos': True}}
            barrier.wait()
            begin = time.perf_counter()
            response = post(args.url, '/generate', body)
            elapsed = time.perf_counter() - begin
            meta = response['meta_info']
            assert meta['completion_tokens'] == output_length, meta
            assert meta['prompt_tokens'] == 4096, meta
            if not args.no_spec:
                assert meta.get('spec_verify_ct', 0) > 0, meta
                assert meta.get('spec_accept_length', 0) > 0, meta
            return dict(index=i, elapsed_s=elapsed, response=response)

        with ThreadPoolExecutor(max_workers=64) as executor:
            futures = [executor.submit(request_one, i) for i in range(64)]
            start = time.perf_counter()
            barrier.wait()
            responses = [future.result() for future in futures]
            elapsed = time.perf_counter() - start
        save(args.out/f'wave-{rep}-responses.json', responses)
        # Drain scheduler request cleanup before the next cache flush.
        time.sleep(1)
        with args.server_log.open('rb') as log:
            log.seek(offset)
            text = log.read().decode(errors='replace')
        (args.out/f'wave-{rep}-server.log').write_text(text)
        rows = decode_samples(text)
        eligible = [row for row in rows if row['eligible']]
        if not warmup:
            assert len(eligible) >= 3, ('Insufficient actual BS64 intervals', rows)
            assert all(row['graph'] is True for row in eligible), eligible[:3]
        completed = sum(r['response']['meta_info']['completion_tokens'] for r in responses)
        verifies = sum(r['response']['meta_info'].get('spec_verify_ct', 0) for r in responses)
        wave = dict(repeat=rep, warmup=warmup, requests=64, input_tokens=4096,
                    output_tokens_per_request=output_length, input_sha256=input_hash,
                    elapsed_s=elapsed, aggregate_full_request_tps=completed/elapsed,
                    weighted_accept_length=completed/verifies if verifies else None,
                    bs64_decode_median_tps=statistics.median(row['tps'] for row in eligible) if eligible else None,
                    bs64_decode_intervals=len(eligible), decode_intervals=rows)
        waves.append(wave)
        save(args.out/'performance.json', waves)
        print('WAVE', rep, 'warmup', warmup, 'actual BS64 decode', wave['bs64_decode_median_tps'],
              'intervals', len(eligible), 'full-request', wave['aggregate_full_request_tps'],
              'AL', wave['weighted_accept_length'], flush=True)


if __name__ == '__main__':
    main()

gsm_eval.py

"""Pinned legacy five-shot GSM8K prompts/scorer, with complete response/error records."""
import argparse,hashlib,json,time
from pathlib import Path
from concurrent.futures import ThreadPoolExecutor,as_completed
import requests
from sglang.test.simple_eval_mixed_prefix_gsm8k import get_few_shot_examples,get_one_example,get_answer_value

def main():
 p=argparse.ArgumentParser();p.add_argument('--out',type=Path,required=True);p.add_argument('--threads',type=int,required=True);p.add_argument('--count',type=int,default=1314);p.add_argument('--url',default='http://127.0.0.1:30021');a=p.parse_args()
 a.out.mkdir(exist_ok=True);data=Path(__file__).with_name('gsm8k-test.jsonl');rows=[json.loads(l) for l in data.read_text().splitlines()];assert len(rows)==1319
 prefix=get_few_shot_examples(rows,5);ids=list(range(5,min(1319,5+a.count)))
 models=requests.get(a.url+'/v1/models',timeout=30);models.raise_for_status();model=models.json()['data'][0]['id']
 def run(i):
  prompt=prefix+get_one_example(rows,i,False);start=time.perf_counter()
  rec=dict(index=i,prompt=prompt,prompt_sha256=hashlib.sha256(prompt.encode()).hexdigest(),expected=get_answer_value(rows[i]['answer']))
  try:
   r=requests.post(a.url+'/v1/chat/completions',json=dict(model=model,messages=[dict(role='user',content=prompt)],temperature=0,top_p=1,max_tokens=4096,seed=0,return_meta_info=True),timeout=(30,1800));r.raise_for_status();j=r.json();rec['response']=j
   ch=j['choices'][0];content=ch['message'].get('content') or '';rec.update(answer=get_answer_value(content),correct=get_answer_value(content)==rec['expected'],truncated=ch['finish_reason']=='length',empty=not bool(content.strip()))
  except Exception as e:rec.update(error=repr(e),correct=False,truncated=False,empty=True)
  rec['elapsed_s']=time.perf_counter()-start;return rec
 results=[];begin=time.perf_counter()
 with (a.out/'samples.jsonl').open('w') as f,ThreadPoolExecutor(max_workers=a.threads) as pool:
  futures=[pool.submit(run,i) for i in ids]
  for future in as_completed(futures):
   r=future.result();results.append(r);f.write(json.dumps(r,ensure_ascii=False)+'\n');f.flush()
   if len(results)%25==0 or len(results)==len(ids):print('GSM',len(results),'/',len(ids),'correct',sum(x['correct'] for x in results),'errors',sum('error' in x for x in results),flush=True)
 summary=dict(count=len(results),correct=sum(r['correct'] for r in results),score=sum(r['correct'] for r in results)/len(results),errors=sum('error' in r for r in results),truncated=sum(r['truncated'] for r in results),empty=sum(r['empty'] for r in results),threads=a.threads,elapsed_s=time.perf_counter()-begin,dataset_sha256=hashlib.sha256(data.read_bytes()).hexdigest(),num_shots=5,held_out_indices=ids,temperature=0,top_p=1,seed=0,max_tokens=4096)
 (a.out/'summary.json').write_text(json.dumps(summary,indent=2));print(json.dumps(summary),flush=True)
 assert summary['errors']==0,summary
 assert summary['score']>=.9,summary
if __name__=='__main__':main()

AIME 2026 commands

The run uses sgl-eval==0.1.0 with its bundled NeMo-Skills data/prompt (645cf567ff08c0ae9cc3fc8e1edbb975b3067816). The evaluator is installed outside the serving environment. Both arms use the same files and sampling settings.

# Run with the serving checkout on PYTHONPATH.
python -m pip install --target /tmp/sgl-eval-pinned --no-deps \
  sgl-eval==0.1.0 math-verify==0.9.0 latex2sympy2_extended==1.11.0 \
  antlr4-python3-runtime==4.9.3 editdistance==0.8.1
export PYTHONPATH="/tmp/sgl-eval-pinned:$PWD/python"
EVAL_DATA=/tmp/sgl-eval-pinned/sgl_eval/_vendored/nemo_skills/dataset/aime26/test.txt
EVAL_PROMPT=/tmp/sgl-eval-pinned/sgl_eval/_vendored/nemo_skills/prompts/matharena-aime.yaml
python -m sgl_eval.cli run aime26 \
  --base-url http://127.0.0.1:30021/v1 --model "$MODEL_PATH" \
  --from-dataset "$EVAL_DATA" --prompt "$EVAL_PROMPT" \
  --num-threads 1 --n-repeats 1 --thinking --reasoning-effort max \
  --temperature 0 --top-p 0.95 --max-tokens 65536 --seed 0 \
  --out-dir results/aime-serial30
python -m sgl_eval.cli run aime26 \
  --base-url http://127.0.0.1:30021/v1 --model "$MODEL_PATH" \
  --from-dataset "$EVAL_DATA" --prompt "$EVAL_PROMPT" \
  --num-threads 64 --n-repeats 16 --thinking --reasoning-effort max \
  --temperature 1 --top-p 0.95 --max-tokens 65536 \
  --out-dir results/aime-repeat16

The repeated lane leaves the request seed unset. Accuracy is the fraction of correct samples, not pass@16. Errors and token-budget truncations are counted separately.

CI

GPU CI is blocked before tests by the global requirement to include main commit 3700c4ee26a1, which also rejects this dsv4.1-based PR (job log). The B300 results above were collected directly against the stated base and candidate.

---

CI States

Latest PR Test (Base): ❌ Run #34478244464
Latest PR Test (Extra): ❌ Run #34478243976
Latest PR Test (AMD ROCm 10): ❌ Run #34478244607