[taneem-ibrahim]

Overview

Clef builds and copies several small metadata tensors for every request. This uses NumPy for the CPU work and reduces the pooler’s host-to-device copies from nine to three per request, leaving the decision-head math unchanged.

Validation

Model: Cloudflare/clef-flash, BF16, MRV2, native /v1/systemone. All pairs below are main → PR; C means concurrent requests, and input tokens are rounded means.

B300: measured roughly 3–5% higher throughput on the tested short, single-question workloads; longer inputs were effectively flat. The controlled study and profiling details below qualify that result.

C	Input tokens / questions	Requests/s	Throughput change	p50 latency (ms)	p99 latency (ms)	Peak device memory (MiB)
8	209 / 1	114.58 → 118.30	+3.25%	69.61 → 67.51	72.75 → 69.59	83,370 → 83,370
16	209 / 1	163.61 → 168.18	+2.79%	97.49 → 94.94	102.26 → 98.57	83,778 → 83,778
32	209 / 1	185.17 → 191.25	+3.28%	172.24 → 167.51	190.32 → 184.20	84,562 → 84,562
64	209 / 1	200.12 → 205.82	+2.85%	318.54 → 310.17	334.01 → 315.42	85,458 → 85,094
16	986 / 8	79.85 → 79.94	+0.11%	200.07 → 199.88	205.75 → 204.44	86,502 → 86,486
8	4946 / 32	17.04 → 17.05	+0.04%	391.45 → 391.66	587.15 → 587.05	87,656 → 87,640

The reference test passed on both versions, 306 CPU + 306 GPU metadata cases matched across four streams, and all 1,024 question results matched main versus PR within serial/C16 modes. Another 112 requests passed with VLLM_GPU_SYNC_CHECK=error on B300. Timed requests for the displayed cases: 93,272 on B300, zero errors. These checks establish output parity, not labeled task accuracy.

Controlled input-length study (B300)

This separate C16 study fixes three options per question and verifies the total input-token count on every response. Each version used two fresh servers in main → PR → PR → main order, with two 30-second windows per case/server after 10 seconds of warmup. The last two servers reversed case order. Profiling was disabled.

Input tokens / questions	Requests/s, main → PR	Change	p50 / p99 latency (ms), main → PR
256 / 1	155.27 → 162.48	+4.64%	102.78 / 107.39 → 97.72 / 104.98
1,024 / 1	79.65 → 79.80	+0.19%	200.65 / 208.20 → 200.14 / 208.16
4,096 / 1	22.75 → 22.78	+0.11%	703.03 / 756.60 → 702.19 / 743.61
4,096 / 8	22.30 → 22.36	+0.29%	717.10 / 775.72 → 715.19 / 775.09
4,096 / 32	21.33 → 21.39	+0.24%	749.04 / 776.80 → 747.32 / 761.79

With one question, the serving gain becomes effectively flat as input length increases. At 4,096 tokens, changing question count from 1 to 8 to 32 also leaves serving throughput effectively flat; the isolated B16 pooler still improves by 3.0%, 4.6% and 8.5%, respectively. Those pooler timings use real head weights with synthetic hidden states and lexical embeddings.

The short-input mean is variable: main windows ranged from 154.18–155.95 requests/s and PR windows from 156.86–172.07; +4.64% is an observed average, not a guaranteed gain. There were 73,659 timed requests with zero errors and 344 exactly matching question results in separate parity checks. Peak sampled whole-device/process memory was 93,462/92,510 MiB on both versions; these include KV cache and retained allocations.

This study uses --max-num-seqs 16 --max-num-batched-tokens 65536, allowing C16 × 4,096 tokens without the original table’s token-budget limit. The other model, dtype, affinity and prefix-cache settings are unchanged. More questions replace state padding and increase total options; this separates total length from question count, not question overhead from option overhead. Both arms use the same existing source/wheel environment with the exact baseline/PR pooler methods applied, not separate current-main rebuilds.

Focused CPU/GPU profile (B300)

Nsight Systems captured warmed-up Q1/C16 requests at 256 and 1,024 total tokens, with temporary NVTX markers around schema parsing, indices, transfers, head execution and frontend encoding. Four fresh servers supplied one accepted three-second capture per variant/length: 1,207 requests, zero errors. Capture head counts match client completions; eight separate paired requests matched exactly.

Input tokens	Profiled _indices host duration/request, main → PR	Pooler H2D copies/request	Head kernels/request
256	0.366 → 0.197 ms	9 → 3	179 → 179
1,024	0.514 → 0.212 ms	9 → 3	185 → 185

Bytes transferred remain 779/request for these inputs. GPU activity overlaps about 31–37% of the pooler host span at 256 tokens and 77–82% at 1,024 tokens. This supports CPU/GPU overlap as an explanation, but profiling perturbs timing and batch formation, so it does not attribute the full unprofiled throughput gain. These are instrumented host spans, not CPU time saved on the critical path.

For short inputs, the head still occupies about 88% of the PR pooler’s host span, including kernel dispatch; frontend encoding is about 0.60 ms/request on both versions. Other frontend costs are not conclusively attributed. Any head-batching experiment belongs in a separate PR.

Capture configuration: CUDA/NVTX/OS-runtime tracing, 100 Hz Python sampling, CUDA graph tracing at graph granularity, 15 seconds of warmup, then a three-second capture started/stopped through the native profiling endpoints. No profiler timings are used in the serving tables. Nsight lost its agent on later captures; only complete first captures from verified fresh servers were retained and checked against client counts.

Methodology, reproduction and test commands

Reproducer: start a fresh server for each arm, selecting its source tree on PYTHONPATH. B300 used GPU 0, server CPUs 0–15, client CPUs 16–19 and --gpu-memory-utilization 0.3:

CUDA_VISIBLE_DEVICES=0 OMP_NUM_THREADS=1 VLLM_USE_V2_MODEL_RUNNER=1 \
  PYTHONPATH=/path/to/base-or-branch \
  taskset -c 0-15 .venv/bin/python -m vllm.entrypoints.cli.main serve /path/to/clef-flash \
  --served-model-name clef --dtype bfloat16 --max-model-len 8192 \
  --max-num-seqs 64 --max-num-batched-tokens 16384 --gpu-memory-utilization 0.3 \
  --limit-mm-per-prompt '{"image":0,"video":0}' --no-enable-prefix-caching \
  --host 127.0.0.1 --port 18971 --seed 0

taskset -c 16-19 .venv/bin/python bench_confirm.py --label main-A --duration 15 --rounds 2

The client warms up each case with max(C, 8) requests, includes draining outstanding requests in elapsed time, and samples memory/clocks every 0.5 seconds. vllm bench serve had no native SystemOne adapter in the tested versions. Save as bench_confirm.py:

import argparse
import asyncio
import json
import math
from pathlib import Path
import statistics
import time
import aiohttp
import pynvml
import psutil

ROOT=Path(__file__).parent
QUESTIONS=[
 {'type':'choice','instructions':'Which team should handle this ticket?','criteria':{'billing':None,'shipping':None,'security':None}},
 {'type':'noul','instructions':'Does the customer report being charged twice?'},
 {'type':'score','instructions':'How urgent is the customer request?','criteria':['routine','soon','urgent']},
 {'type':'choice','instructions':'What language is the message written in?','criteria':{'English':None,'French':None,'Japanese':None}},
 {'type':'noul','instructions':'Does the customer ask for a refund?'},
 {'type':'choice','instructions':'What product category is involved?','criteria':{'subscription':None,'clothing':None,'food':None}},
 {'type':'score','instructions':'How dissatisfied is the customer?','criteria':['satisfied','neutral','dissatisfied']},
 {'type':'noul','instructions':'Does the customer mention a delivery problem?'}]

def request(i,state_words,nq):
    fact='My subscription renewed yesterday. My card was charged twice for the same order. Please refund the duplicate payment. '
    filler='The account has an active monthly plan. Previous payments were processed successfully. The customer has reviewed the invoice and contacted the help desk. '
    words=(fact+filler*(state_words//len(filler.split())+1)).split()[:state_words]
    return {'model':'clef','state':f'Ticket {i}: '+' '.join(words),
            'questions':{f'q{q}':QUESTIONS[q%len(QUESTIONS)] for q in range(nq)}}

def percentile(xs,p):
    xs=sorted(xs)
    at=(len(xs)-1)*p/100
    lo=math.floor(at); hi=math.ceil(at)
    return xs[lo]+(xs[hi]-xs[lo])*(at-lo)

async def main():
    p=argparse.ArgumentParser()
    p.add_argument('--label',required=True)
    p.add_argument('--duration',type=float,default=15)
    p.add_argument('--rounds',type=int,default=3)
    p.add_argument('--profile',action='store_true')
    p.add_argument('--smoke',action='store_true')
    args=p.parse_args()
    pynvml.nvmlInit()
    gpu=pynvml.nvmlDeviceGetHandleByIndex(0)
    url='http://127.0.0.1:18971'
    cases=[(1,64,1),(8,64,1),(16,64,1),(32,64,1),(64,64,1),
           (16,256,8),(8,2048,32)]
    if args.smoke: cases=[(1,64,1),(4,128,8)]
    if args.profile: cases=[(1,64,1),(16,64,1),(16,256,8)]
    report={'label':args.label,'duration_per_round_s':args.duration,'cases':[]}
    timeout=aiohttp.ClientTimeout(total=120)
    async with aiohttp.ClientSession(timeout=timeout,connector=aiohttp.TCPConnector(limit=96)) as session:
        async def call(body):
            started=time.perf_counter()
            async with session.post(url+'/v1/systemone',json=body) as response:
                data=await response.json()
                if response.status!=200: raise RuntimeError((response.status,data))
            return (time.perf_counter()-started)*1000,data
        for concurrency,words,questions in cases:
            bodies=[request(i,words,questions) for i in range(max(128,concurrency*2))]
            await asyncio.gather(*(call(bodies[i%len(bodies)]) for i in range(max(concurrency,8))))
            if args.profile:
                async with session.post(url+'/start_profile',json={'profile_prefix':f'C{concurrency}W{words}Q{questions}','max_iterations':8}) as r:
                    if r.status!=200: raise RuntimeError(await r.text())
            rounds=[]
            for ri in range(args.rounds):
                started=time.perf_counter()
                latencies=[]; usages=[]; samples=[]; errors=[]
                monitoring=True
                memory_samples=[]
                async def monitor():
                    while monitoring:
                        info=pynvml.nvmlDeviceGetMemoryInfo(gpu)
                        memory_samples.append({'at_s':time.perf_counter()-started,
                            'device_used_mib':info.used/1048576,
                            'gpu_utilization':pynvml.nvmlDeviceGetUtilizationRates(gpu).gpu,
                            'temperature_c':pynvml.nvmlDeviceGetTemperature(gpu,pynvml.NVML_TEMPERATURE_GPU),
                            'sm_clock_mhz':pynvml.nvmlDeviceGetClockInfo(gpu,pynvml.NVML_CLOCK_SM),
                            'cpu_server_affinity_percent':psutil.cpu_percent(interval=None,percpu=True)[:16]})
                        await asyncio.sleep(0.5)
                monitor_task=asyncio.create_task(monitor())
                async def worker(w):
                    i=w
                    while time.perf_counter()-started<args.duration:
                        try:
                            latency,data=await call(bodies[i%len(bodies)])
                            latencies.append(latency);usages.append(data['usage']['input_tokens'])
                            if len(samples)<8: samples.append({'body':bodies[i%len(bodies)],'response':data})
                        except Exception as e:
                            errors.append(str(e))
                            break
                        i+=concurrency
                await asyncio.gather(*(worker(w) for w in range(concurrency)))
                elapsed=time.perf_counter()-started
                monitoring=False
                await monitor_task
                if not latencies: raise RuntimeError(errors)
                row={'concurrency':concurrency,'state_words':words,'questions':questions,'round':ri,
                     'requests':len(latencies),'errors':errors,'elapsed_s':elapsed,
                     'requests_per_s':len(latencies)/elapsed,'questions_per_s':len(latencies)*questions/elapsed,
                     'p50_ms':percentile(latencies,50),'p99_ms':percentile(latencies,99),
                     'mean_input_tokens':statistics.mean(usages),'latencies_ms':latencies,'samples':samples,
                     'device_memory_peak_mib':max(x['device_used_mib'] for x in memory_samples),
                     'monitor_samples':memory_samples}
                rounds.append(row)
                print(json.dumps({k:v for k,v in row.items() if k not in ('samples','latencies_ms','monitor_samples')}),flush=True)
            if args.profile:
                async with session.post(url+'/stop_profile') as r:
                    if r.status!=200: raise RuntimeError(await r.text())
            report['cases'].extend(rounds)
            (ROOT/f'serving-{args.label}.json').write_text(json.dumps(report,indent=2))

if __name__=='__main__': asyncio.run(main())

Test plan: run the existing self-contained reference test on each source tree (parent conftest disabled), selecting the reserved GPU:

OMP_NUM_THREADS=1 PYTHONPATH=/path/to/base-or-branch .venv/bin/python -m pytest \
  --noconftest -o addopts= \
  /path/to/base-or-branch/tests/models/multimodal/pooling/test_clef_omni.py -x -q
.venv/bin/pre-commit run --files vllm/model_executor/models/clef_joint_schema.py

---

AI assistance disclosure: Codex (GPT-6) helped draft the code and ran the benchmark/parity checks reported here.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[yewentao256]

LGTM, thanks for the work!

[taneem-ibrahim]

/ci run

[github-actions]

✅ Triggered Buildkite CI #97139 for commit cb8518248617.