Preview: Ouro-1.4B inference with loop-level batching

[hsliuustc0106]

Add the first vllm-lt inference engine for ByteDance/Ouro-1.4B. The scheduler operates at individual recurrent-loop boundaries, allowing requests to exit at different depths and refill the recurrent batch while other requests continue looping.

Implementation

• A vLLM-style LLM facade, engine, scheduler, model runner, and native Ouro model; safetensors loading pins the official checkpoint revision without executing remote model code.
• Full-depth chunked prefill, cumulative-gate adaptive decode, and refill/no-refill scheduling. Prompt processing uses all four loops; subsequent decode tokens follow the configured exit policy.
• Depth-aware paged KV storage with last-exited propagation: a token's final computed KV is copied into skipped deeper slots. Admission reserves each request's complete KV budget to guarantee progress under memory pressure.
• PyTorch reference and Triton paged-attention backends, greedy/seeded sampling, step-wise output collection, cancellation, and a CLI.

Validation

• Fresh CPU suite: 54 passed, with 15 GPU cases intentionally skipped. Ruff lint, formatting, and Git whitespace checks pass.
• Previously recorded execution of the unchanged Python sources: 15 GPU cases passed, covering FP32/FP16/BF16 attention; real Ouro-1.4B FP32 checkpoint comparisons passed at all four loops, with maximum absolute logit error 6.2943e-5.
• Real-checkpoint FP32 generation agreed across serial, refill, and no-refill execution on both attention backends. Request KV blocks were released, and allocated GPU memory reached zero after clearing the process's cuBLAS workspace.
• Wheel packaging and a three-request CLI run with the real checkpoint passed. Numerical results are summarized in the validation record, which links to the original manifests and JSON results in Git history.

Preview limits

This is a synchronous, single-device preview. CUDA graphs, asynchronous lookahead scheduling, distributed execution, incremental KV allocation/preemption, and an HTTP server are not implemented. No throughput improvement or task-accuracy claim is made.

BF16 remains numerically unqualified: the tested generation outputs and exit depths matched across the six configurations, but strict logit tolerances failed at deeper loops. The original failure is preserved; FP32 is the validated default.

See the README, design, and validation record.

Roadmap

Track milestone progress and follow-up PRs in roadmap issue #2.

The Ouro performance roadmap prioritizes faster single-GPU Ouro inference. The next PR establishes a bounded benchmark/profiler baseline, with BF16 numerical diagnosis and an independent last-exited oracle proceeding in parallel. Profiling then selects runtime, CUDA-graph, attention, and KV work; asynchronous lookahead and serving follow later.

The roadmap defines milestone dependencies, acceptance criteria, workload controls, and run budgets. This documentation addition passed local-link, table/fence, and Git whitespace checks; runtime sources are unchanged and no new GPU measurements are claimed.