[Model] Add Hopper FA4 relative attention for Inkling

[WoosukKwon]

Summary

Inkling's current relative-attention path uses the tml-fa4 sheared-bias interface, which is a Blackwell-specific path. On Hopper, that reaches incompatible SM90 call signatures and split-KV constraints.

This change:

• uses regular FA4 score_mod plus aux_tensors for Inkling relative bias on Hopper (SM90);
• keeps the optimized tml-fa4 sheared-bias path on Blackwell (SM100/SM110);
• keeps num_splits=1 on Hopper;
• adds architecture-routing and numerical correctness coverage for the standard score-mod path.

This is Python-only: no CMake, C++, CUDA, or tml-fa4 source changes.

Duplicate-work check

This does not duplicate an existing open PR. Before opening it, I ran:

gh pr list --repo vllm-project/vllm --state open --search "Inkling Hopper FA4 in:title,body"
gh pr list --repo vllm-project/vllm --state open --search "Inkling relative attention score_mod in:title,body"
gh pr list --repo vllm-project/vllm --state open --search "FA4 Hopper relative bias in:title,body"

All three searches returned no results.

Testing

pre-commit run
# Passed

CUDA_VISIBLE_DEVICES=0 /home/woosuk/workspace/vllm/.venv/bin/python -c 'import sys, vllm, vllm.vllm_flash_attn; sys.path.insert(0, "/tmp/vllm-inkling-hopper-fa4"); vllm.__path__.insert(0, "/tmp/vllm-inkling-hopper-fa4/vllm"); import pytest; raise SystemExit(pytest.main(["tests/models/inkling/test_fa4_rel_attention.py", "-q"]))'
# 71 passed

The full test file ran on a GB200. It covers routing for SM90/SM100/SM110/SM120, the existing Blackwell sheared-bias matrix, and a forced numerical test of the regular FA4 score-mod path against the PyTorch reference. Hopper hardware was not available locally, so CI on H200 should validate actual SM90 compilation and execution.

Model evaluation

Served thinkingmachines/Inkling-NVFP4 with Model Runner V2, TP4, no MTP, and max_model_len=8192:

VLLM_USE_V2_MODEL_RUNNER=1 \
FLASH_ATTENTION_CUTE_DSL_CACHE_ENABLED=1 \
CUDA_VISIBLE_DEVICES=0,1,2,3 \
vllm serve thinkingmachines/Inkling-NVFP4 \
  --tokenizer-mode inkling \
  --reasoning-parser inkling \
  --tool-call-parser inkling \
  --enable-auto-tool-choice \
  --tensor-parallel-size 4 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.8 \
  --kernel-config.enable_flashinfer_autotune=False \
  --trust-remote-code

Ran vLLM's canonical GSM8K evaluator over the chat-completions API:

evaluate_gsm8k(
    num_questions=1319,
    num_shots=5,
    max_tokens=2048,
    model="thinkingmachines/Inkling-NVFP4",
    use_chat_completions=True,
    temperature=0.0,
    seed=42,
    request_timeout_seconds=14400,
)

Full GSM8K result:

• Accuracy: 94.768% (1,250/1,319)
• Invalid responses: 2.957% (39/1,319)
• Total output tokens: 478,606
• Evaluation latency: 262.31 seconds
• Throughput: 5.028 questions/s, 1,824.58 output tokens/s
• No evaluator-side HTTP failures or timeouts; no server runtime errors

AI assistance

AI assistance was used to implement and test this change. The human submitter must review every changed line and understands and can defend the change end-to-end before marking this PR ready for review.

[claude]

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.

_{Tip: disable this comment in your organization's Code Review settings.}

[mergify]

Hi @WoosukKwon, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.