Follow-ups from PR #130: golden reference JSON + LazyTensor FP8 dequant

[titaiwangms]

Summary

The golden test case at testdata/cases/vision-language/ministral-3-3b.yaml references the gated HuggingFace model mistralai/Ministral-3-3B-Instruct-2512. Without an HF auth token, the test cannot run — making it unusable in CI.

Problem

• The model is gated (requires HuggingFace authentication)
• CI does not have HF auth tokens configured
• The golden test provides no value without the ability to download and run the model

Proposed Solution

1. Generate the reference JSON data (expected model outputs) using an authenticated HF account
2. Commit the reference JSON alongside the YAML test case in testdata/cases/vision-language/
3. This allows the golden test to run in CI without needing HF authentication (compares against pre-computed reference, not live model)

This follows the pattern used by other golden test cases in the repo.

Additional Context

• Related to PR Add Pixtral/Ministral3 VLM support: dynamic vision, FP8 dequant, PatchMerger fix, MHA #130 (Pixtral/Ministral3 VLM support)
• justinchuby suggested this in PR Add Pixtral/Ministral3 VLM support: dynamic vision, FP8 dequant, PatchMerger fix, MHA #130 review: "Is it possible to generate the json data with your hf login?"
• The test_vision_features_parity integration test also requires HF access — could benefit from the same reference JSON approach

[titaiwangms]

Additional: LazyTensor for FP8 dequantization

From PR #130 review by @justinchuby (_weight_loading.py:225):

Current behavior: _dequantize_fp8_weights() performs weight * scale_inv during weight loading (eager multiplication). This happens before ONNX graph construction.

Suggestion: Defer dequantization to serialization time using LazyTensor, which would reduce peak memory during graph building.

Challenge: FP8 dequant involves multiplication (fp8_weight.to(bf16) * scale_inv), not just a dtype cast. The current _assign_weight pipeline uses LazyTensor for dtype casts after dequant, but deferring the multiplication itself would require carrying scale_inv metadata through the pipeline.

Status: Follow-up optimization. Current eager approach is correct and works.

[titaiwangms]

Feasibility Analysis

Investigated both parts of this issue:

1. Golden Reference JSON Generation

✅ Fully feasible. The existing scripts/generate_golden.py handles VL cases. To generate:

python scripts/generate_golden.py --case testdata/cases/vision-language/ministral-3-3b.yaml --device cuda

This produces L4 (logits) and L5 (generation) JSON reference files.

Important caveat: Even with golden JSON committed, the e2e_golden_test still needs HF access at test time to download safetensors for mobius.build(load_weights=True). The golden JSON removes the need for HF PyTorch inference comparison, but NOT weight download. So skip_reason should remain for unauthenticated CI.

Action needed: Someone with HF access to the gated mistralai/Ministral-3-3B-Instruct-2512 model needs to run the generation script locally and commit the output JSON files.

2. LazyTensor FP8 Dequant

The current eager dequantization in _weight_loading.py (lines 150-199) works correctly — fp8.to(bf16) * scale_inv. Deferring to ir.LazyTensor would require carrying scale_inv metadata through the weight assignment pipeline, since LazyTensor only supports dtype casts (not multiplication). This is a medium-complexity optimization that is correctness-neutral.

Recommendation: Defer LazyTensor FP8 dequant to a separate PR. The current eager approach is functional and correct.

[titaiwangms]

Updated Analysis: Golden Tests Require HF Token

After deeper investigation, generating golden JSON alone is not sufficient to enable CI testing for gated models. Here's why:

The L4/L5 test flow

1. mobius.build(model_id, load_weights=True) — downloads weights from HF (needs auth)
2. Run ONNX inference with real weights
3. Compare outputs against golden JSON reference

The test needs real weights to produce meaningful outputs — without them, inference produces garbage. This is fundamental to how L4/L5 tests work.

Current state of gated VL models

Model	Graph test (L1-L3)	Golden JSON	Integration test (L4-L5)
ministral-3-3b	✅ passes	❌ not generated	❌ skipped (gated)
gemma-3-4b	✅ passes	❌ not generated	❌ skipped (gated)
mllama	✅ passes	❌ not generated	❌ skipped (gated)
qwen3-vl (non-gated)	✅ passes	✅ exists	✅ runs in CI
llava (non-gated)	✅ passes	✅ exists	✅ runs in CI

Options for Ministral-3 integration testing

1. Configure HF_TOKEN as CI secret + generate golden JSON → tests run normally (same pattern as non-gated models)
2. Accept graph-only testing (L1-L3) — already covers architecture correctness without HF access
3. Find a non-gated alternative — unlikely for this specific model

Alternatives we evaluated and ruled out:

• Store ONNX models as test artifacts: 6-44 GB per model — not viable in git
• Pre-compute both ONNX and HF outputs: Just compares two static files, doesn't actually test the build pipeline
• Run tests offline with golden JSON only: Fundamentally impossible — need weights for inference

Recommendation

The cleanest path is configuring HF_TOKEN in CI. This unblocks integration tests for all 3 gated VL models at once, not just Ministral-3.

[titaiwangms]

@justinchuby if the models are gated, L4 and L5 can't be guarded by CI because HF token is needed.

[justinchuby]

True that we cannot run them in the CI, but we can still have the json data you generated (and keep the skip reason), so that we can validate locally without having to rerun transformers.

I don’t think analysis for LazyTensor is accurate: we have capture anything in the function for LazyTensor. We just need to know when to create the ir tensors because in the first stage tensors are torch tensors. If we must manipulate the tensors in the first stage, using an ir tensor will be harder