[FP8] enable hunyuan-image-3 diffusion model with fp8 online quant

[xuechendi]

PLEASE FILL IN THE PR DESCRIPTION HERE ENSURING ALL CHECKLIST ITEMS (AT THE BOTTOM) HAVE BEEN CONSIDERED.

Purpose

#1854

To enable online-quant fp8 for hunyuan-image-3-moe diffusion path. Current Intel Arc B60 only has 24GB memory per card, can't run full hunyuan model with Bf16. Online quant fp8 is required.

Test Plan

stage_args:
  - stage_id: 0
    stage_type: diffusion
    runtime:
      process: true
      devices: "0,1,2,3,4,5,6,7"
      #devices: "0,1,2,3"
      max_batch_size: 1
    engine_args:
      enable_layerwise_offload: false
      vae_use_slicing: false
      vae_use_tiling: false
      cache_backend: null
      cache_config: null
      enable_cache_dit_summary: false
      enforce_eager: true
      enable_cpu_offload: false
      gpu_memory_utilization: 0.9
      quantization: "fp8"
      parallel_config:
        pipeline_parallel_size: 1
        data_parallel_size: 1
        tensor_parallel_size: 8
        #tensor_parallel_size: 4
        enable_expert_parallel: false
        sequence_parallel_size: 1
        ulysses_degree: 1
        ring_degree: 1
        cfg_parallel_size: 1
        vae_patch_parallel_size: 1
        use_hsdp: false
        hsdp_shard_size: -1
        hsdp_replicate_size: 1
      model_stage: diffusion
    final_output: true
    final_output_type: image

Remain issue:

 [diffusers_loader.py:326] Following weights were not initialized from checkpoint: 
{'model.layers.11.mlp.experts.w2_weight_scale', 'model.layers.10.mlp.experts.w13_weight_scale', 'model.layers.3.mlp.experts.w2_weight_scale', 'model.layers.24.mlp.experts.w13_weight_scale', 'model.layers.29.mlp.experts.w2_weight_scale', 'model.layers.27.mlp.experts.w2_weight_scale', 'model.layers.6.mlp.experts.w13_weight_scale', 'model.layers.4.mlp.experts.w2_weight_scale', ...
'model.layers.18.mlp.experts.w2_weight_scale', 'model.layers.28.mlp.experts.w2_weight_scale'}

python text_to_image.py \
--model /mnt/data/tencent--HunyuanImage-3.0 \
--prompt "A brown and white dog is running on the grass" \
--output "outputs/hunyuan_image_3_0/hunyuan_image_3_0_output.png" \
--cfg-scale 4.0 \
--num-inference-steps 50 \
--enforce-eager \
--stage-configs-path xpu/omni_stage_configs/hunyuan_image_3_moe.yaml

Output looks OK

Test Result

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan. Please provide the test scripts & test commands. Please state the reasons if your codes don't require additional test scripts. For test file guidelines, please check the test style doc
• The test results. Please paste the results comparison before and after, or the e2e results.
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model. Please run mkdocs serve to sync the documentation editions to ./docs.
• (Optional) Release notes update. If your change is user-facing, please update the release notes draft.

BEFORE SUBMITTING, PLEASE READ https://github.com/vllm-project/vllm-omni/blob/main/CONTRIBUTING.md (anything written below this line will be removed by GitHub Actions)

[xuechendi]

@lishunyang12 , wondering if anyone is on hunyuan-image-3-moe? If not, would like to init a PR to support

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2bf5674f07

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[hsliuustc0106]

@lishunyang12 PTAL

[lishunyang12]

LGTM from the implementation. @xuechendi If you want to merge it before the quantization framework is settled, can you help testing it afterwards in #1764?

[xuechendi]

@lishunyang12 , sure, I can verify once #1764 landed

[lishunyang12]

Please rebase after PR1908 is merged and this pr will be integrated before 1764.

[xuechendi]

pending on #1908 landing

[hsliuustc0106]

PR Review: [FP8] enable hunyuan-image-3 diffusion model with fp8 online quant

Gate Status: PASSING ✓

All CI checks pass.

---

What This PR Does

This is a well-scoped PR that:

1. Threads quant_config through HunyuanImage3DecoderLayer and HunyuanImage3Model
2. Updates weight loading to allow missing weight_scale weights (expected for online FP8)
3. Uses get_vllm_quant_config_for_layers in the pipeline

---

Minor Suggestions (Non-blocking)

1. Debug Log Level

logger.info(f"quant_config: {quant_config}")

Consider using logger.debug() to avoid noisy logs in production.

2. Warning Message Grammar

logger.warning(
    f"Following weights scale were not initialized from checkpoint: {weights_scale_not_loaded}"
)

Suggest: "Following weight_scale weights were not initialized from checkpoint: ..."

---

Note on "Remain Issue"

The missing weight_scale weights mentioned in the PR description is expected behavior for online FP8 quantization — scales are computed at load time, not loaded from checkpoint.

---

Summary

Validated	Minor Issues
FP8 config properly threaded	Debug log level
Weight scale handling correct	Warning grammar
Manual test shows working output

---

🤖 Generated with Claude Code

[xuechendi]

@hsliuustc0106 , @lishunyang12 , I have rebased this PR after #1908 merged.
Validated in local 8 cards XPU, output image is good, same as before.

[lishunyang12]

Looks good, rebase is clean. The two minor nits from the other review are non-blocking.

[xuechendi]

@lishunyang12 , thanks, I've fixed the nit and rebased

[lishunyang12]

Since you are exploring quantization on a new diffusion architecture. Can you help us find out which one layer in hunyuan-image3 is the most sensitive to FP8 towards final image output so that user know the trade-off and utilize this memory-saving feature more effectively. Not a blocker and you can carry out this experiment after the integration. and add your insight on the doc. https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/diffusion/quantization/fp8/#configuration Also, could you please refer to #1470 and show us a table like this

This would be very useful and insightful:)

[hsliuustc0106]

@yjb767868009 please check whether int8 can work in npu?

[hsliuustc0106]

@ZJY0516 @SamitHuang PTAL whether we have stage config yaml for diffusion model?

[xuechendi]

We don't usually have, this only needed by hunyuan
and I proposed a change to introduce stage selection in #1826
if that merged, we don't really need user to specify stage_configs_yaml here.