Round fused SwiGLU to the storage dtype before ConvRot.

[niehen6174]

Summary

int8_linear(..., input_act="swiglu") is supposed to equal int8_linear(silu(gate)*up, ...).
The fused CUDA/HIP quantizer left silu*up in fp32, so ConvRot's per-row absmax
diverged from the eager chain (which materializes BF16/FP16). One LSB of scale
rewrites the whole INT8 row.

This PR rounds silu and the multiply to the storage dtype before the rotation.

Accuracy

Kitchen unit shapes + MiniMax-H3 fc2 (N=5376, K=14336), isolated build of this patch,
F.silu(gate)*up then int8_linear vs fused:

	bit-neq	max|Δ|
M=4096	0 / 22,020,096	0
M=32700 (H3 1344×768 packed rows)	0 / 175,795,200	0

Same H3 t2va, 50 NFE, seed=42, INT8+FA, old fused kernel (fp32 product) vs
unfused INT8 (eager SwiGLU then quantize):

	video PSNR mean / min	audio PSNR
old vs unfused INT8	22.0 / 19.8	49.8

That is a different diffusion sample, not a slightly noisier tensor.
After this PR, fc2 is bit-exact, so that pair should be identical (PSNR 99).

Speed (why keep the fuse)

H3 fc2, M=32700, RTX 4090 D, CUDA event, 50 DiT blocks/step:

	ms / block
silu*mul + int8_linear	16.44
fused input_act=swiglu (this PR)	12.47
delta	−3.97

20 NFE exclusive 4090 D (SGLang, earlier run with the old fuse; wall-clock
should match because the extra rounding is register-only):

	denoise	e2e	step
INT8+FA, no fuse	276.1 s	294.7 s	13.80 s
+ fused SwiGLU	271.7 s	290.3 s	13.58 s
delta	−4.4 s	−4.4 s	−0.22 s

Test plan

pytest tests/test_int8_input_act.py -k swiglu

CUDA+bf16 int8_linear(x, input_act="swiglu") must torch.equal the eager chain.
FP16/BF16 fused INT8 codes must equal quantize(silu(gate)*up).

Files

• comfy_kitchen/backends/cuda/ops/int8_linear.cu — CUDA quantizer
• comfy_kitchen/backends/hip/hadamard.h — HIP mirror
• tests/test_int8_input_act.py — require bit-exact on the fused CUDA path

[github-actions]

✅ All contributors have signed the CLA. Thank you! This PR is ready to be merged.
_{Posted by the CLA Assistant Lite bot.}

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: bba77187-2f1f-4aae-af21-987869946220

📥 Commits

Reviewing files that changed from the base of the PR and between ff83be3 and 860255d.

📒 Files selected for processing (3)

• comfy_kitchen/backends/cuda/ops/int8_linear.cu
• comfy_kitchen/backends/hip/hadamard.h
• tests/test_int8_input_act.py

Included review availability: Your plan provides up to 2 included reviews per hour; 1 remains after this review.

---

📝 Walkthrough

Walkthrough

Changes

The fused SwiGLU paths now round SiLU and product values according to FP16 or BF16 storage precision. FP32 behavior remains unchanged. Tests now require exact CUDA parity with eager activation results.

Suggested reviewers: christian-byrne

Merge Risk: ⚪ Minimal · up to 86025

The PR is merge-ready after normal checks and review; no actionable merge-blocking risk remains.

🚥 Pre-merge checks | ✅ 2

✅ Passed checks (2 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches 💡 1

🛠️ Fix failing CI checks 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests

✨ Simplify code

• Create PR with simplified code

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[niehen6174]

I have read and agree to the Contributor License Agreement

[niehen6174]

@christian-byrne Could you take a look at this PR when you have a chance? Thanks!