[liquanfeng]

Reopened as one PR after #46580 was reverted in #48054: the model half alone was 2126 lines that nothing in the serving path used. Per that revert note, the transformer and the pipeline that serves it now land together.

Motivation

Add BRIA Fibo 2.0 to SGLang Diffusion. Fibo 2.0 is an 8B flow-matching model that both generates and edits images in the FLUX.2 VAE latent space: Qwen3-VL-4B encodes structured JSON captions, a two-layer Perceiver resamples those features into 256-512 "gist" tokens, and five transformer blocks additionally cross-attend to Qwen3-VL's internal layers. Fibo-2.0-edit-turbo is distilled for 4-step, guidance-free editing; Fibo-2.0-dev serves 30-step generation and editing with classifier-free guidance.

The weights are released under CC BY-NC 4.0 and require accepting BRIA's license on Hugging Face, which is why the end-to-end test is opt-in.

Modifications

Model

• runtime/models/dits/bria_fibo2.py: the Fibo 2.0 transformer. It keeps the Z-Image single-stream block and adds a Perceiver resampler, five injection blocks with gated cross-attention, per-token noisy/clean modulation, three-axis RoPE over the gist, image and context-image planes, and patchify/unpatchify around the FLUX.2 latent layout.
• configs/models/encoders/bria_fibo2.py: builds the six text bundles from fifteen Qwen3-VL hidden states, each normalized to unit RMS per token before concatenation. The in-tree text tower returns one hidden state per decoder layer, one fewer than the reference, so the reference layer indices are shifted and the tuple length is asserted, turning a future collection change into an error instead of silently wrong conditioning.
• Parameter names match the checkpoint one to one; q/k/v stay unfused, because an injection block reads a 7680-wide text bundle through a 3584-wide query.

Pipeline and serving

• configs/pipeline_configs/bria_fibo2.py + runtime/pipelines/bria_fibo2_pipeline.py: one pipeline serves both paths, since sglang resolves the pipeline class from model_index.json and both checkpoints declare the same one.
• model_specific_stages/bria_fibo2/source.py: VAE-encodes every reference image into its own clean latent in batch.source_latents, the field the in-tree LLaDA edit model already uses; the images can have different sizes, so they cannot be concatenated onto the sequence.
• model_specific_stages/bria_fibo2/condition_encoding.py: two stages, one text-only and one image-aware, each stepping aside for the other's requests. They stay separate stages so each keeps its own name and component-residency bookkeeping; wrapping one inside the other left the inner stage unregistered and its encoder on the CPU.
• Edit requests default to TI2I in resolve_task_type: images plus an instruction fit both I2I and TI2I, and the shared resolver otherwise refuses to choose.
• Caption handling mirrors the reference, and an edit prompt becomes one JSON object with a vision marker per image plus the instruction. Qwen3-VL sees the same crop the VAE encodes, shrunk to 768 pixels on the long side.
• A repaint mask greys out the marked region on the original image in preprocess_vae_image: the first hook that receives the batch, and the one that runs before the crops are taken. A mask applies to one image and must match its size; both are rejected otherwise.
• Decoding reuses the FLUX.2 VAE chain, including its BatchNorm latent statistics, and the timestep schedule uses the FLUX linear shift computed from the image token count. The released checkpoints ship no separate tokenizer, so the pipeline exposes the one its processor carries.

API

• POST /v1/images/edits declared a mask upload and dropped it, so a masked edit silently ran as a plain edit. It is now saved and forwarded as SamplingParams.mask_path, the counterpart of image_path.

Docs

• docs/cookbook/diffusion/Fibo/Fibo-2.0.mdx with a command builder and the editing, masked and multi-reference requests.

Validation

Measured on one H100-80GB with the released weights:

Check	Result
Transformer vs diffusers, same weights and inputs, tiny configuration	max abs diff 7e-7, cosine 1.000000000
Full-size transformer, same weights and inputs, 4 Euler steps	final latents cosine 0.999974
VAE decode of the reference's final latents through this code path	PSNR 51.27 dB against the reference image
Text conditioning vs the reference	six bundles, cosine 0.9868-0.9932 (bf16 drift over 36 layers)
End-to-end edit, 1216x832, 4 steps, guidance 1.0	correct edit (black jacket to red leather), identity, pose and background preserved; 5.5 s, 27 GB peak
End-to-end generation, 1024x1024, 30 steps, guidance 5.0 (Fibo-2.0-dev)	correct image for a structured caption; 38.9 s
Masked edit, 1216x832, 4 steps	mean absolute change 0.254 inside the mask vs 0.008 outside (reference: 0.248 / 0.008)
Scheduler	sigmas [1.0, 0.902405, 0.755031, 0.506753, 0.0] for a 1216x832 canvas, identical to the reference

On current main, the opt-in end-to-end file runs green with real weights: 3 passed (turbo generation, turbo editing, dev generation with CFG). The rest: 178 passed, 3 skipped across the unit files (the skips are the diffusers-parity tests, which need diffusers>=0.42), node --test docs/scripts/tests/*.test.mjs 52/52, pre-commit run --all-files clean.

Limitations

• Single GPU: the new blocks do not implement sequence, tensor or Ulysses parallelism yet.
• Text-render hint images (text_images / fonts) are not supported yet; captions carrying text_render entries are accepted, they are just not turned into images.
• Requests whose prompts would need different gist-token counts in the same batch are rejected; send them as separate requests.
• Cache-DiT and the other diffusion acceleration features are not wired up for this model.
• Latents stay in float32 between steps, as in the reference.

On the size

This is 21 files / +4472. The earlier attempt split it at the model/pipeline boundary and that left the model unusable on its own (#48054). If a smaller PR is still preferred, the split that keeps each part working is: (1) model + pipeline + generation path, (2) editing, mask and docs — I can reshape it that way instead.

---

CI States

Latest PR Test (Base): ❌ Run #41610269241
Latest PR Test (Extra): ❌ Run #41610269162
Latest PR Test (AMD ROCm 10): ❌ Run #41610269154

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

[mintlify]

Preview approval required

This pull request comes from a fork, so Mintlify won't build a preview until someone with write access approves commit ac6e451.

Approve preview

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[BBuf]

[BBuf Review Agent] Could you split it below 2500 changed lines (now 4456)?

[liquanfeng]

[BBuf Review Agent] Could you split it below 2500 changed lines (now 4456)?

@BBuf Done — rebased onto main now that #46580 is merged, so the duplicated model files are gone: 15 files, +2329 −1 (2330 changed lines), 3 commits (afe7c64bc).

Local checks on this branch: 184 passed, 3 skipped across the affected test files, node --test docs/scripts/tests/*.test.mjs 52/52, pre-commit clean.

[liquanfeng]

Agreed with the revert in #48054 — the model half was unusable on its own, so the transformer and the pipeline are back in one PR here (21 files, +4472). This branch is rebased onto current main, so it is no longer stacked on anything.

Rebasing onto current main also surfaced two regressions that the split version did not have, both fixed in this PR:

1. ValueError: Specify task_type for this request; compatible tasks: ['I2I', 'TI2I'] — the shared resolver now refuses to choose between two compatible tasks, and an edit request (images plus an instruction) fits both. BriaFibo2PipelineConfig.resolve_task_type now defaults an image request to TI2I, which is what Fibo's edit contract is.
2. Expected all tensors to be on the same device ... index is on cuda:0, ... on cpu — my condition-encoding stage wrapped a second stage instance inside it, and the inner one never got a registered stage name, so its component residency bookkeeping did not apply and the text encoder stayed on the CPU. It is now two ordinary registered stages (text-only and image-aware), each stepping aside for the other's requests.

Verified on current main with the released weights: the opt-in end-to-end file is 3 passed (turbo generation, turbo editing, dev generation with CFG, 151 s on one H100), 178 passed, 3 skipped across the unit files (skips are the diffusers-parity tests, which need diffusers>=0.42), node --test docs/scripts/tests/*.test.mjs 52/52, and pre-commit run --all-files clean.

On the size: splitting at the model/pipeline boundary is what made the model dead code. If you still want smaller PRs, the split that keeps each one working is (1) model + pipeline + the generation path, (2) editing, mask and docs — say the word and I will reshape it that way.