Skip to content

QVAC-18135 feat[api]: add FLUX.2 multi-reference fusion and LoRA adapter support to diffusion API#1838

Merged
maxim-smotrov merged 16 commits into
tetherto:mainfrom
maxim-smotrov:feature/sdk-flux2-fusion-and-lora
May 5, 2026
Merged

QVAC-18135 feat[api]: add FLUX.2 multi-reference fusion and LoRA adapter support to diffusion API#1838
maxim-smotrov merged 16 commits into
tetherto:mainfrom
maxim-smotrov:feature/sdk-flux2-fusion-and-lora

Conversation

@maxim-smotrov

Copy link
Copy Markdown
Contributor

🎯 What problem does this PR solve?

  • FLUX.2 supports conditioning on multiple input images ("multi-reference fusion"), but the SDK's diffusion() API only accepted a single init_image.
  • Users had no way to apply a LoRA adapter on a per-generation basis through the SDK, nor to control whether the adapter persists across subsequent calls.

📝 How does it solve it?

  • Adds init_images: Uint8Array[] to DiffusionClientParams for FLUX.2 fusion; mutually exclusive with init_image (enforced both at compile time via discriminated union and at runtime via Zod refine).
  • Adds FLUX.2 reference-image tunables increase_ref_index and auto_resize_ref_image.
  • Adds per-call lora field, validated as an absolute filesystem path (POSIX, Windows drive-letter, and UNC). Relative paths are rejected because the SDK runs across processes with differing cwds.
  • Adds lora_apply_mode: "auto" | "immediately" | "at_runtime" to sdcppConfigSchema (set at loadModel time) controlling whether the adapter is fused permanently or applied per-call.
  • Bumps @qvac/diffusion-cpp to ^0.5.0 for the underlying addon support and wires the new fields through the bare plugin op and the desktop / mobile / shared test executors.

🧪 How was it tested?

  • Unit tests in sdcpp-plugin.test.ts cover: lora absolute-path validation (POSIX / Windows / UNC accepted; relative, bare-filename, empty, wrong-type rejected), init_images shape validation, increase_ref_index / auto_resize_ref_image type checks, init_imageinit_images mutual exclusion, and lora_apply_mode enum acceptance.
  • E2E test diffusion-fusion-flux2-basic added to tests-qvac covering FLUX.2 fusion with two reference images.

🔌 API Changes

// FLUX.2 multi-reference fusion (mutually exclusive with init_image)
const refA = fs.readFileSync("scientist-a.jpg");
const refB = fs.readFileSync("scientist-b.jpg");
const { outputs } = diffusion({
  modelId,
  prompt: "a portrait using most visual traits from @image1 and the eyes from @image2",
  init_images: [refA, refB],
  width: 768,
  height: 768,
});

// Per-call LoRA adapter (absolute path required)
const { outputs } = diffusion({
  modelId,
  prompt: "a watercolor cat",
  lora: "/home/user/loras/watercolor.safetensors",
});

// LoRA persistence mode is selected at loadModel time
await loadModel(modelSrc, {
  modelType: "diffusion",
  modelConfig: { prediction: "flux2_flow", lora_apply_mode: "immediately" },
});

Comment thread packages/sdk/schemas/sdcpp-config.ts
Comment thread packages/sdk/client/api/diffusion.ts Outdated
Comment thread packages/sdk/test/unit/sdcpp-plugin.test.ts
Address PR review:
- shorten lora_apply_mode description in sdcppConfigSchema and drop the
  external file references the user can't see at usage time
- shorten the LoRA JSDoc block in diffusion.ts to the essentials
- drop unit tests that effectively re-assert zod built-ins (z.boolean(),
  z.string().min(1), individual enum members); keep the
  ABSOLUTE_PATH_PATTERN matrix, the mutual-exclusion refine, and one
  happy-path per new field

Made-with: Cursor
Comment thread packages/sdk/tests-qvac/tests/diffusion-tests.ts
Comment thread packages/sdk/test/unit/sdcpp-plugin.test.ts
@maxim-smotrov maxim-smotrov requested a review from a team as a code owner May 4, 2026 12:41

@lauripiisang lauripiisang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both prior threads resolved correctly:

  • Fusion validation (diffusion-tests.ts) — the new fusionFlux2Basic handler runs the fusion test plus a same-seed/prompt baseline with init_images: undefined and asserts byte divergence > 1%. Catches the silent txt2img-fallback case exactly as suggested.
  • Stream-schema mutual-exclusion regression test (sdcpp-plugin.test.ts) — added, plus a messages.includes("mutually exclusive") substring check that goes one step beyond what was requested.

CI green, gitflow intact, no new high/medium concerns. LGTM.

@maxim-smotrov

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

github-actions Bot commented May 5, 2026

Copy link
Copy Markdown
Contributor

Tier-based Approval Status

**PR Tier:** TIER1

**Current Status:** ✅ APPROVED

**Requirements:**
- 1 Team Member approval ✅ (2/1)
- 1 Team Lead OR Management approval ✅ (1/1)



---
*This comment is automatically updated when reviews change.*

@maxim-smotrov maxim-smotrov merged commit f541c91 into tetherto:main May 5, 2026
16 checks passed
tamer-hassan-tether pushed a commit that referenced this pull request May 5, 2026
…ter support to diffusion API (#1838)

* feat[api]: add FLUX.2 multi-reference fusion and LoRA adapter support to diffusion API

* doc[skiplog]: trim verbose lora docs and prune zod-builtin tests

Address PR review:
- shorten lora_apply_mode description in sdcppConfigSchema and drop the
  external file references the user can't see at usage time
- shorten the LoRA JSDoc block in diffusion.ts to the essentials
- drop unit tests that effectively re-assert zod built-ins (z.boolean(),
  z.string().min(1), individual enum members); keep the
  ABSOLUTE_PATH_PATTERN matrix, the mutual-exclusion refine, and one
  happy-path per new field

Made-with: Cursor

* test[api]: validate FLUX.2 fusion diverges from txt2img baseline and reject conflicting init_image inputs
Proletter pushed a commit that referenced this pull request May 24, 2026
…ter support to diffusion API (#1838)

* feat[api]: add FLUX.2 multi-reference fusion and LoRA adapter support to diffusion API

* doc[skiplog]: trim verbose lora docs and prune zod-builtin tests

Address PR review:
- shorten lora_apply_mode description in sdcppConfigSchema and drop the
  external file references the user can't see at usage time
- shorten the LoRA JSDoc block in diffusion.ts to the essentials
- drop unit tests that effectively re-assert zod built-ins (z.boolean(),
  z.string().min(1), individual enum members); keep the
  ABSOLUTE_PATH_PATTERN matrix, the mutual-exclusion refine, and one
  happy-path per new field

Made-with: Cursor

* test[api]: validate FLUX.2 fusion diverges from txt2img baseline and reject conflicting init_image inputs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants