Skip to content

QVAC-17304 feat[api]: add img2img support to SDK diffusion API#1662

Merged
maxim-smotrov merged 12 commits into
tetherto:mainfrom
maxim-smotrov:feature/sdk-diffusion-img2img
Apr 22, 2026
Merged

QVAC-17304 feat[api]: add img2img support to SDK diffusion API#1662
maxim-smotrov merged 12 commits into
tetherto:mainfrom
maxim-smotrov:feature/sdk-diffusion-img2img

Conversation

@maxim-smotrov

@maxim-smotrov maxim-smotrov commented Apr 17, 2026

Copy link
Copy Markdown
Contributor

🎯 What problem does this PR solve?

  • The SDK diffusion() API only supported txt2img — users could not use the img2img capability of the underlying diffusion add-on.

📝 How does it solve it?

  • Add init_image (Uint8Array on the client, base64 on the wire), strength, and img_cfg_scale to diffusionRequestSchema.
  • Client diffusion() accepts init_image: Uint8Array and base64-encodes it before sending; cross-runtime encodeBase64 helper added in utils/encoding.ts (uses btoa so it works in Node, Bare, and React Native/Hermes).
  • Server bare plugin decodes init_image to a Buffer and forwards init_image, strength, and img_cfg_scale to @qvac/diffusion-cpp.
  • Bump @qvac/diffusion-cpp to ^0.2.0 (img2img-capable).

🧪 How was it tested?

  • Unit tests added in test/unit/sdcpp-plugin.test.ts
  • New runnable example examples/diffusion-img2img.ts demonstrating end-to-end usage.

🔌 API Changes

Here's an example usage of the new img2img API.

import { loadModel, diffusion, SD_V2_1_1B_Q8_0 } from "@qvac/sdk";
import fs from "fs";

const modelId = await loadModel({ modelSrc: SD_V2_1_1B_Q8_0, modelType: "diffusion" });

// SD / SDXL — SDEdit
const initImage = new Uint8Array(fs.readFileSync("input.png"));
const { outputs } = diffusion({
  modelId,
  prompt: "oil painting style, vibrant colors",
  init_image: initImage,
  strength: 0.5, // 0 = keep source, 1 = ignore source
});

// FLUX.2 — in-context conditioning
// NOTE: requires `prediction: "flux2_flow"` set on the model config at loadModel time.
// `strength` is ignored on this path.
const { outputs: fluxOutputs } = diffusion({
  modelId,
  prompt: "turn into watercolor",
  init_image: initImage,
});

const buffers = await outputs;
fs.writeFileSync("out.png", buffers[0]!);

Comment thread packages/sdk/test/unit/sdcpp-plugin.test.ts
Comment thread packages/sdk/schemas/sdcpp-config.ts
Comment thread packages/sdk/test/unit/sdcpp-plugin.test.ts Outdated

@lauripiisang lauripiisang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can't be merged - test changes will crash mobile

…ile and desktop due to file system logic diff
NamelsKing
NamelsKing previously approved these changes Apr 22, 2026
@maxim-smotrov

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

Copy link
Copy Markdown
Contributor

Tier-based Approval Status

**PR Tier:** TIER1

**Current Status:** ✅ APPROVED

**Requirements:**
- 1 Team Member approval ✅ (2/1)
- 1 Team Lead OR Management approval ✅ (1/1)



---
*This comment is automatically updated when reviews change.*

@maxim-smotrov maxim-smotrov merged commit e553896 into tetherto:main Apr 22, 2026
13 checks passed
Proletter pushed a commit that referenced this pull request May 24, 2026
* feat[api]: add img2img support to SDK diffusion API

* fix(sdk): enforce img_cfg_scale default of -1 at the schema level

* test(sdk): split sdcpp diffusion dispatcher tests into focused cases and extract plugin mock into withMockDiffusionPlugin helper

* chore(test/diffusion): cleanup the comments

* test: add diffusion img2img test definition with init_image param

* fix(test/diffusion): split shared executor into separate ones for mobile and desktop due to file system logic diff

---------

Co-authored-by: Simon Iribarren <simon.ig13@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants