Skip to content

QVAC-14104 feat[api]: SDK diffusion plugin integration#873

Closed
donriddo wants to merge 16 commits into
tetherto:feature-media-generationfrom
donriddo:feat/diffusion-sdk-plugin
Closed

QVAC-14104 feat[api]: SDK diffusion plugin integration#873
donriddo wants to merge 16 commits into
tetherto:feature-media-generationfrom
donriddo:feat/diffusion-sdk-plugin

Conversation

@donriddo

@donriddo donriddo commented Mar 12, 2026

Copy link
Copy Markdown
Contributor

🎯 What problem does this PR solve?

  • The diffusion addon had no SDK surface — users couldn't call image generation through the standard QVAC client API (loadModel + generation())
  • The addon existed at the native layer but wasn't wired into the SDK plugin system, registry, or test harness

📝 How does it solve it?

SDK plugin wiring (36 files):

  • sdcpp-config.ts schemas: config, stats, request/response, client params with sampling_method enum matching the C++ SdGenHandlers.cpp::parseSampler() ground truth (14 valid samplers)
  • sdcpp-generation/plugin.ts with definePlugin(), resolveConfig() for companion artifacts, deterministic createModel()
  • generation.ts ops handler (unified txt2img + img2img, mode from init_image presence)
  • generation-stream.ts RPC handler via dispatchPluginStream()
  • generation() client API with streaming/non-streaming support (outputStream for streamed images, outputs for buffered, stats for generation metadata)
  • Wired into: handler-registry, handlers/index, common.ts, model-types, engine-addon-map, load-model, get-model-info, registry, plugin.ts constants, worker.ts, pear/pre.ts, logging/namespaces, index.ts exports

Addon JS layer:

  • Fix companion model path resolution to support absolute paths (was always joining with diskPath)

Addon type definitions:

  • index.d.ts SamplerMethod type aligned to C++ parser (14 valid values)

Registry & codegen:

  • Diffusion model entries added to generated registry (models/registry/models.ts)
  • Codegen naming support for diffusion models in models/update-models/naming.ts and codegen.ts
  • History artifact models/history/d514e414.txt

Tests & examples:

  • 15 integration test definitions: txt2img, img2img, streaming, progress, stats, batch, samplers, schedulers, seed reproducibility, negative prompt, cfg_scale, error cases
  • GenerationExecutor desktop executor using AbstractModelExecutor + ResourceManager pattern
  • Unit tests for schema validation, plugin dispatch, stats fields, naming conventions
  • 3 examples: diffusion-txt2img.ts, diffusion-img2img.ts, diffusion-flux2-klein.ts

🧪 How was it tested?

  • 15/15 generation integration tests pass against FLUX.2 Klein 4B Q4_0 on GPU via the MQTT-based QVAC test framework
    • Resource config: FLUX_2_KLEIN_4B_Q4_0 + companions QWEN3_4B_Q4_K_M (LLM) + FLUX_2_KLEIN_4B_VAE (VAE), device: gpu
  • 77/77 unit tests pass (sdcpp-plugin.test.ts + update-models-naming.test.ts)
  • All 3 examples confirmed working end-to-end via bun run bare:example
  • No new lint/typecheck errors

The generation() client helper exposes progressStream (step/totalSteps/elapsedMs ticks), outputStream (streamed images), outputs (buffered images), and stats (final metadata). The generation-streaming-progress integration test asserts progress tick presence and field validity.

🔌 API Changes

import { generation, loadModel, FLUX_2_KLEIN_4B_Q4_0, FLUX_2_KLEIN_4B_VAE, QWEN3_4B_Q4_K_M } from "@qvac/sdk";

const modelId = await loadModel({
  modelSrc: FLUX_2_KLEIN_4B_Q4_0,
  modelType: "diffusion",
  modelConfig: {
    device: "gpu",
    threads: 4,
    llmModelSrc: QWEN3_4B_Q4_K_M,
    vaeModelSrc: FLUX_2_KLEIN_4B_VAE,
  },
});

// txt2img (non-streaming)
const { outputs, stats } = generation({ modelId, prompt: "a cat" });
const buffers = await outputs;

// txt2img (streaming)
const { outputStream } = generation({ modelId, prompt: "a cat", stream: true });
for await (const { data, outputIndex } of outputStream) { ... }

// img2img
const { outputs } = generation({
  modelId, prompt: "watercolor style",
  init_image: fs.readFileSync("photo.jpg"), strength: 0.75,
});

New exports: generation, GenerationClientParams, GenerationProgressTick, GenerationStreamResponse, DiffusionStats, PLUGIN_DIFFUSION

📦 Models

Added models

FLUX_2_KLEIN_4B_Q4_0
FLUX_2_KLEIN_4B_VAE

(Plus other diffusion entries added by registry codegen.)

⚠️ Notes

  • @qvac/diffusion-cpp uses file:../lib-infer-diffusion — needs proper versioned reference before release
  • Version unchanged at current value — needs bump before merge

@donriddo donriddo requested review from a team as code owners March 12, 2026 23:42
@donriddo donriddo force-pushed the feat/diffusion-sdk-plugin branch from 36dcd49 to eea1898 Compare March 12, 2026 23:45
@donriddo donriddo marked this pull request as draft March 12, 2026 23:53
@donriddo donriddo changed the title feat(diffusion): consolidate SDK plugin, fix sampling_method schema, add integration tests feat(diffusion): SDK plugin integration — client API, RPC handler, schemas, examples, tests Mar 13, 2026
@donriddo donriddo changed the title feat(diffusion): SDK plugin integration — client API, RPC handler, schemas, examples, tests feat(diffusion): SDK plugin integration (client API, RPC handler, schemas, examples, tests) Mar 13, 2026
@donriddo donriddo changed the title feat(diffusion): SDK plugin integration (client API, RPC handler, schemas, examples, tests) feat(diffusion): SDK plugin integration Mar 13, 2026
@donriddo donriddo changed the title feat(diffusion): SDK plugin integration QVAC-14104: SDK plugin integration Mar 13, 2026
@NamelsKing NamelsKing changed the title QVAC-14104: SDK plugin integration QVAC-14104 feat: SDK diffusion plugin integration Mar 14, 2026
@NamelsKing NamelsKing changed the title QVAC-14104 feat: SDK diffusion plugin integration QVAC-14104 feat[api]: SDK diffusion plugin integration Mar 14, 2026
@NamelsKing

Copy link
Copy Markdown
Contributor

i would lean to something different in api perspective generation feels to generic for me. Mb diffusion?

@github-actions

github-actions Bot commented Mar 14, 2026

Copy link
Copy Markdown
Contributor

Tier-based Approval Status

**PR Tier:** TIER1

**Current Status:** ❌ PENDING

**Requirements:**
- 1 Team Member approval ❌ (0/1)
- 1 Team Lead OR Management approval ❌ (0/1)



---
*This comment is automatically updated when reviews change.*

@NamelsKing

Copy link
Copy Markdown
Contributor

u would probably want to rebase once more before targeting main and wire up profiler to diffusion addon, can be found in this PR #836


// FLUX.2 [klein] uses a split-layout: separate diffusion model + LLM text encoder
const diffusionModelPath = process.argv[2];
const llmModelPath = process.argv[3];

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lets replace these with their counterpart srcs from model registry

import path from "path";

const modelPath = process.argv[2];
const inputImagePath = process.argv[3];

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here, we can also upload sample imgs to or use existing from examples/image dir

import fs from "fs";
import path from "path";

const modelPath = process.argv[2];

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here

modelName,
artifacts,
}) as { model: AnyModel; loader: FilesystemDL };
}) as { model: AnyModel; loader: FilesystemDL | undefined };

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

confirming: diffusion addon doesnt use a loader?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no, it doesn't

Wire up the stable-diffusion.cpp plugin through all SDK layers:
- Schema: sdcpp-config.ts with config, stats, request/response schemas
- Plugin: resolveConfig for companion artifacts, createModel, streaming handlers
- Load model: diffusion entries in all 4 schema locations
- Registration: model type, alias, engine-addon map, worker, pear pre-hook
- Type widening: FilesystemDL | undefined for loader-less plugins
…add integration tests

- Fix sampling_method enum to match C++ addon ground truth (dpm++2m not dpm++_2m)
- Add 6 missing sampler values (ipndm, ipndm_v, ddim_trailing, tcd, res_multistep, res_2s)
- Fix addon index.d.ts SamplerMethod type to match C++ parser
- Consolidate generation ops into single unified handler (txt2img + img2img)
- Add dedicated RPC handler, client API, and first-class generation() export
- Add 15 integration test definitions and desktop executor
- Add examples: txt2img, img2img, flux2-klein
- Add comprehensive unit tests for schemas, plugin dispatch, and stats
- Wire diffusion into handler-registry, common schemas, model-config-utils, get-model-info
The bare-client dispatches via handlers/index.ts (direct mode), not
handler-registry.ts (IPC worker mode). Missing entry caused
RPC_NO_HANDLER when running examples via bare runtime.
Add dedicated generateDiffusionName() to produce clean export constants
for diffusion registry models (SD → SD_V2_1, SDXL → SDXL_BASE, FLUX,
VAE). Includes 4 unit tests covering all model families.
Run bun update-models to pull 21 new models (including diffusion) from
the live registry. Replace QVAC_DIFFUSION_MODEL env var in model-manager
with the FLUX_2_KLEIN_4B_Q4_0 registry constant.
Resolve statsPromise after stream loop exits (not only on done:true),
add statsRejecter for error propagation, derive GenerationClientParams
from schema type to prevent drift, and fix lint warnings in generation
ops and test executor.
Revert statsPromise try/catch/rejecter and GenerationClientParams
Omit<> derivation — these diverged from the established patterns
in ocr.ts, translate.ts, and transcription.ts. Also remove unrelated
model history file that was incorrectly included.
ecb1bf8.txt was a codegen artifact from bun run update-models
during the merge — it should not have been committed here.
FLUX.2 models require companion LLM (Qwen3-4B) and VAE models to create
the stable-diffusion context. Without them, SdModel::load() fails.
Also switches device from CPU to GPU and adds img2img test fixture.
…l diffusion examples

All three diffusion examples now default to the required companion
LLM (QWEN3_4B_Q4_K_M) and VAE (FLUX_2_KLEIN_4B_VAE) models, matching
the desktop test configuration. Also switches device from cpu to gpu.
Replace photo.png with elephant.jpg from lib-infer-llamacpp-llm/media.
Update generation test definitions to reference the new filename.
import.meta.dirname is undefined in Bare runtime. Use path.resolve
with a CWD-relative path instead, matching the documented convention
of running examples from the SDK root.
Replace ModelManager usage with AbstractModelExecutor base class,
matching the pattern used by all other executors after PR #836.
The server already emits progress ticks (step/totalSteps/elapsedMs) during
diffusion generation but the client was silently dropping them. Add a
progressStream async generator to the generation() return type so SDK
callers can show progress UI. Update the streaming-progress integration
test to assert progress tick presence and field validity.
Refactor generation() to follow the completion() multi-stream pattern:
a single background processResponses() task drives the RPC stream and
fans out to outputStream, progressStream, outputs, and stats independently.

This fixes two issues with the previous implementation:
- consuming progressStream alone now works (no longer requires
  outputStream iteration to drive the RPC stream)
- RPC errors propagate to all consumers (streams throw, promises reject)
@donriddo donriddo force-pushed the feat/diffusion-sdk-plugin branch from 984f293 to 7481157 Compare March 19, 2026 14:11
Regenerates bun.lock and models/registry/models.ts to restore FLUX
model entries that were lost during rebase conflict resolution.
@gianni-cor gianni-cor deleted the branch tetherto:feature-media-generation March 19, 2026 14:41
@gianni-cor gianni-cor closed this Mar 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants