QVAC-14104 feat[api]: SDK diffusion plugin integration#873
Closed
donriddo wants to merge 16 commits into
Closed
Conversation
36dcd49 to
eea1898
Compare
Contributor
|
i would lean to something different in api perspective |
Contributor
Tier-based Approval Status |
Contributor
|
u would probably want to rebase once more before targeting main and wire up profiler to diffusion addon, can be found in this PR #836 |
|
|
||
| // FLUX.2 [klein] uses a split-layout: separate diffusion model + LLM text encoder | ||
| const diffusionModelPath = process.argv[2]; | ||
| const llmModelPath = process.argv[3]; |
Contributor
There was a problem hiding this comment.
lets replace these with their counterpart srcs from model registry
| import path from "path"; | ||
|
|
||
| const modelPath = process.argv[2]; | ||
| const inputImagePath = process.argv[3]; |
Contributor
There was a problem hiding this comment.
same here, we can also upload sample imgs to or use existing from examples/image dir
| import fs from "fs"; | ||
| import path from "path"; | ||
|
|
||
| const modelPath = process.argv[2]; |
| modelName, | ||
| artifacts, | ||
| }) as { model: AnyModel; loader: FilesystemDL }; | ||
| }) as { model: AnyModel; loader: FilesystemDL | undefined }; |
Contributor
There was a problem hiding this comment.
confirming: diffusion addon doesnt use a loader?
a2189f8 to
80edfde
Compare
Wire up the stable-diffusion.cpp plugin through all SDK layers: - Schema: sdcpp-config.ts with config, stats, request/response schemas - Plugin: resolveConfig for companion artifacts, createModel, streaming handlers - Load model: diffusion entries in all 4 schema locations - Registration: model type, alias, engine-addon map, worker, pear pre-hook - Type widening: FilesystemDL | undefined for loader-less plugins
…add integration tests - Fix sampling_method enum to match C++ addon ground truth (dpm++2m not dpm++_2m) - Add 6 missing sampler values (ipndm, ipndm_v, ddim_trailing, tcd, res_multistep, res_2s) - Fix addon index.d.ts SamplerMethod type to match C++ parser - Consolidate generation ops into single unified handler (txt2img + img2img) - Add dedicated RPC handler, client API, and first-class generation() export - Add 15 integration test definitions and desktop executor - Add examples: txt2img, img2img, flux2-klein - Add comprehensive unit tests for schemas, plugin dispatch, and stats - Wire diffusion into handler-registry, common schemas, model-config-utils, get-model-info
The bare-client dispatches via handlers/index.ts (direct mode), not handler-registry.ts (IPC worker mode). Missing entry caused RPC_NO_HANDLER when running examples via bare runtime.
Add dedicated generateDiffusionName() to produce clean export constants for diffusion registry models (SD → SD_V2_1, SDXL → SDXL_BASE, FLUX, VAE). Includes 4 unit tests covering all model families.
Run bun update-models to pull 21 new models (including diffusion) from the live registry. Replace QVAC_DIFFUSION_MODEL env var in model-manager with the FLUX_2_KLEIN_4B_Q4_0 registry constant.
Resolve statsPromise after stream loop exits (not only on done:true), add statsRejecter for error propagation, derive GenerationClientParams from schema type to prevent drift, and fix lint warnings in generation ops and test executor.
Revert statsPromise try/catch/rejecter and GenerationClientParams Omit<> derivation — these diverged from the established patterns in ocr.ts, translate.ts, and transcription.ts. Also remove unrelated model history file that was incorrectly included.
ecb1bf8.txt was a codegen artifact from bun run update-models during the merge — it should not have been committed here.
FLUX.2 models require companion LLM (Qwen3-4B) and VAE models to create the stable-diffusion context. Without them, SdModel::load() fails. Also switches device from CPU to GPU and adds img2img test fixture.
…l diffusion examples All three diffusion examples now default to the required companion LLM (QWEN3_4B_Q4_K_M) and VAE (FLUX_2_KLEIN_4B_VAE) models, matching the desktop test configuration. Also switches device from cpu to gpu.
Replace photo.png with elephant.jpg from lib-infer-llamacpp-llm/media. Update generation test definitions to reference the new filename.
import.meta.dirname is undefined in Bare runtime. Use path.resolve with a CWD-relative path instead, matching the documented convention of running examples from the SDK root.
Replace ModelManager usage with AbstractModelExecutor base class, matching the pattern used by all other executors after PR #836.
The server already emits progress ticks (step/totalSteps/elapsedMs) during diffusion generation but the client was silently dropping them. Add a progressStream async generator to the generation() return type so SDK callers can show progress UI. Update the streaming-progress integration test to assert progress tick presence and field validity.
Refactor generation() to follow the completion() multi-stream pattern: a single background processResponses() task drives the RPC stream and fans out to outputStream, progressStream, outputs, and stats independently. This fixes two issues with the previous implementation: - consuming progressStream alone now works (no longer requires outputStream iteration to drive the RPC stream) - RPC errors propagate to all consumers (streams throw, promises reject)
984f293 to
7481157
Compare
Regenerates bun.lock and models/registry/models.ts to restore FLUX model entries that were lost during rebase conflict resolution.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🎯 What problem does this PR solve?
loadModel+generation())📝 How does it solve it?
SDK plugin wiring (36 files):
sdcpp-config.tsschemas: config, stats, request/response, client params withsampling_methodenum matching the C++SdGenHandlers.cpp::parseSampler()ground truth (14 valid samplers)sdcpp-generation/plugin.tswithdefinePlugin(),resolveConfig()for companion artifacts, deterministiccreateModel()generation.tsops handler (unified txt2img + img2img, mode frominit_imagepresence)generation-stream.tsRPC handler viadispatchPluginStream()generation()client API with streaming/non-streaming support (outputStreamfor streamed images,outputsfor buffered,statsfor generation metadata)handler-registry,handlers/index,common.ts,model-types,engine-addon-map,load-model,get-model-info,registry,plugin.tsconstants,worker.ts,pear/pre.ts,logging/namespaces,index.tsexportsAddon JS layer:
diskPath)Addon type definitions:
index.d.tsSamplerMethodtype aligned to C++ parser (14 valid values)Registry & codegen:
models/registry/models.ts)models/update-models/naming.tsandcodegen.tsmodels/history/d514e414.txtTests & examples:
GenerationExecutordesktop executor usingAbstractModelExecutor+ResourceManagerpatterndiffusion-txt2img.ts,diffusion-img2img.ts,diffusion-flux2-klein.ts🧪 How was it tested?
FLUX_2_KLEIN_4B_Q4_0+ companionsQWEN3_4B_Q4_K_M(LLM) +FLUX_2_KLEIN_4B_VAE(VAE), device:gpusdcpp-plugin.test.ts+update-models-naming.test.ts)bun run bare:exampleThe
generation()client helper exposesprogressStream(step/totalSteps/elapsedMs ticks),outputStream(streamed images),outputs(buffered images), andstats(final metadata). Thegeneration-streaming-progressintegration test asserts progress tick presence and field validity.🔌 API Changes
New exports:
generation,GenerationClientParams,GenerationProgressTick,GenerationStreamResponse,DiffusionStats,PLUGIN_DIFFUSION📦 Models
Added models
(Plus other diffusion entries added by registry codegen.)
@qvac/diffusion-cppusesfile:../lib-infer-diffusion— needs proper versioned reference before release