Skip to content

feat(diffusion): add LoRA support to addon run config#1474

Merged
gabrielgrigoras-serv merged 13 commits into
mainfrom
feat/diffusion-lora
Apr 23, 2026
Merged

feat(diffusion): add LoRA support to addon run config#1474
gabrielgrigoras-serv merged 13 commits into
mainfrom
feat/diffusion-lora

Conversation

@gabrielgrigoras-serv

@gabrielgrigoras-serv gabrielgrigoras-serv commented Apr 9, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds end-to-end LoRA support to the diffusion addon.

Changes

  • adds optional lora field to JS run config
  • propagates lora through JS → native bridge
  • extends SdGenConfig with loraPath
  • adds handler for lora
  • maps LoRA into sd.cpp via sd_img_gen_params_t.loras
  • uses same ownership pattern as upstream (stable storage for paths)
  • minimal integration test for bridge forwarding
  • unit test for handler parsing

Notes

  • reuses existing LoRA mechanism from pinned sd.cpp fork
  • no behavior change when lora is not set
  • minimal changes, no refactor

Validation

  • JS bridge test passed
  • types check passed

@gabrielgrigoras-serv gabrielgrigoras-serv requested review from a team as code owners April 9, 2026 13:34
@gabrielgrigoras-serv gabrielgrigoras-serv self-assigned this Apr 9, 2026
Comment thread packages/lib-infer-diffusion/test/unit/test_sd_gen_handlers.cpp
aegioscy
aegioscy previously approved these changes Apr 21, 2026

@jesusmb1995 jesusmb1995 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing version and CHANGELOG update

dev-nid
dev-nid previously approved these changes Apr 21, 2026
jesusmb1995
jesusmb1995 previously approved these changes Apr 21, 2026
dev-nid
dev-nid previously approved these changes Apr 21, 2026

@gianni-cor gianni-cor left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this — the core change is small, correct, and safe (the PreparedLoras ownership pattern mirrors the pinned sd.cpp fork's CLI flow, and NRVO/vector-move preserves the captured c_str() across the return, so the current code is fine as a v0).

Requesting changes on two points before merge:

1. Validate params.lora in index.js, or document the silent-no-op

The JSDoc promises "Absolute path to a LoRA adapter", but index.js doesn't enforce it:

  • assertAbsolute(...) is only applied to files.* in the constructor. params.lora is never checked — a relative path from JS silently becomes CWD-relative in sd.cpp.
  • More importantly: apply_loras_immediately in the fork silently skips LoRAs that fail to load (stable-diffusion.cpp:1082: if (!lora || lora->lora_tensors.empty()) continue;). A typo, a wrong format, or a missing file produces a clean "successful" generation with no LoRA applied and no error returned to the caller.

Please do one of:

  • Preferred: add a one-shot validation in _runInternal next to the existing init_image check, e.g.:
    if (params.lora != null) {
      if (typeof params.lora !== 'string' || params.lora.length === 0) {
        throw new TypeError('lora must be a non-empty absolute path string')
      }
      if (!path.isAbsolute(params.lora)) {
        throw new TypeError(`lora must be an absolute path (got: ${params.lora})`)
      }
    }
    Fails fast with a clear error and is consistent with how files.model is validated.
  • Acceptable alternative: leave the validation for a follow-up but update the JSDoc in index.js and the TS doc in index.d.ts to explicitly state the silent-no-op behaviour — something like "If the file cannot be loaded, generation proceeds without the LoRA and no error is raised; callers must verify the file exists and is a valid adapter before calling run()."

Either is fine — the goal is that callers aren't surprised by a working-looking call that didn't actually apply anything.

2. Regenerate test/mobile/integration.auto.cjs

The new test/integration/lora-bridge.test.js is missing from the auto-generated mobile wrapper. It's labelled "AUTO-GENERATED FILE. Run npm run test:mobile:generate to update." and the repo's own test:integration:generate script re-runs that generator, so the tree currently has drift.

Since the new test has skip = isMobile || noGpu, this isn't a functional problem on mobile, but any "no uncommitted changes after generate" check in CI will flag it.

Please either:

  • Run npm run test:mobile:generate and include the regenerated file in this PR, or
  • Confirm with CI (and mention it in a comment) that the drift is tolerated and the next PR will pick it up.

Non-blocking follow-up material (flagging so it doesn't get lost):

  • The surface is single-LoRA, hardcoded multiplier = 1.0f, no is_high_noise exposure. docs/architecture.md already advertises "multiple simultaneous LoRAs with configurable weights", so the JS layer now lags the doc — worth a follow-up ticket (array of { path, multiplier?, is_high_noise? }, or reuse the upstream <lora:path:mul> prompt syntax).
  • Foot-gun if ever extended: a second paths.push_back in prepareLoras will reallocate the vector and invalidate every previously-captured items[i].path. Safe today because there's exactly one element, but a prepared.paths.reserve(n) and/or a short comment there would future-proof it.
  • The integration test asserts only "valid PNG produced". Combined with the silent-no-op above, it would pass even if LoRA loading failed. A cheap upgrade is a second generation with the same seed but without lora, then t.ok(!bytesEqual(a, b)). Matches the PR description's "bridge forwarding" intent a lot more tightly.
  • Minor nit: "lora" is placed under the "Prompt" section in SdGenHandlers.cpp, and the unit test fixture is SdGenHandlers_Prompt. A dedicated "LoRA" section + SdGenHandlers_Lora fixture will age better.

Comment thread packages/lib-infer-diffusion/test/mobile/integration.auto.cjs Fixed
@gianni-cor

Copy link
Copy Markdown
Contributor

/review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants