Skip to content

feat(rl): add LoRA and replay support with inference/trainer startup fixes - #3735

Draft
biswapanda wants to merge 3 commits into
PrimeIntellect-ai:mainfrom
biswapanda:feat/shared-rl-inference-support
Draft

biswapanda wants to merge 3 commits into
PrimeIntellect-ai:mainfrom
biswapanda:feat/shared-rl-inference-support

Conversation

@biswapanda

@biswapanda biswapanda commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Allow explicit renderer-based evaluation without changing the default chat evaluation path.
  • Add a backend-neutral LoRA administration hook and propagate an optional returned adapter model through filesystem weight updates before notifying observers.
  • Preserve native LoRA loading behavior and its model name.
  • Require complete routed-expert branches when replay is configured, reject incomplete rollouts early, and stop after the configured rejection budget.
  • Allow explicitly trusted custom VLM checkpoints through the shared trainer configuration.
  • Check whether NIXL is configured before accessing NIXL-specific model fields, allowing aggregated workers to initialize with the tested vLLM RC.
  • Resolve the image adapter only when a batch contains multimodal references, allowing text-only training without an image adapter while preserving image validation and caching.

Why

These are shared RL interfaces and validation rules, not backend-specific discovery or worker-management behavior. Evaluation needs an explicit request-construction mode; filesystem weight updates need a backend-neutral way to report the active adapter model; replay requires complete branch metadata before training; and explicitly trusted custom VLM checkpoints must pass configuration validation.

Aggregated workers should not inspect removed NIXL-only configuration fields when no NIXL connector is active. Text-only composite-model training should not require an image adapter at startup; image-bearing batches still resolve and validate the adapter before materialization.

Validation

  • 162 focused inference, client, rollout, algorithm, and configuration tests passed against this branch after the two startup corrections were added.
  • Five source-block checks passed for deferred adapter selection, text-only execution, adapter caching, and rejected unsupported/misconfigured image batches. These are source-block checks, not full trainer tests.
  • The watcher regression passed in the SSH dev-pod runtime with GPU visibility disabled.
  • The composed shared-plus-Dynamo stack passed 197 focused tests. Its production code exactly matches the original integration merged with current main; only a native LoRA regression test was added.
  • Ruff and diff whitespace checks passed.
  • The broader recipe-loading suite was not green in the local environment: 38 cases failed with environment-plugin/configuration issues. Those cases were excluded from the focused result above.
  • The composed Nemotron + Dynamo stack completed three AutomationBench optimizer updates with filesystem LoRA, TP4 inference, strict routed-expert replay, confirmed v0-v3 deployed adapters, and all process exit codes 0. It used these exact two production corrections, additional Nemotron conversion/model support, and feat(dynamo): support python based dynamo vLLM workers administration #3629; this is composed-stack evidence, not standalone native acceptance of this PR head.
  • That regression had 50% truncation in the first batch and token-level mismatch KL maxima up to 0.784544. It verifies loop/adapter/replay plumbing, not convergence or universally low KL. No new GPU run was launched for this publication.

Integration order

This PR extracts backend-independent support from #3629. Both target upstream main; merge this prerequisite first, then #3629.

@biswapanda biswapanda changed the title feat(rl): add shared replay validation and adapter administration hooks feat(rl): add shared LoRA and replay support with inference/trainer startup fixes Oct 1, 2026
@biswapanda biswapanda changed the title feat(rl): add shared LoRA and replay support with inference/trainer startup fixes feat(rl): add LoRA and replay support with inference/trainer startup fixes Oct 1, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant