Skip to content

fix(autoscaling): extract load model from Route() and use as well when doing autoscale - #9270

Merged
mudler merged 1 commit into
masterfrom
fix/autoscaling
Apr 8, 2026
Merged

fix(autoscaling): extract load model from Route() and use as well when doing autoscale#9270
mudler merged 1 commit into
masterfrom
fix/autoscaling

Conversation

@mudler

@mudler mudler commented Apr 7, 2026

Copy link
Copy Markdown
Owner

This pull request adds support for storing and retrieving model load metadata to enable the reconciler to reliably replicate model loads during scale-up. It introduces new methods and fields for saving backend type and serialized model options, and refactors the model scheduling logic to use this metadata when loading models on new nodes. The changes also include updates to interfaces, implementations, and test fakes.

Model load metadata storage and retrieval:

  • Added SetNodeModelLoadInfo and GetModelLoadInfo methods to the ModelRouter interface and its implementations, allowing the backend type and serialized model options to be saved and fetched for each loaded model. [1] [2]
  • Extended the NodeModel struct with BackendType and ModelOptsBlob fields to persist this metadata in the database.

Model scheduling and loading refactor:

  • Introduced the scheduleAndLoad helper method in SmartRouter to centralize the logic for picking a node, staging files, loading the model, and recording metadata.
  • Refactored both Route and ScheduleAndLoadModel to use stored metadata for consistent model loading and replica creation, improving reliability during reconciler-driven scale-ups. [1] [2]

Test updates:

  • Updated test fakes in model_router_test.go and router_test.go to implement the new interface methods, ensuring test coverage for the new logic. [1] [2]

Miscellaneous:

  • Added necessary imports for fmt in test files.

Notes for Reviewers

Signed commits

  • Yes, I signed my commits.

…n doing autoscale

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
@mudler mudler changed the title fix(autoscaling): extract load model from Route() and use as well whe… fix(autoscaling): extract load model from Route() and use as well whe doing autoscale Apr 7, 2026
@mudler mudler changed the title fix(autoscaling): extract load model from Route() and use as well whe doing autoscale fix(autoscaling): extract load model from Route() and use as well when doing autoscale Apr 7, 2026
@mudler
mudler merged commit 154fa00 into master Apr 8, 2026
34 checks passed
@mudler
mudler deleted the fix/autoscaling branch April 8, 2026 06:27
@mudler mudler added the bug Something isn't working label Apr 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant