Reimplement model config storage backend, download and install modules#4252

Closed

lstein wants to merge 154 commits intomainfrom

lstein/model-manager-refactor

Commits on Aug 12, 2023

define model configuration classes
Lincoln Stein
committed
blackify
Lincoln Stein
committed
add ABC for config storage
Lincoln Stein
committed
rename ModelConfig to ModelConfigFactory
Lincoln Stein
committed

Commits on Aug 13, 2023

add YAML file storage backend
Lincoln Stein
committed
add class docstring and blackify
Lincoln Stein
committed
Merge branch 'main' into lstein/model-manager-refactor
lstein
authored
change paths to str to make json serializable
Lincoln Stein
committed
fix test to work with string paths
Lincoln Stein
committed
add SQL backend
Lincoln Stein
committed
Merge branch 'main' into lstein/model-manager-refactor
lstein
authored
change model field name "hash" to "id"
Lincoln Stein
committed
Merge branch 'lstein/model-manager-refactor' of github.com:invoke-ai/InvokeAI into lstein/model-manager-refactor
Lincoln Stein
committed
added ability to force config class returned by make_config()
Lincoln Stein
committed

Commits on Aug 15, 2023

rename modules
Lincoln Stein
committed
start installer module
Lincoln Stein
committed
module skeleton written
Lincoln Stein
committed

Commits on Aug 17, 2023

partial rewrite of checkpoint template creator
Lincoln Stein
committed

Commits on Aug 20, 2023

Merge branch 'main' into lstein/model-manager-refactor
Lincoln Stein
committed
fix flake8 errors
Lincoln Stein
committed

Commits on Aug 21, 2023

added main templates
Lincoln Stein
committed

Commits on Aug 23, 2023

reimplement and clean up probe class
Lincoln Stein
committed
install ABC written
Lincoln Stein
committed
added install module
Lincoln Stein
committed
merge with main; resolve conflicts
Lincoln Stein
committed
blackify
Lincoln Stein
committed

Commits on Aug 24, 2023

make ModelSearch pydantic
Lincoln Stein
committed

Commits on Aug 30, 2023

Merge branch 'main' into lstein/model-manager-refactor
Lincoln Stein
committed
add ABC for download manager
Lincoln Stein
committed

Commits on Sep 2, 2023

download of individual files working
Lincoln Stein
committed

Commits on Sep 4, 2023

added download manager service and began repo_id download
Lincoln Stein
committed
merge and resolve conflicts
Lincoln Stein
committed

Commits on Sep 5, 2023

chore: black
psychedelicious
committed
threaded repo_id download working; error conditions not tested
Lincoln Stein
committed
resolve merge conflicts
Lincoln Stein
committed

Commits on Sep 6, 2023

add unit tests for queued model download
Lincoln Stein
committed
add download manager to invoke services
Lincoln Stein
committed
remove unecessary HTTP probe for repo_id model component sizes
Lincoln Stein
committed
fix progress reporting for repo_ids
Lincoln Stein
committed

Commits on Sep 7, 2023

Commits on Sep 8, 2023

add checks for malformed URLs and malicious content dispositions
Lincoln Stein
committed
blackify
Lincoln Stein
committed

Commits on Sep 9, 2023

wire together download and install; now need to write install events
Lincoln Stein
committed
install of repo_ids records author, tags and license
Lincoln Stein
committed
make download manager optional in InvokeAIServices during development
Lincoln Stein
committed

Commits on Sep 10, 2023

incorporate civitai metadata into model config
Lincoln Stein
committed
fix flake8 warnings
Lincoln Stein
committed
increase download chunksize for better speed
Lincoln Stein
committed
Merge branch 'main' into lstein/model-manager-refactor
Lincoln Stein
committed
make it possible to pause/resume repo_id downloads
Lincoln Stein
committed

Commits on Sep 11, 2023

loading works -- web app broken
Lincoln Stein
committed
loader working
Lincoln Stein
committed

Commits on Sep 12, 2023

pytests mostly working; model_manager_service needs rewriting
lstein
committed

Commits on Sep 13, 2023

refactor create_download_job; override probe info in install call
lstein
committed

Commits on Sep 14, 2023

Commits on Sep 15, 2023

model_manager_service now mostly type correct
lstein
committed
blackify
lstein
committed
services rewritten; starting work on routes
Lincoln Stein
committed

Commits on Sep 16, 2023

list_models() API call now working
Lincoln Stein
committed
Merge branch 'main' into lstein/model-manager-refactor
Lincoln Stein
committed
blackify and isort
Lincoln Stein
committed
fix more isort issues
Lincoln Stein
committed
update_model and delete_model working; convert is WIP
Lincoln Stein
committed
loading and conversions of checkpoints working
Lincoln Stein
committed
all methods in router API now tested and working
Lincoln Stein
committed

Commits on Sep 17, 2023

import_model API now working
Lincoln Stein
committed
add logs to confirm that event info is being sent to bus
Lincoln Stein
committed
add install job control to web API
Lincoln Stein
committed
add models.yaml conversion script
Lincoln Stein
committed
attempt to fix flake8 lint errors
Lincoln Stein
committed
fix models.yaml version assertion error in pytests
Lincoln Stein
committed

Commits on Sep 19, 2023

fasthash produces same results on windows & linux
lstein
committed
Merge branch 'main' into lstein/model-manager-refactor
Lincoln Stein
committed
blackify
Lincoln Stein
committed
move incorrectly placed models into correct directory at startup time
Lincoln Stein
committed

Commits on Sep 21, 2023

TUI installer more or less working
lstein
committed
preserve description in metadata when installing a starter model
Lincoln Stein
committed
TUI installer functional; minor cosmetic work needed
Lincoln Stein
committed
remove debug statement
Lincoln Stein
committed
add/delete from command line working; training words downloaded
Lincoln Stein
committed

Commits on Sep 22, 2023

fix install of models with relative paths
Lincoln Stein
committed

Commits on Sep 23, 2023

configure script now working
Lincoln Stein
committed
correctly download the selected version of a civitai model
Lincoln Stein
committed
resolve conflicts with ip-adapter change
Lincoln Stein
committed
automatically convert models.yaml to new format
Lincoln Stein
committed

Commits on Sep 24, 2023

allow priority to be set at install job submission time
Lincoln Stein
committed
resolve conflicts with get_logger() code changes from main
Lincoln Stein
committed

Commits on Sep 25, 2023

merge with main & resolve conflicts
Lincoln Stein
committed
added `cancel_all` and `prune` model install operations to router API
Lincoln Stein
committed

Commits on Sep 26, 2023

add tests for model installation events
Lincoln Stein
committed
Merge branch 'main' into lstein/model-manager-refactor
lstein
authored

Commits on Sep 28, 2023

resolve merge conflicts
Lincoln Stein
committed
reorder pytests to prevent fixture race condition
Lincoln Stein
committed

Commits on Sep 29, 2023

fix migrate script and type mismatches in probe, config and loader
Lincoln Stein
committed
fix one more type mismatch in probe module
Lincoln Stein
committed
remove unused code from invokeai.backend.model_manager.storage.yaml
Lincoln Stein
committed
add README explaining reorg of tests directory
Lincoln Stein
committed
almost all type mismatches fixed
Lincoln Stein
committed

Commits on Sep 30, 2023

more type mismatch fixes
Lincoln Stein
committed
make model merge script work with new model manager
Lincoln Stein
committed
speed up model scanning at startup
Lincoln Stein
committed
do not ignore keyboard interrupt while scanning models
Lincoln Stein
committed

Commits on Oct 3, 2023

merge with main
Lincoln Stein
committed
make merge script read invokeai.yaml when default root passed
Lincoln Stein
committed
make textual inversion training work with new model manager
Lincoln Stein
committed
prevent crash on windows due to lack of os.pathconf call
Lincoln Stein
committed
replace config.ram_cache_size with config.ram and similarly for vram
Lincoln Stein
committed

Commits on Oct 4, 2023

Commits on Oct 5, 2023

refactor model_manager_service.py into small functional modules
Lincoln Stein
committed
fix reorganized module dependencies
Lincoln Stein
committed

Commits on Oct 6, 2023

refactor services
Lincoln Stein
committed

Commits on Oct 7, 2023

allow user to select main database or external file for model record/config db
Lincoln Stein
committed
make autoimport directory optional, defaulting to inactive
Lincoln Stein
committed
fix conversion call
Lincoln Stein
committed
merge with main
Lincoln Stein
committed
restore printing of version when invokeai-web and invokeai called with --version
Lincoln Stein
committed
support clipvision image encoder downloading
Lincoln Stein
committed

Commits on Oct 8, 2023

add support for repo_id subfolders
Lincoln Stein
committed
refactor download queue jobs
Lincoln Stein
committed
fix some type mismatches introduces by reorg
Lincoln Stein
committed
port support for AutoencoderTiny models
Lincoln Stein
committed

Commits on Oct 9, 2023

Update invokeai/app/invocations/model.py

lstein
and
RyanJDick
authored
address all PR 4252 comments from ryan through October 5
Lincoln Stein
committed
Merge branch 'lstein/model-manager-refactor' of github.com:invoke-ai/InvokeAI into lstein/model-manager-refactor
Lincoln Stein
committed
merge with main
Lincoln Stein
committed
improve selection of huggingface repo id files to download
Lincoln Stein
committed
refactor installer class hierarchy
Lincoln Stein
committed
merge with main
Lincoln Stein
committed
fix broken import
Lincoln Stein
committed

Commits on Oct 10, 2023

fix issues with module import order breaking pytest node tests
Lincoln Stein
committed
add documentation for model record service and loader
Lincoln Stein
committed

Commits on Oct 11, 2023

first draft of documentation finished
Lincoln Stein
committed

Commits on Oct 12, 2023

add back source URL validation to download job hierarchy
Lincoln Stein
committed
restore CLI to broken state
Lincoln Stein
committed
Update docs/installation/050_INSTALLING_MODELS.md

lstein
and
RyanJDick
authored
Update invokeai/app/services/model_convert.py

lstein
and
RyanJDick
authored
misc small fixes requested by Ryan
Lincoln Stein
committed
Merge branch 'lstein/model-manager-refactor' of github.com:invoke-ai/InvokeAI into lstein/model-manager-refactor
Lincoln Stein
committed
clean up model downloader status locking to avoid race conditions
Lincoln Stein
committed
remove download queue change_priority() calls completely
Lincoln Stein
committed
prevent potential infinite recursion on exceptions raised by event handlers
Lincoln Stein
committed