Reimplement model config storage backend, download and install modules

docs/contributing/contribution_guides/development.md

@@ -14,6 +14,7 @@ Once you're setup, for more information, you can review the documentation specif
 * #### [InvokeAI Architecure](../ARCHITECTURE.md)
 * #### [Frontend Documentation](./contributingToFrontend.md)
 * #### [Node Documentation](../INVOCATIONS.md)
+* #### [InvokeAI Model Manager](../MODEL_MANAGER.md)
 * #### [Local Development](../LOCAL_DEVELOPMENT.md)
 
 

docs/features/CONFIGURATION.md

@@ -207,11 +207,8 @@ if INVOKEAI_ROOT is `/home/fred/invokeai` and the path is
 
 | Setting  | Default Value  |  Description |
 |----------|----------------|--------------|
-| `autoimport_dir` | `autoimport/main`     | At startup time, read and import any main model files found in this directory |
-| `lora_dir` | `autoimport/lora`     | At startup time, read and import any LoRA/LyCORIS models found in this directory |
-| `embedding_dir` | `autoimport/embedding`  | At startup time, read and import any textual inversion (embedding) models found in this directory |
-| `controlnet_dir` | `autoimport/controlnet`  | At startup time, read and import any ControlNet models found in this directory |
-| `conf_path` | `configs/models.yaml`  | Location of the `models.yaml` model configuration file |
+| `autoimport_dir` | `autoimport/main`     | At startup time, read and import any main model files found in this directory (not recommended)|
+| `model_config_db` | `auto`  | Location of the model configuration database. Specify `auto` to use the main invokeai.db database, or specify a `.yaml` or `.db` file to store the data externally.|
 | `models_dir` | `models`  | Location of the directory containing models installed by InvokeAI's model manager |
 | `legacy_conf_dir` | `configs/stable-diffusion`  | Location of the directory containing the .yaml configuration files for legacy checkpoint models |
 | `db_dir` | `databases`  | Location of the directory containing InvokeAI's image, schema and session database |
@@ -234,6 +231,18 @@ Paths:
 #  controlnet_dir: null
 ```
 
+### Model Cache
+
+These options control the size of various caches that InvokeAI uses
+during the model loading and conversion process. All units are in GB
+
+| Setting  | Default Value  |  Description |
+|----------|----------------|--------------|
+| `disk` | `20.0` | Before loading a model into memory, InvokeAI converts .ckpt and .safetensors models into diffusers format and saves them to disk. This option controls the maximum size of the directory in which these converted models are stored. If set to zero, then only the most recently-used model will be cached. |
+| `ram` | `6.0` | After loading a model from disk, it is kept in system RAM until it is needed again. This option controls how much RAM is set aside for this purpose. Larger amounts allow more models to reside in RAM and for InvokeAI to quickly switch between them. |
+| `vram` | `0.25` | This allows smaller models to remain in VRAM, speeding up execution modestly. It should be a small number. |
+
+
 ### Logging
 
 These settings control the information, warning, and debugging

docs/installation/050_INSTALLING_MODELS.md

@@ -123,11 +123,20 @@ installation. Examples:
 # (list all controlnet models)
 invokeai-model-install --list controlnet
 
-# (install the model at the indicated URL)
+# (install the diffusers model using its hugging face repo_id)
+invokeai-model-install --add stabilityai/stable-diffusion-xl-base-1.0
+
+# (install a diffusers model that lives in a subfolder)
+invokeai-model-install --add stabilityai/stable-diffusion-xl-base-1.0:vae
+
+# (install the checkpoint model at the indicated URL)
 invokeai-model-install --add https://civitai.com/api/download/models/128713
 
-# (delete the named model)
-invokeai-model-install --delete sd-1/main/analog-diffusion
+# (delete the named model if its name is unique)
+invokeai-model-install --delete analog-diffusion
+
+# (delete the named model using its fully qualified name)
+invokeai-model-install --delete sd-1/main/test_model
 ```
 
 ### Installation via the Web GUI
@@ -141,6 +150,24 @@ left-hand panel) and navigate to *Import Models*
 wish to install. You may use a URL, HuggingFace repo id, or a path on
 your local disk.
 
+There is special scanning for CivitAI URLs which lets
+you cut-and-paste either the URL for a CivitAI model page
+(e.g. https://civitai.com/models/12345), or the direct download link
+for a model (e.g. https://civitai.com/api/download/models/12345).
+
+If the desired model is a HuggingFace diffusers model that is located
+in a subfolder of the repository (e.g. vae), then append the subfolder
+to the end of the repo_id like this:
+
+```
+# a VAE model located in subfolder "vae"
+stabilityai/stable-diffusion-xl-base-1.0:vae
+
+# version 2 of the model located in subfolder "v2"
+monster-labs/control_v1p_sd15_qrcode_monster:v2
+
+```
+
 3. Alternatively, the *Scan for Models* button allows you to paste in
 the path to a folder somewhere on your machine. It will be scanned for
 importable models and prompt you to add the ones of your choice.

invokeai/app/api/dependencies.py

@@ -19,14 +19,17 @@
 from invokeai.version.invokeai_version import __version__
 
 from ..services.default_graphs import create_system_graphs
+from ..services.download_manager import DownloadQueueService
 from ..services.graph import GraphExecutionState, LibraryGraph
 from ..services.image_file_storage import DiskImageFileStorage
 from ..services.invocation_queue import MemoryInvocationQueue
 from ..services.invocation_services import InvocationServices
 from ..services.invocation_stats import InvocationStatsService
 from ..services.invoker import Invoker
 from ..services.latent_storage import DiskLatentsStorage, ForwardCacheLatentsStorage
-from ..services.model_manager_service import ModelManagerService
+from ..services.model_install_service import ModelInstallService
+from ..services.model_loader_service import ModelLoadService
+from ..services.model_record_service import ModelRecordServiceBase
 from ..services.processor import DefaultInvocationProcessor
 from ..services.sqlite import SqliteItemStorage
 from ..services.thread import lock
@@ -127,8 +130,12 @@ def initialize(config: InvokeAIAppConfig, event_handler_id: int, logger: Logger
             )
         )
 
+        download_queue = DownloadQueueService(event_bus=events)
+        model_record_store = ModelRecordServiceBase.open(config, conn=db_conn, lock=lock)
+        model_loader = ModelLoadService(config, model_record_store)
+        model_installer = ModelInstallService(config, queue=download_queue, store=model_record_store, event_bus=events)
+
         services = InvocationServices(
-            model_manager=ModelManagerService(config, logger),
             events=events,
             latents=latents,
             images=images,
@@ -141,6 +148,10 @@ def initialize(config: InvokeAIAppConfig, event_handler_id: int, logger: Logger
             configuration=config,
             performance_statistics=InvocationStatsService(graph_execution_manager),
             logger=logger,
+            download_queue=download_queue,
+            model_record_store=model_record_store,
+            model_loader=model_loader,
+            model_installer=model_installer,
             session_queue=SqliteSessionQueue(conn=db_conn, lock=lock),
             session_processor=DefaultSessionProcessor(),
             invocation_cache=MemoryInvocationCache(max_cache_size=config.node_cache_size),