WIP implementation of unique hash keys for models

[lstein]

What type of PR is this? (check all applicable)

• Refactor
• Feature

Have you discussed this change with the InvokeAI team?

• Yes
• No, because:

Have you updated all relevant documentation?

• Yes
• No

Description

This is a work in progress that assigns models a unique hash based on their contents. It uses the imohash module (https://pypi.org/project/imohash/) which generates unique fingerprints from very large files rapidly by sampling the beginning, middle and end of the file. The model hashing works both for single files (e.g. checkpoints) and diffusers-style directories. Because the config files in diffusers contain things like local file paths and the diffusers version, these files are not included in the directory hashing algorithm.

The model hash is now part of the base config for each model, as returned by ModelManager.list_models(). You can get a model using its hash by making a new ModelManager.get_model_by_hash() call.

info = model_manager.list_model('sd-1','main','stable-diffusion-v1-5')
model = model_manager.get_model_by_hash(info['hash'])

This code is intended as an intermediate step towards replacing the composite models.yaml keys (base/type/name) with the unique hash. Right now the models dictionary is still indexed using base/type/name, which allows the same model to go in twice under different names and is sensitive to the user changing the model name or type. However, this PR does the additional step of checking the model hash as a new model is added and raising an exception if a model with the same hash exists.

Related Tickets & Documents

• Related Issue [enhancement]: model config storage & data structure #4043 Refactor Model Management #4071

QA Instructions, Screenshots, Recordings

You will need to pip install imohash to test this PR.

Added/updated tests?

• Yes
• No

Will add tests if this WIP advances.

[lstein]

superseded