Phi 3 vision ISQ support (#410)

* ISQ support for phi3v * Document it
EricLBuehler · Jun 8, 2024 · f257423 · f257423
1 parent ca27588
commit f257423
Show file tree

Hide file tree

Showing 2 changed files with 2 additions and 2 deletions.
diff --git a/docs/PHI3V.md b/docs/PHI3V.md
@@ -1,6 +1,6 @@
 # Phi 3 Vision Support: `microsoft/Phi-3-vision-128k-instruct`
 
-The Phi 3 Vision Model has support in the Rust, Python, and HTTP APIs.
+The Phi 3 Vision Model has support in the Rust, Python, and HTTP APIs. The Phi 3 Vision Model supports ISQ for increased performance.
 
 > Note: The Phi 3 Vision model works best with one image although it is supported to send multiple images.
 

diff --git a/mistralrs-core/src/vision_models/phi3.rs b/mistralrs-core/src/vision_models/phi3.rs
@@ -773,7 +773,7 @@ impl Model {
             cfg,
             embed_tokens.clone(),
             &cfg.embd_layer,
-            vb_m.pp("vision_embed_tokens"),
+            mapper.set_nm_device(vb_m.pp("vision_embed_tokens"), false),
         )?;
         let mut layers = Vec::with_capacity(cfg.num_hidden_layers);
         let vb_l = vb_m.pp("layers");