Support for LLaVA

[varshith15]

• Single Machine
• Sharding
• Generate function integration
• Clean up code (deduplication)

[AlexCheema]

Looks good so far!

Assigned you on the bounty https://docs.google.com/spreadsheets/d/1cTCpTIp48UnnIvHeLEUNg1iMy_Q6lRybgECSFCoVJpE/edit

[varshith15]

hey @AlexCheema

• vision model is fully on the first shard, language model is multi sharded
• integrated llava into generate function as well
• only the main.py integration is left which i feel like should be a seperate PR (could be breaking changes)
• There are no breaking changes to the current flow.
  Please review and merge

[AlexCheema]

We should merge #87 before this as there's some relevant changes.
@varshith15 you can merge that branch already if you want and then I will merge yours as soon as the other is merged.

[AlexCheema]

With changes from #87, we should try to re-use as much as possible from mlx_lm, there's a lot of copied code when it should only be necessary to copy 1 or 2 classes from there and modify them.

[AlexCheema]

Is the test_sharded_llava.py test passing for you?
I thought we'd need to load an mlx-converted model, but it looks like this is the original ordinary model from HF llava-hf/llava-1.5-7b-hf

[AlexCheema]

Is the test_sharded_llava.py test passing for you? I thought we'd need to load an mlx-converted model, but it looks like this is the original ordinary model from HF llava-hf/llava-1.5-7b-hf

NVM this works fine :)

[varshith15]

this isnt needed right @AlexCheema ?

[AlexCheema]

It is with the new convention, I think

[varshith15]

also thanks thanks for pushing the other changes @AlexCheema , they make a lot of sense

[AlexCheema]

It would be great to hook this up to chatgpt_api.py do you want to give that a go @varshith15 ? i.e. allow the user to give an image as input, similar to how chatgpt vision api works.

[varshith15]

@AlexCheema done PRM
the logic to use the pixel_values only in the first pass fits in perfectly with the send_prompt, send_tensor idea (really cool idea)
i've verified, its backward compatible as well
only change left is the UI one, I am little occupied so can't pick that up rn

[AlexCheema]

@AlexCheema done PRM the logic to use the pixel_values only in the first pass fits in perfectly with the send_prompt, send_tensor idea (really cool idea) i've verified, its backward compatible as well only change left is the UI one, I am little occupied so can't pick that up rn

Woah! Incredible work. You even figured out how to generate the grpc service. I’m catching a flight now but will be able to look properly once I arrive.

I can fix up a small UI.

email me at [email protected] for the bounty.

really awesome work, you rock @varshith15

[AlexCheema]

Changed a few things, added support for image upload to tiny chat.
Awesome work @varshith15 please email me [email protected] for the bounty reward!