Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Add GPT Vision Models #161

Open
sebiweise opened this issue Dec 19, 2023 · 1 comment
Open

[Feature] Add GPT Vision Models #161

sebiweise opened this issue Dec 19, 2023 · 1 comment
Labels
enhancement New feature or request

Comments

@sebiweise
Copy link
Collaborator

sebiweise commented Dec 19, 2023

Describe the bug
Add GPT 4 Vision https://platform.openai.com/docs/guides/vision

@sebiweise sebiweise changed the title [Feature]Add GPT Vision Models [Feature] Add GPT Vision Models Dec 19, 2023
@sebiweise sebiweise added the enhancement New feature or request label Dec 19, 2023
@johnnymo87
Copy link
Contributor

Any opinions on what this feature should look like?

I imagine that we agree that if the gpt-4-vision model is selected, we show an "upload images" icon on the left in the message bar.

But beyond that?

The API accepts both image URLs and base64 encoded images. Should we present this choice to the user?

The API permits that the image's detail be set to low/high/auto. Should we present this choice to the user?

If the user uploads an image (rather than provides a URL), should we use supabase storage (S3-equivalent) to hold onto it? The gpt-4-vision docs recommend ...

For long running conversations, we suggest passing images via URL's instead of base64. The latency of the model can also be improved by downsizing your images ahead of time to be less than the maximum size they are expected them to be. For low res mode, we expect a 512px x 512px image. For high res mode, the short side of the image should be less than 768px and the long side should be less than 2,000px.

Just glancing at supabase storage, it looks like it could offer both URLs and resizing, which would be advantageous in long running conversations.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants