Skip to content

feat: multimodal endpoint for image to text #144

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 77 commits into
base: main
Choose a base branch
from
Open

feat: multimodal endpoint for image to text #144

wants to merge 77 commits into from

Conversation

blefo
Copy link
Member

@blefo blefo commented Aug 25, 2025

This PR adds multimodal support to the chat completion endpoint

  • The chat completion endpoint now accepts images in base64 format, enabling text + image model inputs.
  • The request body format aligns with the OpenAI client specification.
  • Web search is updated to support both web search and multimodal inputs.

jcabrero
jcabrero reviewed Aug 26, 2025
View reviewed changes
jcabrero
jcabrero reviewed Aug 26, 2025
View reviewed changes
jcabrero
jcabrero reviewed Aug 26, 2025
View reviewed changes
jcabrero
jcabrero reviewed Aug 26, 2025
View reviewed changes
jcabrero
jcabrero requested changes Aug 26, 2025
View reviewed changes
Copy link
Member

@jcabrero jcabrero left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You need to check the API Model changes as they break tool calling and make them more similar to what OpenAI support for image_url is.

The PR is overall good 👍 Good job.

blefo added 16 commits August 26, 2025 11:50
@blefo
refactor: update message model structure
2fcce35
@blefo
fix: ruff format
3e3cc56
@blefo
test: enhance chat completion tests with multimodal model integration
2a9669c
@blefo
fix: web search + multimodal with 3 sources
3c62952
@blefo
chore: stop tracking docker/compose/docker-compose.gemma-4b-gpu.ci.ym…
a52ec27 
…l and ignore it
@blefo
refactor: clean up imports in tests
ad4fa11
@blefo
fix: add type ignore for role in Message class
f0e5848
@blefo
refactor: update Message class content type to use new ChatCompletion…
338a5ec 
… content parts
@blefo
feat: add content extractor utility for processing text and image con…
e85a711 
…tent from chat completions
@blefo
refactor: improve web search handling and enhance message context wit…
758d809 
…h user queries
@blefo
refactor: integrate content extraction into user query handling and e…
fc9ea5d 
…nhance system message updates
@blefo
feat: add functions to handle multimodal content and extract the last…
758dc0d 
… user query from messages
@blefo
refactor: remove deprecated image support handler and enhance multimo…
eaf2e96 
…dal completion tests
@blefo
refactor: clean up unused imports in web search and content extractor…
7973642 
… modules
@blefo
feat: implement chat completion tests with image support and error ha…
f9b71cd 
…ndling for multimodal content
@blefo
feat: enhance chat completion tests with rate limit configurations an…
746edbf 
…d improved error handling for multimodal content
blefo added 27 commits August 29, 2025 16:19
@blefo
test#2: remove llama-1b
da0602f
@blefo
fix: update the script for gemma
fccbd1d
@blefo
fix#2
03ca9eb
@blefo
fix: add service startup logs
359f518
@blefo
fix: update gemma ci config
967eace
@blefo
fix: added logs for services
8b5a073
@blefo
fix: gemma config
e9cc0da
@blefo
fix: gemma config
3dcd4e5
@blefo
fix: gemma config
86c8e0a
@blefo
fix: gemma config
cfc2e07
@blefo
fix: gemma config
96f78af
@blefo
fix: gemma config
f1c7b4d
@blefo
fix: gemma config
f4451ca
@blefo
fix: update gemma config
25bea10
@blefo
fix: gemma config
bd8ba99
@blefo
fix: gemma config
eb4dbe0
@blefo
fix: use qwen-2b instead of gemma-4b for ci pipeline
eb3f3de
@blefo
fix: update qwen config
b0f36c6
@blefo
fix: qwen config
7c2b140
@blefo
fix: update qwen config
4375c03
@blefo
fix: config as list
471c7cb
@blefo
fix: qwen config
a842da8
@blefo
fix: avoid parsing error
9115580
@blefo
fix: qwen config format
30fb2c9
@blefo
fix: update config
aa43f24
@blefo
fix: update config
8f6b06e
@blefo
fix: enfore eager
244d14d
jcabrero
jcabrero reviewed Sep 1, 2025
View reviewed changes
packages/nilai-common/src/nilai_common/api_model.py
Comment on lines -11 to -12
class Message(ChatCompletionMessage):
role: Literal["system", "user", "assistant", "tool"] # type: ignore
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

class Message (ChatCompletionMessageParam):
    pass

@jcabrero jcabrero linked an issue Oct 8, 2025 that may be closed by this pull request
Add new models to the catalogue #128
Closed
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jcabrero jcabrero jcabrero requested changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add new models to the catalogue

2 participants

@blefo @jcabrero