-
Notifications
You must be signed in to change notification settings - Fork 14
Basic multimodal embedding support #349
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR adds multimodal embedding support to LMI by upstreaming utilities from paper-qa and implementing VertexAI multimodal embedding testing. The changes include new byte/string conversion utilities, image validation functions, token estimation for multimodal content, and comprehensive test coverage for multimodal embeddings.
- Adds utility functions for handling binary data and image validation
- Implements multimodal token estimation for rate limiting
- Adds comprehensive testing for VertexAI multimodal embeddings with GCS integration
Reviewed Changes
Copilot reviewed 8 out of 10 changed files in this pull request and generated 3 comments.
Show a summary per file
File | Description |
---|---|
packages/lmi/src/lmi/utils.py | Added utility functions for byte/string conversion, image validation, and image encoding |
packages/lmi/src/lmi/embeddings.py | Enhanced token estimation to support multimodal content and updated rate limiting |
packages/lmi/tests/test_utils.py | Added comprehensive tests for byte/string conversion utilities |
packages/lmi/tests/test_embeddings.py | Added multimodal embedding tests and token estimation tests |
packages/lmi/tests/conftest.py | Added fixtures for PNG image testing and GCS integration |
packages/lmi/pyproject.toml | Added image optional dependency with pillow requirement |
pyproject.toml | Added mypy ignore for google.cloud.storage module |
.github/workflows/tests.yml | Added GCP authentication for CI testing |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
This PR:
gather_evidence
paper-qa#1046 to LMI