perf(multimodal): optimize payload handling with zero-copy

[ppraneth]

Summary

This PR optimizes the handling of large multimodal payloads (images, audio) by replacing Vec<u8> with bytes::Bytes throughout the multimodal crates. This change leverages atomic reference counting to enable true zero-copy cloning during crucial request-building phases, eliminating significant memory copying overhead.

Key Changes

• llm-multimodal

  • Updated ImageFrame and MultiModalTensor to store data as bytes::Bytes.
  • Refactored MediaConnector to decode images directly from Bytes without cloning.

Performance Impact

Benchmarks demonstrate a massive reduction in processing time for ingestion and payload creation across all payload sizes, moving from microseconds to nanoseconds.

Payload Size	Metric	Baseline (Vec<u8>)	Optimized (bytes::Bytes)	Improvement
1 MB	Ingestion Clone	~27.2 µs	~69.1 ns	~390× Faster
	Payload Creation	~26.6 µs	~147.6 ns	~180× Faster
5 MB	Ingestion Clone	~416.5 µs	~99.8 ns	~4170× Faster
	Payload Creation	~420.5 µs	~148.0 ns	~2840× Faster
10 MB	Ingestion Clone	~445.6 µs	~63.0 ns	~7000× Faster
	Payload Creation	~447.2 µs	~147.8 ns	~3000× Faster

Checklist

• cargo +nightly fmt passes
• cargo clippy --all-targets --all-features -- -D warnings passes
• (Optional) Documentation updated

Summary by CodeRabbit

• Chores
  • Updated a dependency to enable serialization features.
• Refactor
  • Reworked image/multimodal data representation for more efficient memory handling and made image frame fields accessible.
• Bug Fixes / Error Handling
  • Added a new timeout error and improved error mapping/reporting for media decode and blocking operations.

[coderabbitai]

📝 Walkthrough

Walkthrough

Updated the bytes dependency to enable serde features; added a Timeout(Duration) error variant; refactored image decoding to use ImageReader with a blocking task and Cursor; migrated internal raw byte storage from Arc<Vec<u8>> to bytes::Bytes, made ImageFrame fields public and Clone, and changed MultiModalTensor.data to bytes::Bytes.

Changes

Cohort / File(s)	Summary
Dependency multimodal/Cargo.toml	Changed bytes dependency from bytes = "1.8.0" to bytes = { version = "1.8.0", features = ["serde"] }.
Error Variant multimodal/src/error.rs	Added Timeout(Duration) variant to MediaConnectorError with message media fetch timed out after {0:?}.
Types / Serialization multimodal/src/types.rs	Replaced Arc<Vec<u8>> with bytes::Bytes; made ImageFrame fields pub and derived Clone; changed MultiModalTensor.data from Vec<u8> (and removed serde_bytes) to bytes::Bytes; adjusted constructors and accessors accordingly.
Image decoding / Media multimodal/src/media.rs	Refactored decode_image to decode via ImageReader::with_guessed_format over a Cursor inside a blocking task, store original bytes alongside decoded image, and map blocking/decoding failures to existing media error variants.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Poem

I nibble bytes beneath the moonlit stream,
Cursor in paw, decoding every dream,
Fields unfurled, I show them all with pride,
Timeout watches by the riverside,
A rabbit claps — the bytes and images glide. 🐇✨

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main objective: optimizing payload handling with zero-copy by replacing Vec with bytes::Bytes for atomic reference counting.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch zcopy

---

No actionable comments were generated in the recent review. 🎉

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[gemini-code-assist]

Summary of Changes

Hello @ppraneth, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a significant performance enhancement by optimizing how large multimodal payloads, such as images and audio, are handled within the system. By transitioning from standard byte vectors to bytes::Bytes, the change leverages atomic reference counting to facilitate zero-copy operations during critical data processing stages. This architectural shift drastically reduces memory allocations and copying, leading to substantial improvements in processing speed and overall efficiency for multimodal data.

Highlights

• Zero-Copy Payload Handling: Replaced Vec<u8> with bytes::Bytes across multimodal crates to enable true zero-copy cloning for large payloads, significantly reducing memory copying overhead.
• Performance Optimization: Achieved massive reductions in processing time for ingestion and payload creation, with improvements ranging from 180x to 7000x faster, as demonstrated by benchmarks.
• Refactored Image Decoding: Updated the MediaConnector to decode images directly from bytes::Bytes using image::ImageReader and std::io::Cursor, eliminating intermediate data conversions.
• Error Handling Enhancement: Introduced a new ImageDecode error variant for more specific handling of image decoding failures.

Changelog

• multimodal/Cargo.toml
  • Enabled the serde feature for the bytes crate dependency.
• multimodal/src/error.rs
  • Added a new ImageDecode error variant to MediaConnectorError for specific image decoding failures.
• multimodal/src/media.rs
  • Removed the image::DynamicImage import.
  • Refactored MediaConnector::decode_image to use bytes::Bytes directly with std::io::Cursor and image::ImageReader for decoding.
  • Updated error handling in decode_image to use the new ImageDecode error variant.
• multimodal/src/types.rs
  • Derived Clone for the ImageFrame struct.
  • Changed the raw_bytes field in ImageFrame from Arc<Vec<u8>> to bytes::Bytes.
  • Updated the raw_bytes() getter method in ImageFrame to return a slice from bytes::Bytes.
  • Changed the data field in MultiModalTensor from Vec<u8> to bytes::Bytes and removed the #[serde(with = "serde_bytes")] attribute.

Activity

• No activity has occurred on this pull request yet.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request significantly optimizes multimodal payload handling by switching from Vec<u8> to bytes::Bytes, which enables zero-copy cloning and yields impressive performance gains as shown in the benchmarks. The changes are well-implemented across the llm-multimodal crate. I have a couple of suggestions to further improve the error handling logic for image decoding to make it more robust and idiomatic, which align with the repository's guidelines on graceful error handling and code clarity.