Skip to content

fix(google-generativeai,vertexai): image support for Gemini models #3340

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
nirga merged 8 commits into from
Aug 24, 2025
Merged

fix(google-generativeai,vertexai): image support for Gemini models #3340

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
20231b2
feat(google-generativeai,vertexai): Add image support for Gemini models
nirga Aug 23, 2025
535e612
fix(google-generativeai,vertexai): Remove fallback logic for failed i…
nirga Aug 23, 2025
29bbfde
fix: Address PR feedback on config defaults and asyncio.run usage
nirga Aug 23, 2025
144929b
Revert "fix: Address PR feedback on config defaults and asyncio.run u…
nirga Aug 23, 2025
009db80
feat(gemini): Use OpenAI's battle-tested run_async pattern for sync i…
nirga Aug 23, 2025
ef1b148
fix(sample-app): Fix linting issues in image examples
nirga Aug 24, 2025
0c80bc9
fix: Address PR feedback - type conversion and code readability impro…
nirga Aug 24, 2025
544322e
Merge branch 'main' into gemini-image-support
nirga Aug 24, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions ...ntation-google-generativeai/opentelemetry/instrumentation/google_generativeai/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
emit_message_events,
)
from opentelemetry.instrumentation.google_generativeai.span_utils import (
set_input_attributes,
set_input_attributes_sync,
set_model_request_attributes,
set_model_response_attributes,
set_response_attributes,
Expand Down Expand Up @@ -103,7 +103,7 @@ def _handle_request(span, args, kwargs, llm_model, event_logger):
if should_emit_events() and event_logger:
emit_message_events(args, kwargs, event_logger)
else:
set_input_attributes(span, args, kwargs, llm_model)
set_input_attributes_sync(span, args, kwargs, llm_model)

set_model_request_attributes(span, kwargs, llm_model)

Expand Down Expand Up @@ -249,10 +249,12 @@ def _wrap(
class GoogleGenerativeAiInstrumentor(BaseInstrumentor):
"""An instrumentor for Google Generative AI's client library."""

def __init__(self, exception_logger=None, use_legacy_attributes=True):
def __init__(self, exception_logger=None, use_legacy_attributes=True, upload_base64_image=None):
super().__init__()
Config.exception_logger = exception_logger
Config.use_legacy_attributes = use_legacy_attributes
if upload_base64_image:
Config.upload_base64_image = upload_base64_image

def instrumentation_dependencies(self) -> Collection[str]:
return ("google-genai >= 1.0.0",)
Expand Down
6 changes: 6 additions & 0 deletions ...mentation-google-generativeai/opentelemetry/instrumentation/google_generativeai/config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,9 @@
from typing import Callable


class Config:
exception_logger = None
use_legacy_attributes = True
upload_base64_image: Callable[[str, str, str, str], str] = (
lambda trace_id, span_id, image_name, base64_string: str
Copy link
Copy Markdown
Contributor

@ellipsis-dev ellipsis-dev Bot Aug 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The lambda returns the type 'str' instead of a string value. This default no-op doesn't match expected behavior. Consider returning a valid string (or an async no-op function) instead of the type.

Suggested change
lambda trace_id, span_id, image_name, base64_string: str
lambda trace_id, span_id, image_name, base64_string: ""

)
Comment on lines +7 to +9
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot Aug 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Fix default upload_base64_image: current lambda returns the str type, not a URL string

The default should return a valid string (ideally a harmless placeholder URL) rather than the str class object. As-is, any call site that uses the default will get <class 'str'>, which breaks downstream JSON formatting for image_url.url and can crash or silently misbehave.

Apply this diff:

-    upload_base64_image: Callable[[str, str, str, str], str] = (
-        lambda trace_id, span_id, image_name, base64_string: str
-    )
+    # Default: do not upload, but return a harmless placeholder URL.
+    upload_base64_image: Callable[[str, str, str, str], str] = (
+        lambda trace_id, span_id, image_name, base64_string: "about:blank"
+    )

Optionally, consider letting this be Optional[Callable] with None default and handling the fallback in span_utils for more explicit behavior. I can draft that change across both google_generativeai and vertexai configs to keep them consistent.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
upload_base64_image: Callable[[str, str, str, str], str] = (
lambda trace_id, span_id, image_name, base64_string: str
)
# Default: do not upload, but return a harmless placeholder URL.
upload_base64_image: Callable[[str, str, str, str], str] = (
lambda trace_id, span_id, image_name, base64_string: "about:blank"
)
🤖 Prompt for AI Agents 
In
packages/opentelemetry-instrumentation-google-generativeai/opentelemetry/instrumentation/google_generativeai/config.py
around lines 7-9, the default upload_base64_image lambda currently returns the
str class object rather than a URL string; replace the default implementation so
it returns a harmless placeholder URL string (e.g.
"https://example.com/placeholder.png") instead of str; alternatively, if you
prefer the Optional approach, change the type to Optional[Callable[[str, str,
str, str], str]] = None and handle the fallback where upload_base64_image is
invoked (e.g., in span_utils) to return a placeholder URL when None.

Loading