Skip to content

fix(google): set include_server_side_tool_invocations when mixing function declarations with built-in tools - #4754

Open
liramon2 wants to merge 4 commits into
strands-agents:mainfrom
liramon2:gemini-tools
Open

liramon2 wants to merge 4 commits into
strands-agents:mainfrom
liramon2:gemini-tools

Conversation

@liramon2

@liramon2 liramon2 commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Description

The Gemini Developer API requires tool_config.include_server_side_tool_invocations = true when both function declarations and built-in tools (e.g. googleSearch, codeExecution) are present in the same request. Neither SDK set this flag, causing every such request to fail:

400 INVALID_ARGUMENT  Please enable tool_config.include_server_side_tool_invocations
to use Built-in tools with Function calling.

This PR makes both SDKs detect the combination and set the flag automatically. The flag is skipped on Vertex AI, which does not support it. This also bumps gemini to google-genai>=1.68.0, which is the version introducing the flag.

Related Issues

Closes #3639

Type of Change

Bug fix

Testing

  • I ran hatch run prepare
  • I ran npm run check

Checklist

  • I have read the CONTRIBUTING document
  • I have reviewed and understand every line of code in this PR, including any generated by AI tools, and I can explain why it works
  • My change is focused and reasonably small; I have split unrelated work into separate PRs
  • I have added any necessary tests that prove my fix is effective or my feature works
  • I have updated the documentation accordingly
  • I have added an appropriate example to the documentation to outline the feature, or no new docs are needed
  • My changes generate no new warnings
  • Any dependent changes have been merged and published

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

…ction declarations with built-in tools

The Gemini Developer API requires tool_config.include_server_side_tool_invocations
when both function declarations and built-in tools (e.g. GoogleSearch) are present
in the same request. Neither SDK set this flag, causing every such request to fail
with 400 INVALID_ARGUMENT.

Set the flag automatically in both Python and TypeScript when the combination is
detected. The flag is skipped on Vertex AI, which does not support it, and is not
set when the caller provides an explicit tool_config via params.

Closes strands-agents#3639
@github-actions github-actions Bot added bug Something isn't working complexity/high A touched function exceeds cognitive complexity 25; may be worth splitting size/xs area-model Related to models or model providers labels Sep 30, 2026
@codecov

codecov Bot commented Sep 30, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@liramon2

Copy link
Copy Markdown
Contributor Author

@strandly-the-agent Review this PR

Comment thread strands-ts/src/models/__tests__/google.test.ts
Comment thread strands-py/src/strands/models/gemini.py Outdated
@github-actions

Copy link
Copy Markdown
Contributor

Assessment: Comment

Focused, well-scoped bug fix with good cross-SDK parity on the core condition (toolSpecs + built-in tools + not-Vertex) and solid test coverage on both sides (no-choice, auto, Vertex-skip, built-in-only). A couple of non-blocking items below.

Review Categories
  • Cross-SDK parity: The "caller owns toolConfig" behavior is explicit and tested in Python but incidental (spread ordering) and untested in TS — worth a mirroring test to lock it in.
  • Testing: New TS assertions are per-field; a full-object toEqual would catch unexpected/regressed fields and match the existing style in that suite.
  • Code quality (minor): Python may mutate a caller-supplied ToolConfig held in persisted config; idempotent but a small smell.

Nice job skipping the flag on Vertex and covering it with a dedicated test in both SDKs.

@strandly-the-agent strandly-the-agent left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes requested — one 🔴: the Vertex skip only inspects client_args, so the other two ways of selecting Vertex now hard-fail a request that succeeds on main.

🔴 gemini.py:358-360 — GOOGLE_GENAI_USE_VERTEXAI=true and the non-legacy client_args={"enterprise": True} both leave client_args["vertexai"] unset, so the flag is sent to Vertex and google-genai raises ValueError: include_server_side_tool_invocations parameter is only supported in Gemini Developer API mode… before the request leaves the process. TS gets this right by reading this._client.vertexai. Inline.

🟡 strands-py/pyproject.toml:50 — floor is google-genai>=1.67.0, but the field first exists in 1.68.0; at the floor gemini.py:365 raises AttributeError: 'ToolConfig' object has no attribute 'include_server_side_tool_invocations' on every mixed-tools request. One-line bump (closed #3646 had it). TS needs nothing: @google/genai 2.6.0 already declares the field.

🟡 A caller-supplied tool_config gets three different treatments for one semantic input — ToolConfig instance → flag injected into the caller's own object; equivalent dict → skipped; TS → dropped entirely. Inline.

The core condition, the Vertex-skip intent and the test matrix are otherwise right, and all three fixes are small.

Verified, questions, appendix

✅ Verified at 2abb0171 (base 0ab57b24)

  • cd strands-py && uv run --extra gemini --extra dev pytest tests/strands/models/test_gemini.py -q → 79 passed; cd strands-ts && npx vitest run --project unit-node src/models/__tests__/google.test.ts → 87 passed; ruff check + ruff format --check clean on the touched Python files.
  • Vertex really does reject the flag (not taking the PR description's word): google/genai/models.py:4008-4012 raises inside _ToolConfig_to_vertex, and the @google/genai dist carries the mirror throw. So sending it on Vertex is a hard failure, not a silent no-op — which is what makes the 🔴 bite.
  • 🔴 regression proof (vertex-env-regression.txt): identical config — GOOGLE_GENAI_USE_VERTEXAI=true, one agent tool + gemini_tools=[Tool(google_search=…)]. Base 0ab57b24 gets as far as the wire (fails only on missing ADC); pr-4754 raises the ValueError locally. Also reproduced via client_args={"enterprise": True} (repro-output.txt case C).
  • Floor (google-genai-floor-check.txt): at 1.67.0 ToolConfig has field: False and _format_request_config → AttributeError; at 1.68.0 the flag is set. @google/genai 2.6.0's dist/genai.d.ts declares includeServerSideToolInvocations?: boolean on ToolConfig, so ^2.6.0 is fine.
  • Precedence (repro-output.txt, ts-toolconfig-precedence.txt): instance → flag injected, sent is caller_tc → True, caller object mutated; dict → None; TS → {"functionCallingConfig":{"mode":"NONE"}}, no flag.
  • TS structured output is covered by the same condition: StructuredOutputTool is registered on the tool registry (agent.ts:1580) and toolSpecs are read from that registry (agent.ts:2083), so Agent({ structuredOutputSchema, model: new GoogleModel({ builtInTools }) }) gets the flag with zero user tools. Python's structured_output uses response_schema with tool_specs=None, so it never mixes.

Questions

  • (drives the 🟡, otherwise non-blocking) Is "the caller owns tool_config" the right rule for a field that is a transport requirement rather than a policy choice? Someone who sets params={"tool_config": …} for an unrelated reason keeps exactly the 400 this PR fixes. If the answer is "the flag always goes on", both SDKs should merge it in; if it's "caller wins", Python shouldn't inject into a supplied ToolConfig either.
  • (non-blocking) Gating on gemini_tools being non-empty rather than on the built-ins actually being server-side (url_context, etc.) is broader than the API's trigger. Harmless as far as I can tell — deliberate?

Reading order

gemini.py:357-366 → model.ts:345-352 → the two Vertex tests.

Appendix — non-blocking (3)

  • The new tests don't reference #3639; AGENTS.md:57 asks a regression test for a discovered bug to link the issue it guards against.
  • test_stream_gemini_tools_flag_skipped_when_params_owns_tool_config proves "the caller owns it" only for the dict shape; the ToolConfig-instance shape behaves the opposite way and is untested (see the 🟡).
  • Docs/changelog: nothing needed that I can find — site/src/content/changelog/sdk/ is per-release rather than per-PR, and google.mdx:174-209 already presents built-in + function tools as a supported combination.

Comment thread strands-py/src/strands/models/gemini.py Outdated
Comment thread strands-py/src/strands/models/gemini.py Outdated
Comment thread strands-ts/src/models/google/model.ts Outdated
@github-actions github-actions Bot added size/s and removed size/xs labels Sep 30, 2026
@liramon2

Copy link
Copy Markdown
Contributor Author

@strandly-the-agent Review this PR

@strandly-the-agent strandly-the-agent left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up at 76ddf6a5: the 🔴 is fixed and both SDKs now share one tool_config rule. Two 🟡 left before I'd approve — one still open from last round, one introduced by the fix (my suggestion, so partly on me).

  • 🔴 Vertex detection → fixed. GOOGLE_GENAI_USE_VERTEXAI=true and client_args={"enterprise": True} both now send no flag (re-ran the repro).
  • 🟡 caller tool_config mutated / dict skipped / TS dropped → fixed. Instance is copied via to_json_dict(), dict is merged, TS merges after the params spread and keeps an explicit false; both SDKs behave the same and each case has a test.
  • 🟡 google-genai>=1.67.0 floor → still open (the thread was resolved but pyproject.toml:50 is unchanged). At 1.67.0 the new code fails differently: ValidationError: tool_config.include_server_side_tool_invocations — Extra inputs are not permitted (ToolConfig is extra='forbid' there). One-line bump to >=1.68.0.
  • 🟡 new: is_vertex = self._get_client().vertexai runs unconditionally, so every Gemini request now builds a second genai.Client (stream() builds its own two lines later). Inline, with a fix.
Verified, appendix

✅ Verified at 76ddf6a5

  • uv run --extra gemini --extra dev pytest tests/strands/models/test_gemini.py -q → 81 passed; npx vitest run --project unit-node src/models/__tests__/google.test.ts → 88 passed; ruff check / ruff format --check / mypy clean on gemini.py.
  • Repro re-run (formatter level, _format_request(...)["config"]["tool_config"]): env-var Vertex → None; enterprise=True → None; caller ToolConfig instance → flag sent, caller_tc.include_server_side_tool_invocations stays None; dict tool_config → {'function_calling_config': {'mode': 'AUTO'}, 'include_server_side_tool_invocations': True}.
  • Floor: scratch venv pinned to google-genai==1.67.0 → ValidationError above; 1.68.0+ → flag set.
  • Extra client per request: _format_request_config → _get_client() → genai.Client(**client_args) (gemini.py:142) on every call, including requests with no gemini_tools. In Vertex mode with no explicit project that constructor also runs google.auth.default() (google/genai/_api_client.py:789-794), so that now happens twice per request.

Appendix — non-blocking (2)

  • New tests still don't reference #3639 (AGENTS.md:57 asks regression tests to link the issue they guard).
  • Gating on gemini_tools being non-empty rather than on the built-ins actually being server-side is broader than the API's trigger — deliberate? Harmless as far as I can tell.

Comment thread strands-py/src/strands/models/gemini.py Outdated
Co-authored-by: Strandly (The Agent) <strands-agents+strandly@amazon.com>
@liramon2
liramon2 marked this pull request as ready for review October 1, 2026 00:23
@liramon2
liramon2 requested a review from a team as a code owner October 1, 2026 00:23
@liramon2
liramon2 requested a review from arielnabavian October 1, 2026 00:23
@github-actions

github-actions Bot commented Oct 1, 2026

Copy link
Copy Markdown
Contributor

Assessment: Approve

Re-reviewed after the latest revision — all three items from my previous round are addressed, and I verified the tests locally (11 relevant Python tests pass, all 88 TS google.test.ts tests pass with no type errors).

Resolved from prior review
  • Python shared-object mutation: Fixed — the flag is now applied to a copied dict (to_json_dict() / dict(tc or {}) + setdefault) instead of mutating a caller-supplied ToolConfig in place. Vertex detection is also cleaner via _get_client().vertexai.
  • TS params ordering: Fixed — the flag logic now runs after Object.assign(config, params) and uses ?? true, so a caller-supplied toolConfig is merged into (not overwritten), and an explicit includeServerSideToolInvocations: false is respected. The new parametrized tests ("merges flag into params-supplied toolConfig", "preserves explicit …=false") lock this in.
  • TS assertions: Now use full-object toEqual on toolConfig, matching the Python full-request assertions and catching unexpected fields.
Note on the Codecov report

The Codecov "0% patch coverage / 6 missing lines" on gemini.py appears stale/misleading — a local coverage run shows the new lines (357–363) are exercised by the passing tests and are not in the missing list. No action needed, but worth noting so it doesn't block merge.

Dependency bump to google-genai>=1.68.0,<3.0.0 is correct (that's the version introducing the flag) and keeps a sensible upper bound. Clean fix with strong cross-SDK parity — nice work.

@liramon2

liramon2 commented Oct 1, 2026

Copy link
Copy Markdown
Contributor Author

@strandly-the-agent Review this PR

@strandly-the-agent strandly-the-agent left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve at f7804c5f — both remaining 🟡 are fixed; nothing open from me.

  • google-genai floor → fixed, >=1.68.0 (pyproject.toml:50).
  • Extra genai.Client per request → fixed: _get_client() is now the last operand of the short-circuit, so it only runs for requests with both tool_specs and gemini_tools. Verified by counting genai.Client constructions in _format_request: 0 without gemini_tools, 0 with gemini_tools but no tools, 1 with both.
Verified
  • uv run --extra gemini --extra dev pytest tests/strands/models/test_gemini.py -q → 81 passed; ruff check / ruff format --check / mypy clean on gemini.py. TS unchanged since 76ddf6a5 (88 passed there).
  • Delta 76ddf6a5..f7804c5f is only the floor bump and the short-circuit reorder; earlier verified fixes (Vertex via resolved client, non-mutating tool_config merge, TS merge after params spread) untouched.
  • uv.lock is untracked in this repo, so nothing to update for the bump.

@github-actions

github-actions Bot commented Oct 1, 2026

Copy link
Copy Markdown
Contributor

Assessment: Approve (re-confirmed after refactor: trim tests)

Reviewed the latest refactor: trim tests commit. The consolidation is healthy — the Python side was folded into a parametrized test, and every critical scenario is still covered on both SDKs: base case, toolChoice/AUTO, Vertex-AI skip, merge-into-params-toolConfig, and preserve-explicit-false. The implementation itself is unchanged from my previous approval.

Verified locally:

  • Python: 79 tests pass; gemini.py at 98% coverage with the new lines (357–363) fully exercised.
  • TypeScript: all 88 google.test.ts tests pass, no type errors.
  • Codecov now reports "All modified and coverable lines are covered by tests" — the earlier 0% was stale, as suspected, and is now resolved.

No outstanding concerns. Good to merge. 🚀

@github-actions

github-actions Bot commented Oct 1, 2026

Copy link
Copy Markdown
Contributor

Assessment: Approve (re-confirmed on 6f9084ab)

Reviewed the newly force-pushed refactor: trim tests commit (6f9084ab). The diff versus the version I approved earlier today is purely cosmetic — a single test-function signature was re-wrapped onto one line (Python test file: 117 → 115 added lines). All test bodies and the implementation (gemini.py, model.ts, pyproject.toml) are byte-identical, and every critical scenario remains covered on both SDKs.

Re-ran on the new HEAD: 79 Python test_gemini.py tests pass. Codecov continues to report all modified lines covered.

Nothing further from me — this remains good to merge. 🚀

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area-model Related to models or model providers bug Something isn't working complexity/high A touched function exceeds cognitive complexity 25; may be worth splitting size/s

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Gemini: built-in tools + function tools returns 400

2 participants