Skip to content

test(vertex_ai): pin gemini-3.1-flash-lite reasoning mapping#23090

Open
vincentkoc wants to merge 5 commits intoBerriAI:mainfrom
vincentkoc:vincentkoc-code/gemini-3-1-flash-lite
Open

test(vertex_ai): pin gemini-3.1-flash-lite reasoning mapping#23090
vincentkoc wants to merge 5 commits intoBerriAI:mainfrom
vincentkoc:vincentkoc-code/gemini-3-1-flash-lite

Conversation

@vincentkoc
Copy link
Contributor

@vincentkoc vincentkoc commented Mar 8, 2026

Relevant issues

Related: #22674, #22920

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

  • I have Added testing in the tests/test_litellm/ directory, Adding at least 1 test is a hard requirement - see details
  • My PR passes all unit tests on make test-unit
  • My PR's scope is as isolated as possible, it only solves 1 specific problem
  • I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

CI (LiteLLM team)

  • Branch creation CI run
    Link:

  • CI run for the last commit
    Link:

  • Merge / cherry-pick CI run
    Links:

Type

✅ Test

Changes

  • Add explicit tests/test_litellm/ regression coverage for gemini-3.1-flash-lite-preview
  • Cover all six supported reasoning_effort values for this model: minimal, low, medium, high, disable, none
  • Assert the model stays on the thinkingLevel path and does not regress to thinkingBudget
  • Assert includeThoughts stays aligned with the enabled vs disabled effort cases

Validation

  • PYTEST_XDIST_WORKER=gw0 uv run pytest tests/test_litellm/llms/vertex_ai/gemini/test_vertex_and_google_ai_studio_gemini.py -k gemini_31_flash_lite -q
    • 6 passed, 87 deselected
  • uv run ruff check tests/test_litellm/llms/vertex_ai/gemini/test_vertex_and_google_ai_studio_gemini.py
    • still blocked by pre-existing lint violations in this file unrelated to this diff

Reference

@vercel
Copy link

vercel bot commented Mar 8, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
litellm Error Error Mar 8, 2026 5:48pm

Request Review

@vincentkoc
Copy link
Contributor Author

@Sameerlite this is a narrow follow-up to #22674 and #22920. It adds explicit tests/test_litellm/ coverage for gemini-3.1-flash-lite-preview reasoning mapping, based on the current Vertex AI model docs: https://docs.cloud.google.com/vertex-ai/generative-ai/docs/models/gemini/3-1-flash-lite

@vincentkoc vincentkoc marked this pull request as ready for review March 8, 2026 05:19
@greptile-apps
Copy link
Contributor

greptile-apps bot commented Mar 8, 2026

Greptile Summary

This PR adds a single parametrized regression test (test_reasoning_effort_maps_to_thinking_level_gemini_31_flash_lite) that pins all six reasoning_effortthinkingConfig mappings for gemini-3.1-flash-lite-preview in the tests/test_litellm/ directory.

Key points:

  • All six reasoning_effort values (minimal, low, medium, high, disable, none) are covered with correct expected values verified against the implementation in _map_reasoning_effort_to_thinking_level.
  • The test exercises the full map_openai_params integration layer, not just the private helper — consistent with the pattern used by the neighbouring test_reasoning_effort_maps_to_thinking_level_gemini_3 test.
  • Three assertions per case: the exact thinkingLevel string, includeThoughts boolean, and the absence of thinkingBudget (confirming the model stays on the thinkingLevel code path).
  • No real network calls; this is a pure-unit mock test as required by repository conventions.
  • The PR description reports 6 passed, 87 deselected from a local run, satisfying the requirement for evidence that the issue is resolved.

Confidence Score: 5/5

  • This is a test-only PR with no production code changes; it is safe to merge.
  • The change is entirely additive test code. All expected parameter values were cross-checked against the _map_reasoning_effort_to_thinking_level implementation and are correct. The test uses the full integration path (map_openai_params) and makes no network calls. Previous review concerns (incomplete parametrize coverage, missing thinkingLevel assertion, bypassing the integration layer) are fully addressed in this revision.
  • No files require special attention.

Important Files Changed

Filename Overview
tests/test_litellm/llms/vertex_ai/gemini/test_vertex_and_google_ai_studio_gemini.py Adds 6 parametrized regression tests for all reasoning_effort values for gemini-3.1-flash-lite-preview through the full map_openai_params integration layer; asserts thinkingLevel, includeThoughts, and absence of thinkingBudget.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A["test_reasoning_effort_maps_to_thinking_level_gemini_31_flash_lite\n(parametrized × 6 effort values)"] --> B["VertexGeminiConfig.map_openai_params(\n  model='gemini-3.1-flash-lite-preview'\n)"]
    B --> C{"_is_gemini_3_or_newer?"}
    C -- "Yes ('gemini-3' in model)" --> D["_map_reasoning_effort_to_thinking_level(effort, model)"]
    D --> E{"is_gemini3flash?\n('gemini-3.1-flash' in model)"}
    E -- "True" --> F["minimal → thinkingLevel='minimal'"]
    E -- "True" --> G["low → thinkingLevel='low'"]
    E -- "True" --> H["medium → thinkingLevel='medium'"]
    E -- "True" --> I["high → thinkingLevel='high'"]
    E -- "True" --> J["disable → thinkingLevel='minimal', includeThoughts=False"]
    E -- "True" --> K["none → thinkingLevel='minimal', includeThoughts=False"]
    F & G & H & I & J & K --> L["Assert thinkingConfig.thinkingLevel\nAssert thinkingConfig.includeThoughts\nAssert 'thinkingBudget' NOT in thinkingConfig"]
Loading

Last reviewed commit: 7d4bbcd

@vincentkoc
Copy link
Contributor Author

Addressed the Greptile follow-ups on the latest head:

  • expanded the regression test to cover all six reasoning_effort values for gemini-3.1-flash-lite-preview
  • verified the targeted test selection passes locally with:
    • PYTEST_XDIST_WORKER=gw0 uv run pytest tests/test_litellm/llms/vertex_ai/gemini/test_vertex_and_google_ai_studio_gemini.py -k gemini_31_flash_lite -q
    • result: 6 passed, 87 deselected

The PYTEST_XDIST_WORKER=gw0 env var avoids the repo-level module reload path that hits the known litellm.llms.base_llm.evals import issue in this environment.

…e_ai_studio_gemini.py

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant