diff --git a/e2e/cases/22_gemini_credential_custom_api_base.md b/e2e/cases/22_gemini_credential_custom_api_base.md new file mode 100644 index 000000000000..9c7560b43f0b --- /dev/null +++ b/e2e/cases/22_gemini_credential_custom_api_base.md @@ -0,0 +1,100 @@ +# Case 22 — Gemini provider with custom `api_base` + +## Goal + +End-to-end validation that the UI change in PR #24 (exposing `api_base` +on the Google AI Studio credential form) reaches the gemini provider +runtime correctly. Specifically: + +1. A deployment with `litellm_params.model = gemini/...` plus a custom + `api_base` pointing at a Gemini-compatible gateway resolves to the + gemini provider code path (no ADC, no Vertex auth). +2. The outbound request hits `{api_base}/models/{model}:generateContent` + with an `x-goog-api-key` header — matching what a self-hosted gateway + like Raven Router or anispark's ai-router exposes. +3. The response surface (status, `x-litellm-model-api-base`, + `x-litellm-model-group`) reflects the custom api_base, not Google's + default `generativelanguage.googleapis.com`. + +This case guards the runtime half of PR #24. The UI half is covered by +`tests/test_litellm/proxy/public_endpoints/test_public_endpoints.py::test_google_ai_studio_provider_fields_expose_api_base`. + +## Background + +Before this fix, the only LiteLLM UI options for "URL + API key Gemini" +were: + +- **OpenAI credential** mislabeled (admin confusion, wrong logo/group in + the dashboard). +- Raw admin API PATCH of `litellm_params.api_base` (no UI path). + +The runtime gemini provider already supported custom `api_base` via +`litellm/llms/vertex_ai/vertex_llm_base.py:415` — +`_check_custom_proxy` builds `{api_base}/models/{model}:{endpoint}` and +attaches `x-goog-api-key: {gemini_api_key}` (line 484). The +`_ensure_access_token_async` ADC path is skipped because +`custom_llm_provider == "gemini"` (line 714). The UI just needed to +expose the `api_base` field — see `provider_create_fields.json` → +`Google_AI_Studio`. + +## Preconditions + +- `e2e/.env` has `GEMINI_API_KEY` and `GEMINI_API_BASE` set, pointing at + a Gemini-compatible gateway. Examples: + - `https://generativelanguage.googleapis.com/v1beta` (canonical Google + AI Studio; works with an `AIza...` key) + - `https://ai-router-hk.anispark.ai/v1beta` (anispark / Raven Router + with their own sk-style key) + - any self-hosted proxy that serves + `POST /v1beta/models/{model}:generateContent` with + `x-goog-api-key` auth. +- Optionally `MODEL_GEMINI` to override the upstream model id (default + `gemini/gemini-3.1-pro-preview`). +- `e2e/tools/proxy status` reports `ready` against an image built AFTER + this PR's branch (`e2e/tools/proxy build` if unsure — the rendered + config gains a `gemini-custom-base` deployment when `GEMINI_API_KEY` + is set). +- Without `GEMINI_API_KEY` the case exits 77 (SKIP) — the proxy doesn't + render the deployment, so there's nothing to test. + +## Steps + +```bash +bash e2e/cases/data/22_gemini_credential_custom_api_base.sh +echo "exit=$?" +``` + +The fixture issues a single chat completion call against the +`gemini-custom-base` deployment and inspects the response headers: + +- **C1** `POST /v1/chat/completions` model=`gemini-custom-base` → + expects HTTP 200, body has `choices`, header + `x-litellm-model-api-base` equals the value of `GEMINI_API_BASE` from + `e2e/.env`, header `x-litellm-model-group == gemini-custom-base`. + +## Expected — GREEN + +``` +PASS C1: HTTP 200, x-litellm-model-api-base=https:///v1beta +PASS C1: x-litellm-model-group=gemini-custom-base +PASS C1: body has 'choices' array +``` + +## Failure modes + +| Symptom | Likely cause | +|---|---| +| `SKIP: GEMINI_API_KEY not set` | Add `GEMINI_API_KEY` and `GEMINI_API_BASE` to `e2e/.env`, then `e2e/tools/proxy restart`. | +| `FAIL C1: HTTP 500 ... DefaultCredentialsError` | The deployment routed to `vertex_ai`/`vertex_ai_beta` instead of `gemini`. Verify `MODEL_GEMINI` does not have `vertex_ai/` prefix and the rendered config shows `model: gemini/...` for `gemini-custom-base`. | +| `FAIL C1: HTTP 200 but body is HTML ()` | `api_base` is the gateway's UI root, not the API path. Add `/v1beta` (or the equivalent for your gateway) to `GEMINI_API_BASE`. | +| `FAIL C1: x-litellm-model-api-base != GEMINI_API_BASE` | LiteLLM is ignoring the custom api_base. Confirm the deployment block in the rendered config has `api_base: os.environ/GEMINI_API_BASE` and not the default Google endpoint. | +| `FAIL C1: 401 / 403` from upstream | `GEMINI_API_KEY` is wrong for this gateway, or the gateway expects a different auth header (some non-Google gateways want `Authorization: Bearer` instead of `x-goog-api-key`). Confirm with a direct curl to `{api_base}/models/{model}:generateContent`. | + +## Cross-reference + +- PR #24 (`fix(ui): expose api_base on Google AI Studio credential form`) +- `litellm/llms/vertex_ai/vertex_llm_base.py:415` — + `_check_custom_proxy` URL construction for gemini provider +- `litellm/llms/vertex_ai/vertex_llm_base.py:714` — gemini provider + skipping ADC +- `tests/test_litellm/proxy/public_endpoints/test_public_endpoints.py::test_google_ai_studio_provider_fields_expose_api_base` — UI field metadata assertion diff --git a/e2e/cases/data/22_gemini_credential_custom_api_base.sh b/e2e/cases/data/22_gemini_credential_custom_api_base.sh new file mode 100755 index 000000000000..795121a2b0a3 --- /dev/null +++ b/e2e/cases/data/22_gemini_credential_custom_api_base.sh @@ -0,0 +1,95 @@ +#!/usr/bin/env bash +# Case 22 fixture — see e2e/cases/22_gemini_credential_custom_api_base.md +# +# Asserts that a deployment with `litellm_params.model = gemini/...` +# plus a custom `api_base` routes through the gemini provider and the +# x-litellm-model-api-base header surfaces the custom URL (proving the +# UI api_base field added to the Google_AI_Studio credential form +# actually reaches the runtime). +# +# Exits 0 on PASS, 77 on SKIP, anything else on FAIL. + +set -u + +PROXY="${PROXY_URL:-http://localhost:4011}" +KEY="${MASTER_KEY:-sk-e2e-test}" +MODEL="gemini-custom-base" +FAILED=0 + +pass() { echo "PASS: $*"; } +fail() { echo "FAIL: $*"; FAILED=1; } +skip() { echo "SKIP: $*"; exit 77; } + +# Pull GEMINI_API_BASE from .env so we can assert the response header +# matches. We don't need the key here — the proxy holds it. +ENV_FILE="$(dirname "$0")/../../.env" +if [ -f "$ENV_FILE" ]; then + # Source the value without exporting other vars unintentionally. + EXPECTED_API_BASE=$( + grep -E '^GEMINI_API_BASE=' "$ENV_FILE" | tail -1 | sed -E 's/^GEMINI_API_BASE=//; s/^"//; s/"$//; s/^'"'"'//; s/'"'"'$//' + ) +else + EXPECTED_API_BASE="" +fi + +if ! curl -sSf -o /dev/null -m 3 "$PROXY/health/readiness"; then + skip "proxy not ready at $PROXY" +fi + +# If the proxy didn't render the deployment, GEMINI_API_KEY was unset. +if ! curl -sSf -m 5 -H "Authorization: Bearer $KEY" "$PROXY/v1/models" \ + 2>/dev/null | grep -q '"id"[[:space:]]*:[[:space:]]*"'"$MODEL"'"'; then + skip "deployment '$MODEL' missing from /v1/models — set GEMINI_API_KEY (and optionally GEMINI_API_BASE / MODEL_GEMINI) in e2e/.env, then re-run e2e/tools/proxy restart" +fi + +if [ -z "$EXPECTED_API_BASE" ]; then + skip "GEMINI_API_BASE not set in e2e/.env — case 22 needs a custom api_base to assert against" +fi + +# ---- C1: chat/completions with the custom-api_base gemini deployment ---- +read -r -d '' BODY <=1" 2>/dev/null; then + pass "body has 'choices' array" +else + fail "body missing or malformed 'choices' array" + echo "--- body (truncated) ---" + head -c 600 "$C1_BODY"; echo +fi + +rm -f "$C1_BODY" "$C1_HDRS" +exit $FAILED diff --git a/e2e/tools/proxy b/e2e/tools/proxy index df29efd23e18..6a04f1f8321a 100755 --- a/e2e/tools/proxy +++ b/e2e/tools/proxy @@ -109,6 +109,12 @@ def render_config() -> Path: ) anthropic_base = os.environ.get("ANTHROPIC_API_BASE", "").strip() openai_base = os.environ.get("OPENAI_API_BASE", "").strip() + gemini_base = os.environ.get("GEMINI_API_BASE", "").strip() + gemini_key_set = bool(os.environ.get("GEMINI_API_KEY", "").strip()) + gemini_model = _ensure_provider_prefix( + _env_or("MODEL_GEMINI", "gemini/gemini-3.1-pro-preview"), + implied_provider="gemini", + ) def _emit_model( model_name: str, model: str, key_env: str, base_env: str, has_base: bool, @@ -154,6 +160,19 @@ def render_config() -> Path: returned_model_name="public-name-for-clients", ), ] + # Dedicated deployment for case 22: gemini/ provider with a custom + # api_base, validating that the UI api_base field added to the + # Google_AI_Studio credential form actually flows through to a + # Gemini-compatible gateway (e.g. /v1beta path). + # Only emitted when GEMINI_API_KEY is configured so cases that don't + # need it (most of the suite) don't fail with "missing key". + if gemini_key_set: + blocks.append( + _emit_model( + "gemini-custom-base", gemini_model, + "GEMINI_API_KEY", "GEMINI_API_BASE", bool(gemini_base), + ) + ) body = ( "# AUTOGENERATED by e2e/tools/proxy from e2e/.env at proxy-start time.\n" "# Do not edit by hand — edit e2e/.env and re-run `e2e/tools/proxy start`.\n" @@ -163,7 +182,12 @@ def render_config() -> Path: f"# claude-haiku-cache -> {haiku_model}\n" f"# gpt-4o-mini-cache -> {openai_model}\n" f"# claude-renamed -> {sonnet_model} (returned_model_name=public-name-for-clients; case 20)\n" - "model_list:\n" + + ( + f"# gemini-custom-base -> {gemini_model} (case 22; GEMINI_API_BASE={gemini_base or ''})\n" + if gemini_key_set + else "" + ) + + "model_list:\n" + "\n\n".join(blocks) + "\n\n" "litellm_settings:\n" diff --git a/litellm/proxy/public_endpoints/provider_create_fields.json b/litellm/proxy/public_endpoints/provider_create_fields.json index 163c9648de7c..86c489e18237 100644 --- a/litellm/proxy/public_endpoints/provider_create_fields.json +++ b/litellm/proxy/public_endpoints/provider_create_fields.json @@ -1243,6 +1243,16 @@ "provider_display_name": "Google AI Studio", "litellm_provider": "gemini", "credential_fields": [ + { + "key": "api_base", + "label": "API Base", + "placeholder": "https://generativelanguage.googleapis.com/v1beta", + "tooltip": "Override only when using a Gemini-compatible gateway (e.g. /v1beta path on a self-hosted proxy). LiteLLM appends '/models/{model}:generateContent' so include '/v1beta' but not the trailing slash.", + "required": false, + "field_type": "text", + "options": null, + "default_value": "https://generativelanguage.googleapis.com/v1beta" + }, { "key": "api_key", "label": "API Key", diff --git a/tests/test_litellm/proxy/public_endpoints/test_public_endpoints.py b/tests/test_litellm/proxy/public_endpoints/test_public_endpoints.py index a65462d3f9b1..23ae6047475a 100644 --- a/tests/test_litellm/proxy/public_endpoints/test_public_endpoints.py +++ b/tests/test_litellm/proxy/public_endpoints/test_public_endpoints.py @@ -166,9 +166,9 @@ def test_anthropic_provider_fields_support_byok(): "Anthropic api_key must be optional so admins can configure BYOK models " "without entering a key. See BYOK tutorial." ) - assert fields_by_key["api_key"].get("tooltip"), ( - "Anthropic api_key must have a tooltip explaining the BYOK use case." - ) + assert fields_by_key["api_key"].get( + "tooltip" + ), "Anthropic api_key must have a tooltip explaining the BYOK use case." assert "api_base" in fields_by_key, ( "Anthropic provider form must expose api_base so cloud customers " "can override the upstream URL without env var access." @@ -176,17 +176,66 @@ def test_anthropic_provider_fields_support_byok(): api_base_field = fields_by_key["api_base"] assert api_base_field["required"] is False assert api_base_field["field_type"] == "text" - assert api_base_field.get("tooltip"), ( - "api_base should have a tooltip explaining it is optional." - ) + assert api_base_field.get( + "tooltip" + ), "api_base should have a tooltip explaining it is optional." # UI forms render fields in credential_fields order; api_base should come first # so an admin sees the URL override before the key field. field_order = [f["key"] for f in anthropic["credential_fields"]] - assert field_order.index("api_base") < field_order.index("api_key"), ( - "api_base must appear before api_key in credential_fields (matches AI21 and ANTHROPIC_TEXT convention)." + assert field_order.index("api_base") < field_order.index( + "api_key" + ), "api_base must appear before api_key in credential_fields (matches AI21 and ANTHROPIC_TEXT convention)." + + +def test_google_ai_studio_provider_fields_expose_api_base(): + """The Google AI Studio (gemini) credential form must let admins set a custom + api_base so they can point at a Gemini-compatible gateway (e.g. a self-hosted + proxy at /v1beta) without env var access. + + The runtime gemini provider already supports custom api_base via + `vertex_llm_base._check_custom_proxy`; the UI just needs to expose the field. + """ + app_instance = FastAPI() + app_instance.include_router(router) + test_client = TestClient(app_instance) + + response = test_client.get("/public/providers/fields") + assert response.status_code == 200 + providers = response.json() + + google_ai = next( + (p for p in providers if p["provider"] == "Google_AI_Studio"), None + ) + assert google_ai is not None, "Google_AI_Studio provider entry not found" + assert google_ai["litellm_provider"] == "gemini" + + fields_by_key = {f["key"]: f for f in google_ai["credential_fields"]} + assert "api_key" in fields_by_key + assert "api_base" in fields_by_key, ( + "Google_AI_Studio provider form must expose api_base so admins can " + "point at a Gemini-compatible gateway without env var access." + ) + + api_base_field = fields_by_key["api_base"] + assert api_base_field["required"] is False + assert api_base_field["field_type"] == "text" + # Default value should match the canonical Google AI Studio endpoint that + # LiteLLM's gemini provider talks to when api_base is unset, so leaving the + # default in the form behaves identically to leaving it blank. + assert ( + api_base_field["default_value"] + == "https://generativelanguage.googleapis.com/v1beta" ) + # UI forms render fields in credential_fields order; api_base should come + # first so an admin sees the URL override before the key field (matches + # OpenAI and Anthropic conventions). + field_order = [f["key"] for f in google_ai["credential_fields"]] + assert field_order.index("api_base") < field_order.index( + "api_key" + ), "api_base must appear before api_key in credential_fields." + def test_public_model_hub_with_healthy_model(): """Test that health information is populated for a healthy model"""