fix(ci): make the env-key doc gate see bare get_secret and get_secret_str reads

[yassin-berriai]

TLDR

Problem this solves:

• Bare get_secret/get_secret_str reads bypassed the env-key doc gate
• 334 environment variables were invisible to it
• It exited 0 while 142 live keys were undocumented

How it solves it:

• One pattern, prefix optional, with a negative lookbehind
• Gate accepts a key documented on any docs page
• Provider credentials count where they are actually documented

Relevant issues

Linear ticket

Resolves LIT-5178

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

The gate is the artefact being changed, so the proof is the gate's own output rather than a proxy request. Each run is python tests/documentation_tests/test_env_keys.py from the repo root with BerriAI/litellm-docs checked out at docs/my-website, which is exactly how test-code-quality.yml and test-unit-documentation.yml invoke it.

Before, litellm at 8ec562f, the commit this branch sits on, and litellm-docs at b8a9bfb1, the state before its rows landed. The gate passes, and passing is the bug:

$ python tests/documentation_tests/test_env_keys.py; echo "exit=$?"
All keys are documented in 'environment settings - Reference'. - frozenset({...683 keys...})
exit=0

At that same commit pair the widened patterns show what it was not looking at. It sees 683 keys where 1017 are actually read, and 142 of the 334 it had been blind to are mentioned nowhere on the docs site:

$ python tests/documentation_tests/test_env_keys.py
Exception: Environment variables read under ./litellm but mentioned nowhere in the docs:
['A2A_API_BASE', 'AI21_API_BASE', 'AIMLAPI_KEY', ... 'ZAI_API_BASE']

After, same litellm commit 8ec562f, litellm-docs at 5464d203, which carries the 142 rows from litellm-docs#785:

$ python tests/documentation_tests/test_env_keys.py; echo "exit=$?"
Every environment variable read under ./litellm is documented
exit=0

Only the docs tree differs between those two runs, so the 142 rows are demonstrably what closes the gap rather than anything else moving underneath.

One number to explain rather than leave looking inconsistent: the first measurement of this work found 143 undocumented keys and 1018 read, and both are one higher than the figures above. The extra key was AI211_API_KEY, a misspelling this gate surfaced, corrected in #35985 which is now in the base commit. Its correct spelling AI21_API_KEY was already read elsewhere and already documented, so removing it drops both counts by exactly one.

The middle run is the one that matters: the gate now names the keys it used to be blind to, and the only way to make it quiet is to document them.

Type

🐛 Bug Fix

✅ Test

Changes

tests/documentation_tests/test_env_keys.py is the gate that fails CI when a user-facing environment variable read under litellm/ is undocumented. Two of its patterns required a litellm. prefix, so a module importing get_secret or get_secret_str directly and calling it bare was invisible to it. Measured on the commit this PR branches from, that hid 334 keys: 99 reached through bare get_secret across 195 call sites, and 319 through bare get_secret_str across 738. The gate saw 683 keys where 1017 are actually read.

The three get_secret patterns collapse into one, (?<![\w.])(?:litellm\.(?:utils\.)?)?get_secret(?:_str|_bool)?. The prefix is optional, litellm.utils. is accepted because four call sites reach get_secret through the module, and the negative lookbehind keeps that from matching client.get_secret( on an unrelated object, of which there are two in the tree.

Both of those last two parts are defensive, and neither should be simplified out later on the grounds that it changes nothing. With and without them the pattern finds the same 380 keys today, because all six attribute-style call sites pass a variable or a keyword argument and so fail the quoted-literal capture. What they change is the next call written there: litellm.utils.get_secret("SOME_NAME") would otherwise be an env read the gate cannot see, and client.get_secret("SOME_NAME") on a vault client would otherwise be reported as an undocumented env var.

Widening the patterns alone would have demanded over 300 new rows in the central reference table, nearly all of them provider credentials and base URLs. A docs change at that scale gets rubber-stamped, and rubber-stamped documentation is worse than none. Rather than narrow what the gate checks, this widens where it looks: a key counts as documented if it is mentioned on any page of the docs site, not only in proxy/config_settings.md. Provider credentials genuinely are documented, on their own provider pages; the gate simply was not looking there. That resolved 179 of the 321 keys the table alone was missing.

The tradeoff is real and worth stating rather than absorbing: this gives a weaker guarantee about where a reader will find the key. It could be tightened later by requiring the mention to appear in a table rather than anywhere on the page.

The scan covers docs/ only, not blog/ or release_notes/. Including them buys exactly one additional key, and a release note mentioning a variable is not documentation of it.

One case shows the edge of that guarantee. Of the 179 keys accepted on a non-table mention, exactly one, REGION, is accepted on a coincidence: routing.md contains the prose SET 'EU' REGION NAME, which is not documentation of an environment variable. Requiring the mention to contain an underscore would rule that out, but it would also make HOSTNAME and JITTER unsatisfiable by any text, including the reference-table rows they already have, so the rule is broken rather than merely stricter. Every other one of the 179 is a genuine mention on the relevant provider page.

Two smaller consequences. Since a table row is itself a mention, parsing the reference table separately became a code path that can never disagree with the general one, so it is gone rather than kept alongside a duplicate. And the pass/fail decision moved into undocumented_env_keys(base_dir, docs_dir) so it can be tested against a synthetic source tree and docs tree, leaving main a thin shell; a missing docs checkout now says so explicitly instead of reporting all 1017 keys as undocumented, which would read as a catastrophic regression rather than a setup error.

Two call shapes stay invisible on purpose, named here so the boundary reads as considered rather than incomplete. A keyword-only get_secret(secret_name="X") is not matched, because the argument regex requires a quoted first positional; there are four such sites today and all pass a name that is documented anyway. And a bare utils.get_secret("X") reached through from litellm import utils is not matched, because the pattern requires litellm. before utils.; there are zero such sites today. Neither is worth widening for, and the second would be actively harmful: matching a bare utils. prefix would match any of the 39 modules named utils.py under litellm/, reintroducing the false-positive class the lookbehind exists to prevent. The one from . import utils in the tree, in the opik integration, refers to one of those unrelated modules.

The remaining 142 keys needed documenting first, or code-quality and documentation would go red for every open PR the moment this merged. They landed as BerriAI/litellm-docs#785, merged as 0e7f3231, each row written from its call site. That docs PR was safe to merge ahead of this one because the gate as it exists on staging passes against it too, which I checked rather than assumed.

Worth recording that the widened gate justified itself before landing. It surfaced a genuine source defect nobody was looking for: utils.py resolved the ai21 key from AI211_API_KEY, with a doubled 1, where every other ai21 site reads AI21_API_KEY. That was corrected in #35985, merged as 8ec562f ahead of this, so no misspelling ended up documented. Tracing it also turned up that the containing function has no callers at all, filed separately as a follow-up rather than folded in here, since removing it is a public API removal.

Tests

tests/test_litellm/test_env_key_doc_gate.py covers the call shapes the gate must see and the ones it must not, following the placement the earlier get_secret_bool fix used, since tests/documentation_tests/ is invoked as a script and never collected by pytest.

Mutation-checked, 7 of 7 killed: restoring the litellm. prefix requirement, dropping the lookbehind, refusing the litellm.utils. prefix, narrowing the mention scan back to reference-table rows, scanning only the reference table page, dropping the word boundaries from the mention pattern, and skipping .mdx pages each turn at least one test red.

Two of those mutations are the reason the tests are shaped the way they are. Reverting the docs lookup inside main originally left every test green, because the tests only exercised the helpers, which is the same class of defect this PR fixes; hence undocumented_env_keys being a separately testable function rather than inline logic. And the word-boundary test needs a glued-prefix fixture such as useQSTASH_ALPHA, because a longer-token fixture cannot distinguish the two patterns: a greedy non-overlapping findall never yields the suffix either way.

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[greptile-apps]

Greptile Summary

This PR expands the environment-key documentation gate to recognize bare and module-qualified secret-helper calls, then checks documentation mentions across the full Markdown/MDX documentation tree.

• Consolidates the secret-helper extraction patterns while excluding unrelated object methods.
• Extracts documentation-shaped keys from all documentation pages rather than only the central reference table.
• Adds synthetic-tree tests for supported call shapes, false-positive boundaries, documentation scanning, and missing documentation.

Confidence Score: 5/5

The PR appears safe to merge; no concrete changed-code failure remains.

The widened extractor targets the repository’s actual secret helpers, both CI callers provision the required documentation checkout, and the tests cover the principal matching and documentation-scan boundaries.

Important Files Changed

Filename	Overview
tests/documentation_tests/test_env_keys.py	Broadens static environment-key extraction, scans all documentation pages, and introduces a testable helper for computing undocumented keys; no actionable defect was established.
tests/test_litellm/test_env_key_doc_gate.py	Adds focused regression coverage for bare secret-helper calls, attribute false positives, Markdown/MDX scanning, token boundaries, and end-to-end gate behavior.

_{Reviews (1): Last reviewed commit: "fix(ci): make the env-key doc gate see b..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!