test(e2e): add secret manager lanes for HashiCorp Vault and CyberArk Conjur

[yujonglee-berri]

Adds tests/e2e/secret_manager/ against the actual hashicorp/vault:1.20 and cyberark/conjur:1.24 containers. The Vault and Conjur test clients call each manager's HTTP API to write and read secrets. SecretStore is their shared typing interface, not a mock store

What it covers

Four behaviors, none of which mention a backend: a deployment key resolves from the manager, the value the manager holds is the one used, a generated virtual key is written to the manager, and a deleted one is removed from it

The proxy's own environment never holds the seeded secret, so a deployment can only get its key from the manager: the runner holds OPENAI_API_KEY and the tests copy it in under a fresh name per test, which rules out the os.environ fallback get_secret takes when the manager errors

Shape

key_management_system is global to a proxy, so each backend is its own lane against its own proxy. E2E_SECRET_MANAGER=<system> both opts the suite in and names the backend; the proxy boots from gateway/secret_manager_<system>_ci_config.yml, and the tests reach the same manager through that backend's SecretStore

A backend declares its capabilities, and a test that needs one carries @pytest.mark.requires_capability(...), so it is deselected rather than failed or skipped where it does not apply. CyberArk's proxy-side delete returns not_supported and Conjur keeps the key, so the delete test runs only on the Vault lane

Adding a backend touches no test and no marker: a store module, a registry entry, a lane config, and an up_<system> in secret_manager/backend.sh, which runs any backend in Docker and writes the proxy's and the tests' env. A half-added backend shows up when its lane runs, since tests/e2e allows no unit tests

Buildkite steps for both lanes: BerriAI/project-releaser#263

Testing

Against real containers (hashicorp/vault:1.20, cyberark/conjur:1.24):

• Each store round-trips write, overwrite with non-ASCII, read, absent read and idempotent destroy, and fails clearly on a wrong credential
• Each store cross-checked against litellm's own manager class both ways: the manager reads what the store seeds, and the store reads what the manager writes
• Collection: 4 tests on Vault, 3 on CyberArk with the delete test deselected, 0 with the opt-in unset

make lint-e2e-basedpyright is clean; ruff, the raw-requests check, the coverage registry tests, the strict collector and the e2e-changed gate tests all pass

Not run: the full proxy lane. Both managers are enterprise-gated and need LITELLM_LICENSE, and one test needs a real OPENAI_API_KEY; neither was available locally. The cross-checks above exercise the manager classes in-process instead

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[devin-ai-integration]

🔍 Devin Review: 2 flags

Not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)

[greptile-apps]

Confidence Score: 4/5

The secret-manager lanes appear sound, but the PR should not merge until the newly excluded LangSmith test has an actual CI execution path.

Findings

1.  LangSmith Test Never Runs ▶

Summary

Adds backend-agnostic live E2E coverage for gateway secret-manager integration with HashiCorp Vault and CyberArk Conjur.

• Introduces backend stores, capability-based test selection, and Docker setup for both managers.
• Covers deployment-secret resolution and virtual-key storage/deletion behavior.
• Adds gateway configurations, coverage-registry entries, contributor guidance, and shared external HTTP support.
• Fixes the previously reported fixture shape and credential-file handling issues.

_{Reviews (2) · Last reviewed commit: "Merge remote-tracking branch 'origin/mai..."}

[yujonglee-berri]

@greptile-apps wdyt