fix(ci): repair the security sweep and Lens billing integration tests for ROI, JWKS, and release identity changes

[devin-ai-integration]

TLDR

Problem this solves:

• integration-security and integration-accounting fail every scheduled CircleCI run on main
• Three merged PRs changed behavior the integration tests still pin

How it solves it:

• Security sweep skips the new GitHub-backed ROI repositories route
• Security sweep expects the new credential JWKS export to 404 for non-federation credentials
• Lens billing tests run on an owned proxy with a release identity and claim with protocol 4

Details

Red on main in scheduled pipelines 90840 (0ed1c08), 90836 and 90824

integration-security (2239960, 2239784, 2239126), about 35 cases per run

• Failing ids include tests/integration/security/test_stored_config_slots.py::test_master_key_from_env_authorizes_admin_calls_only[success] and tests/integration/security/test_request_path_slots.py::test_request_credential_reaches_only_the_provider[D4-messages], failing with Owned proxy tried to reach external hosts: [b'CONNECT api.github.com:443 HTTP/1.1']
• Culprit 1a70233, feat(roi): measure shipping velocity, quality, and recorded spend #44426 (feat(roi): measure shipping velocity, quality, and recorded spend)
• Mechanism: feat(roi): measure shipping velocity, quality, and recorded spend #44426 adds GET /roi-calculator/observed/repositories, which lists repositories from GitHub (api.github.com by default). The S2 route sweep calls every GET route as the admin, so the owned proxy reached https://api.github.com/user/repos and hit the egress trap. The older /roi-calculator/repositories route was already deny-listed for the same reason
• Fix: add the observed route to ROUTE_DENY_LIST with that reason

integration-security, second cause, 4 cases per run

• Failing ids: tests/integration/security/test_stored_config_slots.py::test_named_credential_reaches_only_the_provider[success], [provider_4xx], test_team_model_config_credential_override_reaches_only_the_provider[success], [provider_4xx], failing with GET /credentials/<id>/jwks as admin -> 404
• Culprit 0ed1c08, feat(anthropic): workload identity federation and pluggable identity sources #44448 (feat(anthropic): workload identity federation and pluggable identity sources)
• Mechanism: feat(anthropic): workload identity federation and pluggable identity sources #44448 adds GET /credentials/{credential_name:path}/jwks, which by design 404s for any credential that is not an anthropic internal_issuer credential. The canary credential is not one, so the sweep's wrong-id check fired
• Fix: add the route to NOT_FOUND_EXPECTED with that reason, as done for the other routes that 404 by design

integration-accounting (2239963, 2239782, 2239121)

• Failing ids: tests/integration/spend/test_lens_billing.py::test_lens_bills_selected_key_and_rechecks_its_permissions[False], [True], test_worker_spend_logs_do_not_expose_investigation_content[False], [True], failing with POST /lens/workers/register: 503 ... This LiteLLM build has no release identity
• Culprit cb17588, feat(lens): coordinate worker releases and bundled installs #44428 (feat(lens): coordinate worker releases and bundled installs)
• Mechanism: feat(lens): coordinate worker releases and bundled installs #44428 intentionally makes worker registration return 503 when the gateway has no release identity, and makes /lens/worker/claim require protocol_version=4 plus a matching worker_release. The CI gateway runs from an editable install, so it has no release tag, and the tests still claimed with protocol_version=2
• Fix: both tests boot an owned proxy with LITELLM_RELEASE_TAG=v0.0.0-lens-integration, claim with protocol_version=4&worker_release=<tag>, and the first test also asserts the returned worker image. Every billing, permission and revocation assertion is unchanged

Mutation checks: removing the deny-list entry brings back CONNECT api.github.com:443; removing the NOT_FOUND_EXPECTED entry brings back the JWKS 404 failure; dropping the release tag override brings back the 503; a wrong worker_release makes the claim answer 409

Deferred to later scheduled runs (also red on main, not claimed here): unit, integration-extensions, integration-providers, integration-mcp, integration-cost, integration-management, integration-management-replica, integration-database-replica, and the remaining integration-accounting spend-precision case

Screenshots / Proof of Fix

Test-only change, so the proof is the previously red CircleCI jobs passing on this branch

PR branch pipeline at 635f9fa: 90843

• integration-security passed (2240084), 131 tests green, red on main in 90840
• integration-accounting (2240077): all four Lens billing cases pass. Its one remaining failure, test_documented_header_and_body_tags_reach_recorded_branch_and_pr_cost (spend 0.011 instead of 0.033), fails identically on main and is deferred
• Every other red job on the PR is equally red on main in 90840 (unit, integration-cost, integration-mcp, integration-extensions, integration-providers, integration-management, integration-management-replica, integration-database-replica); none of them run the two changed files
• llm_translation_testing failed test_non_streaming_reasoning_effort against a live provider (no reasoning_content in the reply); it passed on main, does not touch the changed files, and passed on re-run (2240131)

Type

✅ Test

• 635f9fa passes /live-pr-risk

Link to Devin session: https://app.devin.ai/sessions/0b0b380b69554d92af98b2d2af0f59d4
Open in Devin Desktop: https://app.devin.ai/desktop/session/0b0b380b69554d92af98b2d2af0f59d4?variant=devin
Requested by: @mateo-berri

---

Note

Low Risk
Test-only changes to deny lists and integration harness setup; no production code paths modified.

Overview
Updates integration tests so scheduled integration-security and integration-accounting jobs match recent product behavior instead of failing on main.

The security route sweep (S2) now skips GET /roi-calculator/observed/repositories because it calls the connected GitHub/GitLab API (same egress trap as the existing /roi-calculator/repositories deny entry). It also treats admin 404 on GET /credentials/{credential_name}/jwks as expected when the canary credential is not an Anthropic internal_issuer federation credential.

Lens billing integration tests boot an owned proxy with LITELLM_RELEASE_TAG=v0.0.0-lens-integration, route all Lens traffic through that client, claim jobs with protocol_version=4 and a matching worker_release, and assert the registered worker image tag. Billing, permission, spend-log privacy, and revocation assertions are unchanged.

^{Reviewed by Cursor Bugbot for commit 635f9fa. Bugbot is set up for automated code reviews on this repo. Configure here.}

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

• Disable automatic comment, CI, and merge conflict monitoring

[mateo-berri]

bugbot run

[greptile-apps]

Confidence Score: 5/5

[Low risk] Test file updates for security and billing integration tests.

The PR appears safe to merge; no actionable regression was identified.

Summary

The PR updates integration tests for the observed repositories route, credential JWKS responses, and Lens worker release identity.

• Excludes an external-API route from the security sweep and accepts the canary credential’s expected JWKS 404.
• Runs Lens billing tests on release-identified owned proxies and claims work using protocol version 4.

_{Reviews (1) · Last reviewed commit: "fix(ci): expect the credential JWKS expo..."}

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 635f9fa. Configure here.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[mateo-berri]

LGTM