refactor(e2e): remove bob_the_builder; drive remediation from a Grafana alert (provisioned outside the repo)

[yassin-berriai]

Relevant issues

Linear ticket

Resolves LIT-4551

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests (removal only; no new test, the removed path had no test)
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

This is harness-internal cleanup with no runtime LLM surface, so the proof is that the module is fully removed, the session-finish cleanup it shared a hook with is untouched, and the E2E_RESULT signal that downstream alerting consumes is still emitted by the harness. Captured at head 82d878e20b

bob existed on the base and is gone on head, with no residual references:

$ git ls-tree origin/litellm_internal_staging tests/e2e/bob_the_builder.py
100644 blob 18aff2edc989288bd22fa456c2da7307bbf30e16	tests/e2e/bob_the_builder.py
$ git ls-tree HEAD tests/e2e/bob_the_builder.py
        (nothing; the file is deleted)

$ git grep -n "bob_the_builder\|E2E_DEVIN_REMEDIATION\|remediate" -- tests/ litellm/
        (no matches)

The pytest_sessionfinish hook keeps its spend-log truncation path and the _E2E_TEST_RAN guard that protects it; only the trailing remediation call was removed. The alerting that replaces bob reads the E2E_RESULT line the harness already ships, so the emitter is untouched:

$ cd tests/e2e && python -c 'from e2e_result_reporter import result_from_pytest, format_e2e_result_line; \
    r = result_from_pytest(nodeid="logging/test_langfuse_e2e.py::TestLangfuse::test_success", when="call", \
    failed=True, skipped=False, passed=False, duration_seconds=1.5, covers=("logging.langfuse.success.logs_spend",)); \
    print(format_e2e_result_line(r))'
E2E_RESULT package=logging file=test_langfuse_e2e.py outcome=failed duration_ms=1500 node_id=logging/test_langfuse_e2e.py::TestLangfuse::test_success covers=logging.langfuse.success.logs_spend

Type

🧹 Refactoring

🚄 Infrastructure

Changes

This deletes tests/e2e/bob_the_builder.py and unwires it from the e2e runner

bob was an opt-in (E2E_DEVIN_REMEDIATION=1) pytest_sessionfinish step that, on a red run, collected the failing tests plus their tracebacks and called the Devin MCP devin_session_create tool through the LiteLLM proxy's own /mcp-rest gateway to open Linear tickets and fix PRs. The only wiring was the trailing remediation block in conftest.py's pytest_sessionfinish; the spend-log reset path and its _E2E_TEST_RAN guard are left as they were

The remediation is now driven from a Grafana alert on the E2E_RESULT lines the harness already emits into Loki, provisioned directly in Grafana rather than checked into this repo, so nothing here needs to carry the alerting config

Link to Devin session: https://app.devin.ai/sessions/ef60065427004a0faf31d5385fa78736
Requested by: @yassin-berriai

[gemini-code-assist]

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[yassin-berriai]

@greptileai

[greptile-apps]

Greptile Summary

This PR removes bob_the_builder.py and its pytest_sessionfinish hook, replacing the in-runner Devin remediation call with Grafana alerting-as-code that reads the E2E_RESULT Loki signal the harness already emits. The replacement eliminates the circular dependency where the proxy being unhealthy (the most common failure mode) also broke the remediation path.

• Deletion: bob_the_builder.py and its import/call in conftest.py are fully removed; the spend-log cleanup and _E2E_TEST_RAN guard in pytest_sessionfinish are untouched.
• New alerting: alert_rules.yaml uses a 1m evaluation interval over a 5m LogQL lookback (5× overlap) so boundary/late-ingested lines are never missed; notification_settings group by package with a 4h repeat_interval, replacing the old hand-rolled sha256 dedup.
• Secrets handling: Slack webhook and Loki datasource UID are both $__env{}-referenced so nothing environment-specific lands in git.

Confidence Score: 5/5

Safe to merge — the change is a clean removal of a fragile module and its hook, with no production code path affected.

The only runtime change is removing a try/except block that was already fully swallowing every exception, so the suite's exit status and all other conftest behaviour are identical before and after. The new Grafana YAML is inert until provisioned and has no effect on Python tests. No application logic, SDK surface, or proxy path is touched.

No files require special attention. All five changed files are in the e2e test infrastructure tree.

Important Files Changed

Filename	Overview
tests/e2e/bob_the_builder.py	Deleted — the circular-dependency in-runner remediation module is fully removed with no residual references.
tests/e2e/conftest.py	Cleanly unwires the bob_the_builder import and remediate() call from pytest_sessionfinish; spend-log cleanup path and _E2E_TEST_RAN guard are untouched.
tests/e2e/grafana/alerting/alert_rules.yaml	New Grafana alert rule with 1m evaluation interval and 5m LogQL lookback (5x overlap), correct A→B→C reduce/threshold chain, env-var-referenced datasource UID, and simplified notification routing.
tests/e2e/grafana/alerting/contact_points.yaml	New Slack contact point; webhook URL correctly env-var-referenced, resolve messages suppressed, Go template syntax valid.
tests/e2e/grafana/alerting/README.md	Operational runbook for the new alerting provisioning, colocated with the YAML files it documents.

_{Reviews (4): Last reviewed commit: "docs(e2e): note the alert selector must ..." | Re-trigger Greptile}

[greptile-apps]

Greptile Summary

This PR removes bob_the_builder.py — an in-runner pytest_sessionfinish hook that called Devin through the proxy's /mcp-rest gateway to open fix PRs on red e2e runs — and replaces it with Grafana Alerting provisioning that fires off the E2E_RESULT Loki signal the harness already emits. The conftest.py change is surgical: only the trailing remediation block is removed; the spend-log reset and _E2E_TEST_RAN guard are untouched.

• bob_the_builder.py is fully deleted with no residual imports or env-var references in the tree.
• tests/e2e/grafana/alerting/ adds three files: alert_rules.yaml (LogQL rule evaluated every 5 m, grouped and suppressed by package), contact_points.yaml (Slack webhook externalized via $__env{}), and README.md (design rationale and apply instructions). The two environment-supplied values (LOKI_DATASOURCE_UID, SLACK_E2E_WEBHOOK_URL) keep secrets out of git.

Confidence Score: 4/5

Safe to merge — the deletion of bob_the_builder and its conftest.py hook is clean and complete, and the new Grafana provisioning files are additive infrastructure-as-code with no runtime impact on the proxy or test suite.

The conftest.py edit is minimal and correct; the spend-log reset path is unaffected. The Grafana rule's LogQL window matches exactly its evaluation interval, which can cause edge-case boundary gaps where a failure logged right at an interval boundary escapes both windows — a minor but real operational concern worth a one-line fix before the alert goes live.

alert_rules.yaml — the [5m] LogQL range should be widened slightly relative to the 5m evaluation interval to avoid boundary gaps.

Important Files Changed

Filename	Overview
tests/e2e/bob_the_builder.py	File deleted — in-runner Devin remediation module removed as intended; no residual references remain
tests/e2e/conftest.py	Trailing remediation call removed from pytest_sessionfinish; spend-log reset path and _E2E_TEST_RAN guard are untouched and correct
tests/e2e/grafana/alerting/alert_rules.yaml	Grafana Alerting provisioning for e2e failures; query logic is correct but the [5m] LogQL window exactly matches the 5m evaluation interval, which can cause boundary gaps
tests/e2e/grafana/alerting/contact_points.yaml	Slack contact point provisioning; webhook correctly externalized via env var, disableResolveMessage=false will send resolved notifications to the Devin channel
tests/e2e/grafana/alerting/README.md	New documentation file added directly to the repo; design rationale and operational instructions are clear and accurate

_{Reviews (2): Last reviewed commit: "refactor(e2e): remove bob_the_builder; d..." | Re-trigger Greptile}

[yassin-berriai]

osv-scan is failing here for base drift, not this PR. It is red on essentially every open PR right now (a CVE published after the base branch last bumped its lockfile), and this PR changes no dependency files (git diff origin/litellm_internal_staging...HEAD lists no uv.lock/pyproject/requirements changes). The dependency bump belongs in its own PR on the base branch.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[yassin-berriai]

Addressed both notes from the last review in HEAD dca50c6.

Boundary-gap on the LogQL window: the rule now evaluates every 1m over the 5m lookback instead of every 5m, so windows overlap 5x and a line ingested late or near a boundary is always caught by several consecutive evaluations. Grouping and repeat_interval on notification_settings keep that to one notification. I also set disableResolveMessage: true on the Slack contact point so Devin only ever acts on a firing (red) suite, never on a recovery message.

README placement: this is provisioning-adjacent operational documentation colocated with the YAML it describes, matching the existing tests/e2e/grafana/status_history_panels.md in the same directory. It is not user-facing product documentation for the docs site, so it belongs next to the config rather than in the docs repo.

I verified the change end to end against a real Grafana 11.4.0 plus Loki: provisioning loads with no errors, and after pushing a failing E2E_RESULT line the rule fires per package and the Slack contact point delivers the notification with the drill-down query (payload in the PR description). @greptileai please review the current head dca50c6

[yassin-berriai]

The misc / Run tests red is base drift, not this PR. The only failing test is tests/test_litellm/test_gpt_realtime_mode.py::test_get_model_info_reports_realtime_mode, which asserts get_model_info("gpt-realtime-mini")["mode"] == "realtime" but the model data resolves it to "chat". It fails identically on a clean litellm_internal_staging checkout with zero changes, and this PR is 100 percent under tests/e2e/ plus Grafana YAML (it touches no litellm/ code or model_prices_and_context_window.json). So it is failing on every open PR that merges current staging. The fix belongs in a separate PR on the base branch.

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_lit4551_remove_bob_the_builder (82d878e) with litellm_internal_staging (442fdc1)}

[yassin-berriai]

@greptileai please review the current head cbba2b0

[yassin-berriai]

The proxy-infra / Run tests red is a base-suite flake on current staging, not this PR. Different tests fail on each run (first run: mcp_server/outbound_credentials tests; rerun: tests/test_litellm/proxy/test_budget_reservation.py with "object MagicMock can't be used in await", a test-isolation leak), and all of them pass locally on this branch (test_budget_reservation.py: 51 passed). This PR is docs/e2e-only and touches none of those trees. It is red on several other open PRs right now (for example #33769, #33767, #33765) and green on others, which is the rotating flake signature. CI merges into current staging (advanced from my branch base e5a9f3f to 442fdc1), so the drift surfaces here.