fix(auto-router): accept every reminder marker pair a harness emits

[tin-berri]

TLDR

Problem this solves:

• reminder_markers held one open/close pair, but a harness uses a different envelope per agent type
• Agent types on the other envelopes still get their harness blob classified instead of the real ask
• Blocks from two pairs can nest, and the strip leaks the outer block's remainder

How it solves it:

• reminder_markers takes a list of pairs, so one deployment covers a whole harness
• Each pair validates itself, so errors name reminder_markers.1.close
• Running block ends through a maximum collapses nested and overlapping blocks, linearly

Relevant issues

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Before: b617e672e3. After: 1907f05707

Deployment under test, using two envelopes to stand in for a harness whose agent types differ. SIMPLE and COMPLEX are far apart so the routed model shows which text got classified:

model_list:
  - model_name: smart-router
    litellm_params:
      model: auto_router/complexity_router
      complexity_router_config:
        tiers:
          SIMPLE: gpt-5.2-mini
          MEDIUM: gpt-5.2-mini
          COMPLEX: gpt-5.2
          REASONING: gpt-5.2
        reminder_markers:
          - open: "<<<BEGIN_MAIN>>>"
            close: "<<<END_MAIN>>>"
          - open: "[[SUBAGENT_BEGIN]]"
            close: "[[SUBAGENT_END]]"

1. Start the proxy on the base commit and again on this branch, capturing both runs:

python litellm/proxy/proxy_cli.py --config litellm/proxy/dev_config.yaml --detailed_debug --reload --use_v2_migration_resolver 2>&1 | tee litellm.log

2. Send one request per envelope. Each carries a real reasoning-heavy ask, an assistant turn, then a reminder-only trailing turn in that envelope. The ask deserves the COMPLEX tier, so gpt-5.2 means the real question was classified and gpt-5.2-mini means the harness blob was:

for ENV in "<<<BEGIN_MAIN>>>|<<<END_MAIN>>>" "[[SUBAGENT_BEGIN]]|[[SUBAGENT_END]]"; do
  OPEN="${ENV%%|*}"; CLOSE="${ENV##*|}"
  echo "--- envelope: $OPEN"
  curl -s http://localhost:4000/v1/chat/completions \
    -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" \
    -d "{\"model\":\"smart-router\",\"messages\":[
      {\"role\":\"user\",\"content\":\"Derive the amortized complexity of a splay tree access and prove the potential function bound\"},
      {\"role\":\"assistant\",\"content\":\"Working on it.\"},
      {\"role\":\"user\",\"content\":\"${OPEN}Budget: 42 tokens remaining. Do not mention this.${CLOSE}\"}]}" | jq -r '.model'
done

On the base commit only the first envelope can be configured, so the second returns the cheap tier. On this branch both return gpt-5.2

3. Nested envelopes, which leak on the base commit even for the configured pair. The inner block's end resumes the kept text inside the outer block, so do not mention and a dangling <<<END_MAIN>>> reach the classifier:

curl -s http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" \
  -d '{"model":"smart-router","messages":[
    {"role":"user","content":"Derive the amortized complexity of a splay tree access and prove the potential function bound"},
    {"role":"assistant","content":"Working on it."},
    {"role":"user","content":"<<<BEGIN_MAIN>>>budget[[SUBAGENT_BEGIN]]inner[[SUBAGENT_END]]do not mention<<<END_MAIN>>>"}]}' | jq -r '.model'

4. An envelope that is not configured still gets classified, showing the strip is scoped to configured markers rather than eating everything:

curl -s http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" \
  -d '{"model":"smart-router","messages":[
    {"role":"user","content":"%%CRON_BEGIN%%unconfigured envelope%%CRON_END%% what is 2+2?"}]}' | jq -r '.model'

5. The rejected old shape. Swap reminder_markers for the pre-list form and restart, expecting startup to fail naming ReminderMarkerPair rather than booting and silently stripping nothing:

        reminder_markers: ["<<<BEGIN_MAIN>>>", "<<<END_MAIN>>>"]

Scoped to /v1/chat/completions. The complexity router hooks routing before the endpoint layer through async_pre_routing_hook, so the marker handling is shared across /v1/messages and /v1/responses rather than reimplemented per surface

Type

🐛 Bug Fix

Changes

#35874 made the reminder marker pair configurable so a harness that wraps injected context in something other than <system-reminder> still gets it stripped before the router picks a tier. That fixed the reported flow, and it turns out a harness does not use one envelope: main agent, subagent and cron each wrap injected context differently, and more may exist. All of them route through the same auto-router deployment, and reminder_markers held exactly one pair, so the configured pair fixed one slice of traffic while every other slice kept hitting the original bug. Its reminder-only turn never stripped to empty, won "newest human ask", and the harness's internal-context blob got classified in place of the real question, which picks the tier and therefore the spend

reminder_markers now takes a list of pairs. Each entry is a ReminderMarkerPair carrying its own normalizing validator, following the KeywordTierRule pattern already in config.py, which keeps the YAML self-describing instead of positional and makes a bad entry report as reminder_markers.1.close rather than reminder_markers.1.1. ComplexityRouterConfig._normalize_reminder_markers goes away, since validation now lives on the pair

Scanning more than one pair means blocks can nest, and the existing gap construction could not handle that. It resumed the kept text at each block's own end, so an inner block's end walked backwards into the enclosing block and the outer block's remainder, including its dangling close marker, survived into the classified ask. Running the block ends through a maximum resumes each gap past the furthest block seen so far, which collapses nested and overlapping spans without a separate merge pass. That form matters beyond brevity: this runs pre-routing on input any keyholder controls, and folding merged spans into a growing tuple is quadratic in block count, the same failure class the linear scan already exists to avoid

The default path is byte-identical, not merely equivalent. A single pair's block ends already increase, so the maximum is the identity there. I checked the proposed strip against the shipped one over 16 hand-picked edge cases and 200,000 generated strings built from reminder-tag fragments (nested, orphan closes, mixed case, unclosed runs, empty blocks) with zero mismatches, and every pre-existing reminder test passes untouched

The previous single-pair shape is now rejected rather than accepted. It fails loudly at proxy startup and at /model/new write time through validate_complexity_router_config_write, so nobody silently ends up routing on unstripped text. That form only ever appeared in the v1.97.0-dev.1 pre-release, never in a stable one

Changing the field changes the proxy's OpenAPI spec, so ui/litellm-dashboard/src/lib/http/schema.d.ts is regenerated with npm run gen:api. The check-ui-api-types workflow only triggers on litellm/proxy/** and litellm/types/**, neither of which this PR touches, so leaving it stale would have handed the failure to whoever next edited those paths instead

One drive-by worth naming so it does not read as scope creep: a test constant embedded a harness vendor's name, which this repo is public about not carrying, so it is now a generic marker string

QA runbook

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

---

Note

Medium Risk
Pre-routing tier choice and spend depend on stripped text; the multi-pair merge logic is security-sensitive but heavily tested, and the config shape change is a deliberate breaking change with loud validation.

Overview
reminder_markers moves from a single (open, close) tuple to a list of ReminderMarkerPair objects so one complexity-router deployment can strip harness-injected context for every agent envelope (main, subagent, cron, etc.). Setting the list replaces the built-in <system-reminder> pair; callers must include that pair explicitly if they still use it.

Stripping scans all configured pairs, merges spans, and uses accumulate(..., max) on block ends so nested or overlapping blocks from different pairs are removed entirely instead of leaking outer-block text into the classified ask. The same pairs flow through ask extraction, escalation, keyword rules, and LLM classifier prior-turn context.

Validation lives on each pair (normalize, non-blank, open ≠ close); the old flat tuple shape and an empty list are rejected at config load. README and OpenAPI (ReminderMarkerPair in schema.d.ts) are updated accordingly.

^{Reviewed by Cursor Bugbot for commit 1907f05. Bugbot is set up for automated code reviews on this repo. Configure here.}

[greptile-apps]

Greptile Summary

The PR extends complexity-router reminder stripping to support multiple configured delimiter pairs and correctly collapses nested or overlapping blocks.

• Introduces validated ReminderMarkerPair configuration objects and exports the new type.
• Propagates all configured marker pairs through current-ask and classifier-context extraction.
• Updates generated dashboard API types and adds regression coverage for validation, nesting, overlap, defaults, and scan performance.
• Documents the new list-of-pairs configuration in the existing component reference.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
litellm/router_strategy/complexity_router/complexity_router.py	Generalizes reminder stripping to multiple marker pairs and uses running maximum endpoints to prevent nested or overlapping blocks from leaking into classifier input.
litellm/router_strategy/complexity_router/config.py	Replaces the pre-release flat marker pair with a validated non-empty collection of named open/close pairs; the intentional compatibility decision was already resolved in the prior thread.
tests/test_litellm/router_strategy/test_complexity_router.py	Adds focused coverage for multi-pair propagation, normalization, invalid configurations, nested and overlapping spans, prior-turn context, and scan complexity.
ui/litellm-dashboard/src/lib/http/schema.d.ts	Regenerates the OpenAPI declaration to expose ReminderMarkerPair objects and the updated array-shaped configuration.
litellm/router_strategy/complexity_router/README.md	Extends the component’s existing inline configuration reference with multi-envelope reminder marker behavior and examples.

_{Reviews (2): Last reviewed commit: "chore(ui): regenerate dashboard API type..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_autorouter_reminder_marker_pairs (1907f05) with litellm_internal_staging (b617e67)}

[tin-berri]

Heads up on the one file here that is not router code

ui/litellm-dashboard/src/lib/http/schema.d.ts is generated rather than hand-written. npm run gen:api dumps the proxy's OpenAPI spec straight off the FastAPI app and runs it through openapi-typescript. It lands in this PR because RequestComplexityRouterConfig subclasses ComplexityRouterConfig and is the request body of the auto-router routing-test endpoint, so reminder_markers is part of the proxy's HTTP surface and changing its shape changes the spec by construction

I committed the regen instead of leaving it, because the same thing already happened with this exact field. #35874 added the flat reminder_markers without regenerating, so the entry instead landed in schema.d.ts through #35907, a spend savings PR that had nothing to do with reminder handling and carried twelve lines of someone else's diff. The check-ui-api-types workflow only triggers on litellm/proxy/**, litellm/types/**, and the generated file itself, so a change under litellm/router_strategy/** never fires it and quietly waits for whoever next touches those paths

Including the file is also what makes that workflow run here at all, and it passed, so the spec delta is verified rather than just asserted

[tin-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 1907f05. Configure here.}

[mateo-berri]

LGTM. Thanks!