chore(e2e): add --user-id sticky routing + master test runner - #6
Merged
Merged
Conversation
Two related improvements to the e2e test harness, both driven by
real-prod observations of the corp Anthropic gateway behavior:
1. `e2e/tools/call` learns `--user-id`. Forwarded to the proxy as the
OpenAI `user` field, which litellm in turn maps to Anthropic's
`metadata.user_id`. The corp gateway at maasapi.* uses this field
for sticky upstream-key load balancing — requests sharing the
same user_id land on the same upstream API key, so the second
request's prefix can read the first request's cache write. Without
sticky routing, the gateway round-robins anonymous requests across
~10+ upstream accounts each with its own cache namespace, and
cache_read never hits in test scenarios.
This was misdiagnosed initially as "gateway doesn't support cache
reads". Prod logs show that requests carrying device_id in their
end_user blob do achieve cache_read>0 (e.g. spend_logs entries
e0ab6f96 and eec6a70d both hitting cache_read=128k-135k tokens
under the same b0cb8e4d... device_id). Without device_id, the
same gateway shows cache_read=0 on subsequent calls.
2. All Anthropic e2e cases (01, 02, 03, 04, 08, 09) now pass
`--user-id` so cache-related assertions are deterministic
regardless of upstream-account routing. Case 03 (cache READ) flips
from SKIP-on-miss to required GREEN now that cache_read can be
reliably triggered.
3. `e2e/tools/run-all-cases` ships as a first-class tool (was a
/tmp scratch script). One PASS/FAIL/SKIP line per case, summary
at the bottom, exits 0 iff every case PASSes (SKIPs allowed).
Per-case logic split into `case_NN()` functions; adding a new case
means dropping a fixture + adding one function + one invocation.
Fixes a metric-snapshotting bug in the old runner: the previous
`snap()` returned the LAST matching `/metrics` series, which silently
compared two different series across before/after snapshots once
later cases minted new virtual keys / teams. New `snap_sum()` sums
across all series matching the label selector, giving stable totals
even as the series set grows. Case 01 now passes cleanly through
the runner.
`--skip-paid` flag skips real-provider cases (05, 06, 13) for a
~$0 smoke pass against the harness itself.
4. `e2e/cases/data/11_error_information_message_populated.sh` polls
spend_logs up to 20s instead of a flat 2s sleep — async logger lag
was producing flake.
Test plan
- `e2e/tools/run-all-cases` → 13/13 PASS
- Case 03 cache_read deterministically hits 1827 tokens with --user-id
- Case 01 no longer false-FAIL when runner is invoked after case 09
has minted virtual keys/teams
- Black 24.10.0 formatting clean on e2e/tools/call
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two related improvements to the e2e harness, plus a missing tooling piece.
e2e/tools/call --user-id <id>— forwards to OpenAIuser/Anthropic
metadata.user_id, which the corp gateway uses forsticky upstream-key load balancing
--user-idso cache-relatedassertions are deterministic
e2e/tools/run-all-cases— first-class master runner (was a/tmpscratch script); fixes a metric-snapshot bug; ships withper-case
case_NN()functions for easy extensionWhy
--user-idmattersThe corp Anthropic gateway (
maasapi.corp.anispark.ai) round-robinsanonymous requests across multiple upstream API keys, each with its
own per-account cache namespace. Two byte-identical requests from
"no one" land on different upstream accounts; cache_read never hits.
We initially mis-diagnosed this as "gateway doesn't support cache
read". Prod evidence proves otherwise: spend_logs entries
e0ab6f96andeec6a70d(samedevice_id=b0cb8e4d...in theirend_userblob) both hitcache_read=128k-135k tokens. Othercontemporaneous requests without device_id show
cache_read=0.Adding
--user-idreproduces sticky routing in tests:e2e/tools/call --provider anthropic --user-id user-42 \ --cache ephemeral --ttl 5m --seed s1 --prompt-tokens 1500 # → first call: cache_creation=1827 e2e/tools/call --provider anthropic --user-id user-42 \ --cache ephemeral --ttl 5m --seed s1 --prompt-tokens 1500 # → second call: cache_read=1827 ✓Without
--user-idthe second call also returnscache_creation=1827because the gateway routed it to a different upstream account.
Why the runner ships as a tool
The previous runner lived in
/tmp— every reviewer who wanted toverify the suite had to re-attach context to assemble the glue.
With it in-tree:
Exits
0iff every case PASSes (SKIPs allowed).Metric-snapshot bug in the old runner
The old
snap()returned the last matching/metricsseries. Oncelater cases minted new virtual keys / teams, the "last matching line"
swapped from one series to another, and the before/after delta turned
nonsensical. Case 01 routinely false-failed.
New
snap_sum()sums across all matching series; before/after staysstable as the series set grows.
What changes
e2e/tools/call--user-idflag → OpenAIuserfielde2e/tools/run-all-casessnap_sum,--skip-paid,case_NN()structure, pre-flight proxy checke2e/cases/data/03_prometheus_anthropic_read.she2e/cases/data/11_error_information_message_populated.shsleep 2(async logger lag fix)e2e/cases/{01,02,03,04,08,09}_*.md--user-idto every Anthropic calle2e/cases/README.mdrun-all-cases+ new-case SOPe2e/README.mdrun-all-cases+--user-idTest plan
e2e/tools/run-all-cases→ 13/13 PASS (no flakes across 3 runs)--user-idhas minted virtual keys/teams
--skip-paidworks: 05/06/13 marked SKIP, remaining 10 cases rune2e/tools/calle2e/— pure additive harness workOut of scope
user_idfrom prod clients (would actually raise cache hitrate and lower bills — recommend tracking as a prod config change,
not a harness change)
PR feat(router): backfill missing cost fields from canonical entry for known models #4 fixes this going forward but doesn't backfill)