Skip to content

chore(e2e): add --user-id sticky routing + master test runner - #6

Merged
songkuan-zheng merged 1 commit into
ship/v1.83.10from
chore/e2e-user-id-and-runner
May 18, 2026
Merged

chore(e2e): add --user-id sticky routing + master test runner#6
songkuan-zheng merged 1 commit into
ship/v1.83.10from
chore/e2e-user-id-and-runner

Conversation

@songkuan-zheng

Copy link
Copy Markdown
Collaborator

Summary

Two related improvements to the e2e harness, plus a missing tooling piece.

  1. e2e/tools/call --user-id <id> — forwards to OpenAI user /
    Anthropic metadata.user_id, which the corp gateway uses for
    sticky upstream-key load balancing
  2. All Anthropic cases now pass --user-id so cache-related
    assertions are deterministic
  3. e2e/tools/run-all-cases — first-class master runner (was a
    /tmp scratch script); fixes a metric-snapshot bug; ships with
    per-case case_NN() functions for easy extension

Why --user-id matters

The corp Anthropic gateway (maasapi.corp.anispark.ai) round-robins
anonymous requests across multiple upstream API keys, each with its
own per-account cache namespace. Two byte-identical requests from
"no one" land on different upstream accounts; cache_read never hits.

We initially mis-diagnosed this as "gateway doesn't support cache
read". Prod evidence proves otherwise: spend_logs entries
e0ab6f96 and eec6a70d (same device_id=b0cb8e4d... in their
end_user blob) both hit cache_read=128k-135k tokens. Other
contemporaneous requests without device_id show cache_read=0.

Adding --user-id reproduces sticky routing in tests:

e2e/tools/call --provider anthropic --user-id user-42 \
    --cache ephemeral --ttl 5m --seed s1 --prompt-tokens 1500
# → first call: cache_creation=1827

e2e/tools/call --provider anthropic --user-id user-42 \
    --cache ephemeral --ttl 5m --seed s1 --prompt-tokens 1500
# → second call: cache_read=1827 ✓

Without --user-id the second call also returns cache_creation=1827
because the gateway routed it to a different upstream account.

Why the runner ships as a tool

The previous runner lived in /tmp — every reviewer who wanted to
verify the suite had to re-attach context to assemble the glue.
With it in-tree:

e2e/tools/run-all-cases               # full suite, ~$0.05 in providers
e2e/tools/run-all-cases --skip-paid   # only free cases for quick smoke

Exits 0 iff every case PASSes (SKIPs allowed).

Metric-snapshot bug in the old runner

The old snap() returned the last matching /metrics series. Once
later cases minted new virtual keys / teams, the "last matching line"
swapped from one series to another, and the before/after delta turned
nonsensical. Case 01 routinely false-failed.

New snap_sum() sums across all matching series; before/after stays
stable as the series set grows.

What changes

File Change
e2e/tools/call new --user-id flag → OpenAI user field
e2e/tools/run-all-cases new tool — master runner with snap_sum, --skip-paid, case_NN() structure, pre-flight proxy check
e2e/cases/data/03_prometheus_anthropic_read.sh new fixture (was inline in /tmp runner) — required GREEN, no longer SKIP-on-miss
e2e/cases/data/11_error_information_message_populated.sh poll spend_logs up to 20s instead of flat sleep 2 (async logger lag fix)
e2e/cases/{01,02,03,04,08,09}_*.md add --user-id to every Anthropic call
e2e/cases/README.md document run-all-cases + new-case SOP
e2e/README.md tools cheat-sheet adds run-all-cases + --user-id

Test plan

  • e2e/tools/run-all-cases13/13 PASS (no flakes across 3 runs)
  • Case 03 cache_read deterministically hits 1827 tokens with --user-id
  • Case 01 no longer false-FAILs when runner is invoked after case 09
    has minted virtual keys/teams
  • --skip-paid works: 05/06/13 marked SKIP, remaining 10 cases run
  • Black 24.10.0 formatting clean on e2e/tools/call
  • No code changes outside e2e/ — pure additive harness work

Out of scope

Two related improvements to the e2e test harness, both driven by
real-prod observations of the corp Anthropic gateway behavior:

1. `e2e/tools/call` learns `--user-id`. Forwarded to the proxy as the
   OpenAI `user` field, which litellm in turn maps to Anthropic's
   `metadata.user_id`. The corp gateway at maasapi.* uses this field
   for sticky upstream-key load balancing — requests sharing the
   same user_id land on the same upstream API key, so the second
   request's prefix can read the first request's cache write. Without
   sticky routing, the gateway round-robins anonymous requests across
   ~10+ upstream accounts each with its own cache namespace, and
   cache_read never hits in test scenarios.

   This was misdiagnosed initially as "gateway doesn't support cache
   reads". Prod logs show that requests carrying device_id in their
   end_user blob do achieve cache_read>0 (e.g. spend_logs entries
   e0ab6f96 and eec6a70d both hitting cache_read=128k-135k tokens
   under the same b0cb8e4d... device_id). Without device_id, the
   same gateway shows cache_read=0 on subsequent calls.

2. All Anthropic e2e cases (01, 02, 03, 04, 08, 09) now pass
   `--user-id` so cache-related assertions are deterministic
   regardless of upstream-account routing. Case 03 (cache READ) flips
   from SKIP-on-miss to required GREEN now that cache_read can be
   reliably triggered.

3. `e2e/tools/run-all-cases` ships as a first-class tool (was a
   /tmp scratch script). One PASS/FAIL/SKIP line per case, summary
   at the bottom, exits 0 iff every case PASSes (SKIPs allowed).
   Per-case logic split into `case_NN()` functions; adding a new case
   means dropping a fixture + adding one function + one invocation.

   Fixes a metric-snapshotting bug in the old runner: the previous
   `snap()` returned the LAST matching `/metrics` series, which silently
   compared two different series across before/after snapshots once
   later cases minted new virtual keys / teams. New `snap_sum()` sums
   across all series matching the label selector, giving stable totals
   even as the series set grows. Case 01 now passes cleanly through
   the runner.

   `--skip-paid` flag skips real-provider cases (05, 06, 13) for a
   ~$0 smoke pass against the harness itself.

4. `e2e/cases/data/11_error_information_message_populated.sh` polls
   spend_logs up to 20s instead of a flat 2s sleep — async logger lag
   was producing flake.

Test plan
  - `e2e/tools/run-all-cases` → 13/13 PASS
  - Case 03 cache_read deterministically hits 1827 tokens with --user-id
  - Case 01 no longer false-FAIL when runner is invoked after case 09
    has minted virtual keys/teams
  - Black 24.10.0 formatting clean on e2e/tools/call
@songkuan-zheng
songkuan-zheng merged commit f97a29e into ship/v1.83.10 May 18, 2026
1 check passed
@songkuan-zheng
songkuan-zheng deleted the chore/e2e-user-id-and-runner branch May 18, 2026 11:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant