test(e2e): register case 24 (Tier=real SKIP) + add case 25 backend error.type contract - #69
Merged
Conversation
## Case 24 (anthropic_beta_overrides_bedrock_gateway) — register The runbook landed in Wave 6d (commit 99f6ff1) as a .md file only — no data fixture, no README index entry, no case_NN function in the runner. After this commit, case 24 is recorded in the index as Tier=real and `case_24()` in the runner SKIPs unconditionally with a clear "Tier=real — requires Bedrock" message. Rationale: the behaviour case 24 verifies (Bedrock validation of the older tool-search-tool flag, after `anthropic_beta_overrides` rewrites the auto-injected advanced-tool-use header) cannot be driven by the mock — Bedrock's beta-flag validator is the whole point. Keep the manual runbook around for human / Claude execution against real Bedrock; skip in automation so the e2e summary still records it. ## Case 25 (backend auth error.type contract) — new Locks the wire contract that PR #68's frontend 401 redirect taxonomy depends on: every 401/403 response carries a structured `error.type` drawn from `ProxyErrorTypes`. Wave 6a added `_classify_auth_failure` and the UI reads its output to pick redirect vs inline error. Without an e2e gate a future upstream auth refactor could silently route exceptions around the classifier; unit tests on the classifier itself would still pass while the field disappears from the wire. The fixture runs four probes (one per ProxyErrorTypes value) against the running proxy: A1 no Authorization header → auth_invalid_credentials A2 malformed key (no sk- prefix) → auth_invalid_credentials A3 sk- key absent from DB → token_not_found_in_db A4 internal_user role + admin route → auth_permission_denied `auth_session_expired` is intentionally left to unit tests in test_auth_exception_handler.py — driving it deterministically would require a `duration:"1s"` key + a sleep, which is fragile under load. ## Verification ``` e2e/tools/run-all-cases --mock-only → Total: 24 Pass: 16 Fail: 0 Skip: 8 PASS 25 auth-error-type-contract: PASS: all 4 auth error.type contract probes hit expected values SKIP 24 anthropic-beta-overrides-bedrock-gateway (Tier=real — requires Bedrock) ``` Tier: B (internal e2e infra).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two-fer cleanup PR. Case 24 was orphaned (runbook only, no index/runner) since Wave 6d — register it as Tier=real SKIP. Case 25 NEW — locks the wire contract that PR #68's UI 401 taxonomy depends on (4 probes, one per ProxyErrorTypes value). E2E mock-only now: 16 PASS / 0 FAIL / 8 SKIP.