Repository navigation
test: add Anthropic Messages API conformance suite using the real SDK - #1278
Merged
Merged
Conversation
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
Warning Review limit reachedNext included review available in 42 minutes. View limit detailsLimit details: You’ve used the included review currently available. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Review configuration: ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (7)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Owner
Author
Adversarial review (pipeline mode)Stream: CodeRabbit CLI -- SKIPPED. Stream: adversarial-pr-review Skill -- SKIPPED, structural capability gap, not a workaround. This subagent's toolset for this task is Read/Write/Edit/Bash only; there is no Skill tool available to invoke Stream: plain self-review (this pass, no specialized tool) -- ran manually against the diff ( Findings:
No blocking findings from this pass. Recommend merging once CI is green; the four filed issues (#1259, #1260, #1261, #1274) are the actual follow-up work, not blockers on this test-only PR. |
3 of 5 tasks
sakibsadmanshajib
left a comment
sakibsadmanshajib
left a comment
Owner
Author
There was a problem hiding this comment.
Ran the review pipeline's plain adversarial pass on this one; CodeRabbit and Codex are both rate limited/over quota tonight and skipped (confirmed from their own comments on this PR). Left a few inline notes below, nothing here changes my read that this is a real, substantive conformance suite worth having.
Runs the genuine anthropic Python SDK (never hand-rolled HTTP) against edge-api's /v1/messages surface: non-streaming and streaming messages.create, typed system content blocks with cache_control, multi-block content arrays, every tool_choice form plus a full tool-use round trip, sampling params via extra_body (the current SDK no longer exposes temperature/top_p/top_k as typed kwargs), thinking, vision, count_tokens, models.list, typed error classes for invalid model/bad params/auth failure/oversized body, and a provider-identity leak guard on both success and error paths. Four real defects were found this way and are tracked as xfail (strict=False, never silently skipped) referencing filed issues: GET /v1/models rejects the SDK's default x-api-key header and returns the wrong error envelope (#1259); empty completions serialize content as JSON null instead of [] (#1260); count_tokens 401s every API-key caller even though /v1/messages accepts the same key (#1261); and content_block_start omits an empty text field, crashing the SDK's own stream accumulator on the first delta of any text response (#1274). Verified live: built and ran the full containerized sdk-tests-py suite (docker compose run sdk-tests-py) against a throwaway Postgres seeded via scripts/ci-throwaway-db.sh + ci-seed-api-key.sh, the same recipe ci.yml's live-integration job uses, on a live edge-api/control-plane/litellm stack. 18 real passes, 4 tracked xfails, zero regressions to the existing 14 OpenAI-suite tests in the same run (32 passed, 4 xfailed total). HIVE_TOOLS_MODEL is added to the sdk-tests-py compose service so the new tool-use tests have a tools_supported alias to call, matching the pattern sdk-tests-js already uses (issue #1088).
Two real findings from PR review, both fixed at the root rather than
argued around:
- xfail markers were strict=False while the module docstring and PR
description both claimed strict=True ("fails loudly if the bug is
ever fixed without updating the test"). The code did not deliver
what the docstring promised: a fix to any of #1259/#1260/#1261/#1274
landing without a matching test update would have quietly become an
ordinary XPASS with no signal. Flipped all four decorators to
strict=True so the code now matches the stated, and correct, intent.
- `import httpx2` was undeclared: it only resolved because anthropic
1.2.0 happens to pull in httpx2 as a transitive dependency, with
nothing in this package's own pyproject.toml holding that in place.
This is the exact failure mode the comment directly above the new
anthropic pin in pyproject.toml warns about (openai 3.x's httpx to
httpx2 rename silently broke a direct httpx import here before).
Switched to the project's own already-pinned httpx, matching
test_error_shape.py's convention in the same package. Also dropped
an unused `import json` caught in the same pass.
No assertion logic changed; both fixes are import/decorator-only.
sakibsadmanshajib
force-pushed
the
test/anthropic-sdk-conformance
branch
from
August 28, 2026 21:50
844eccc to
c4ac56c
Compare
…eaking a provider name from /v1/models Two defects the live SDK conformance lane found the first time it actually ran on this branch, both of which made the suite red for real reasons. 1. A pooled alias turned the same request into a coin flip. Hive forwards four top-level fields verbatim from the Anthropic Messages surface because the OpenAI chat-completions shape has no equivalent for them: top_k, thinking, cache_control and session_id. That is right when the resolved upstream understands the field and fatal when it does not. hive-free load balances across providers with different tolerances, so an Anthropic call carrying top_k drew a member that answers with a hard 400 naming that field, and the customer saw "hive-free is not available." for a field the next member would have served. dispatchWithRetry now recognises a 400 that blames one of those four fields, strips just that field, and retries. Each field is stripped at most once, the strip only happens on an attempt that has a successor, and a 400 blaming a field the caller actually sent (temperature, tools, response_format) is never touched: dropping one of those would silently change the request and answer 200 for something nobody asked for. 2. model_aliases.summary is served verbatim as the description of every GET /v1/models entry, which makes it a customer-facing surface under the provider-blind rule. The two voice aliases named their upstream in prose, so every caller listing models read the provider name. The new migration rewrites both summaries, scoped by the exact old text so it cannot overwrite a summary an operator has since retuned. The stale part went with it: the text still named a model that was shut down on 2025-12-31. Buglog entry to be appended to main after this merges, per the buglog protocol in .claude/rules/openwolf.md: carried in the pull request body.
sakibsadmanshajib
added a commit
that referenced
this pull request
Aug 29, 2026
…#1274, #1260, #1261, #1259) (#1296) Closes #1274. Closes #1260. Closes #1261. Closes #1259. ## Are these one root cause or four? Two families, not one, and not four independent fixes. **Family A, Go zero-value JSON encoding dropping values the Anthropic contract requires to be present (#1274 and #1260).** Both are the gateway emitting a valid Go zero value where Anthropic requires an explicit empty one, and in both cases the Go-side assertion a normal test would write cannot see the defect at all: `len(content) == 0` is true for a nil slice and an empty one alike, and a `Text` field set to `""` looks correct right up until `encoding/json` runs. The mechanism differs (`omitempty` on a string in one, a nil slice with no tag in the other), so it is one class with two instances rather than a single line to change. **Family B, an auth model that only ever covered POST /v1/messages (#1261 and #1259).** Every other route on the Anthropic surface was left to inherit an auth path built for a different dialect. `count_tokens` is the only route here that does not delegate to the chat chain, so it never gained the API-key authority that chain provides, and `GET /v1/models` was never given the leaf `APIKeyNormalizer` that `/v1/messages` has carried since #954. The two share a cause but not a line of code. #1259 additionally carries a response-shape defect that belongs to neither family: the route answered an Anthropic client with the OpenAI list envelope and the OpenAI error envelope. Fixed here in the same pass, since fixing only the auth half would have moved the failure one step later. ## Ground truth API shapes were verified against the live specification on 2026-08-28, not from model memory, per the repository rule. - `https://docs.claude.com/en/docs/build-with-claude/streaming` shows `content_block_start` as `{"type":"text","text":""}` for a text block and `{"type":"tool_use","id":...,"name":...,"input":{}}` for a tool_use block, in every one of its five worked examples. - `https://docs.claude.com/en/api/models-list` shows the list envelope as `data` (entries of `type`/`id`/`display_name`/`created_at`) plus `has_more`, `first_id` and `last_id`, the last two typed "string or null". - The SDK's own accumulator (`anthropic/lib/streaming/_messages.py`, `accumulate_event`) was read directly to confirm the failure mechanism: it does `content.text += event.delta.text` for a `text_delta`, which is the `None += str` crash, and for an `input_json_delta` it **assigns** `content.input` from its own `jiter` buffer rather than reading the block-start value. That is why the missing `text` was fatal and the missing `input` was not, and it is why this PR treats the two differently in its claims. `openwolf bug search anthropic` and `openwolf bug search omitempty` both returned no prior entries. ## What changed | Issue | Change | | --- | --- | | #1274 | `StreamContentBlock.Text` is now `*string` so the empty string serializes, following the precedent `StreamEvent.Index` already set in this same struct for the identical reason. `Input json.RawMessage` added and set to `{}` on a tool_use block start. | | #1260 | `FromOAIResponse` seeds `Content` with an empty non-nil slice at construction and appends into it, which covers the early `len(resp.Choices) == 0` return as well as the normal path. | | #1261 | `anthropic.Deps.AuthorizeAPIKey` added; `count_tokens` accepts a JWT session principal or an API-key principal. Nil leaves it session-only and fail-closed. The refusal is the authorizer's own verdict run through the shared `WriteAuthFailure` status mapping and then reshaped into the Anthropic envelope, so no status logic is duplicated. | | #1259 | `GET /v1/models` is registered through a named `modelsHandler` applying `anthropic.APIKeyNormalizer` at the leaf and `anthropic.ModelsCompat` for shape. | ### Why the leaf normalizer on /v1/models is not redundant `authSelectorMiddleware` already wraps every `/v1/*` route in `APIKeyNormalizer` (#954), but only when JWT auth is wired: `handler = authSelectorMiddleware(jwtMW, handler)` sits behind `if jwtMW != nil`. On a deployment where Supabase JWT config is absent, edge-api logs "JWT auth wiring skipped" and mounts no selector at all, so nothing normalizes `x-api-key` and `handleModels` reads an empty `Authorization` header. `/v1/messages` kept working there purely because it carries its own leaf wrapper. That asymmetry is exactly the reported symptom in #1259 (same key, same connection, `/v1/messages` fine and `/v1/models` 401) and it is why the fix is a leaf wrapper rather than a change to the selector. ### Provider-blind invariant The `ModelsCompat` translation reads `id`, `created` and `name` and writes `type`, `id`, `display_name` and `created_at`. `owned_by` and every other OpenAI field are dropped rather than passed through, so this strictly reduces what reaches an Anthropic client. `TestModelsCompat_AnthropicClientGetsTheAnthropicListShape` asserts `owned_by` is absent from the body. Nothing on any path touched here emits a provider name, an upstream id or a `system_fingerprint`; the existing `TestHandleModelsDoesNotLeakProviderNames` and `TestSSETranslator_MessageID_NeverLeaksUpstreamID` guards are unchanged and still pass. ## Mutation check Every added test was proved load-bearing by reverting the fix it guards and observing it fail for the right reason, then restoring and observing it pass. Full transcript, six mutations, all red, then green on restore: | # | Mutation applied | Test | Result | | --- | --- | --- | --- | | 1 | `StreamContentBlock.Text` back to a plain `string`, call site back to `Text: ""` | `TestSSETranslator_TextBlockStartCarriesExplicitEmptyText` | FAIL: `text content_block_start omits the "text" key entirely ... map[type:text]` | | 2 | Delete `Input: json.RawMessage("{}")` from the tool_use block start | `TestSSETranslator_ToolUseBlockStartCarriesExplicitEmptyInput` | FAIL: `tool_use content_block_start omits the "input" key: map[id:call_1 name:get_weather type:tool_use]` | | 3 | Delete the `Content: []ResponseBlock{}` seed | `TestFromOAIResponse_EmptyCompletionSerializesContentAsArray` | FAIL on both subtests: `got {... "content":null ...}` | | 4 | `if h.deps.AuthorizeAPIKey == nil` forced to `if true`, i.e. count_tokens back to session-only | `TestHandler_CountTokens_AcceptsAPIKeyPrincipal`, `..._RejectedAPIKeyKeepsTheAuthorizersOwnRefusal` | FAIL: `want 200 got 401 body={"type":"error","error":{"type":"authentication_error","message":"missing user"}}`, and `error.message: want the authorizer's own refusal got missing user` | | 5 | `if !IsAnthropicClient(r)` forced to `if true`, i.e. never re-shape | `TestModelsCompat_*` (3 tests) | FAIL: OpenAI body passed through, `first_id`/`last_id` nil, `envelope: want top-level type=error got <nil>` | | 6 | `modelsHandler` unwrapped to `return handleModels(client, authorizer)` | `TestModelsHandlerServesAnAnthropicSDKClient` | FAIL: `x-api-key caller: want 200 got 401: {"error":{"message":"You didn't provide an API key...` | After restoring all six, `go test ./apps/edge-api/internal/anthropic/... ./apps/edge-api/cmd/server/...` is `ok` for both packages, and the wider `go test ./apps/edge-api/... -count=1 -short` passes with no failures. `gofmt -l` reports none of the touched files. Four of the added tests are deliberately non-regression guards rather than defect guards, and do not go red under any mutation above. The count said two on first posting and was corrected in review; the table is the record anyone reads later, so it should say what is actually true. - `TestModelsCompat_OpenAIClientIsUntouched` and `TestModelsHandlerKeepsTheOpenAIShapeForOpenAIClients` exist to fail if a later change starts re-shaping unconditionally and empties Open WebUI's model picker. Both compare byte-identical bodies, so the inverse mutation of always reshaping does turn them red. - `TestHandler_CountTokens_WithoutAnAPIKeyAuthorityFailsClosed` stays green under mutation 4, since a handler wired without an API-key authority writes 401 either way. It pins the fail-closed default and goes red if someone makes a nil authority permissive. - `TestHandler_CountTokens_SessionPrincipalDoesNotConsultTheAPIKeyAuthority` also stays green under mutation 4, since the session branch returns before the nil check is reached. It goes red if the two principal checks are ever reordered. `TestIsAnthropicClient` is a unit test of a new pure function that none of the six mutations touch, so it is neither a defect guard nor a non-regression guard for anything in the table. ## Review round two Three findings from review, all fixed in `25b2820`, all in `apps/edge-api/internal/anthropic`. **Refusal retry headers were being dropped.** `apierrors.WriteAuthFailure` is the shared source of truth precisely so a retryable 429 is never collapsed into a non-retryable refusal, and it delivers the retryable part through headers rather than the body: `retry-after` plus the four `x-ratelimit-*` values on a real 429, and `retry-after` on both the degraded-limiter branch and the `upstream_unavailable` branch. Recording the refusal in a `headerlessRecorder` and reshaping only its body threw all of that away. `count_tokens` lost it latently, and `GET /v1/models` lost it live through `authorizeAliasRequest`, which also left the two client shapes disagreeing about retry metadata for the same refusal on the same route. `headerlessRecorder.reshapeInto` now carries the headers over and reshapes in one call, so a call site cannot take one half of a refusal without the other. `Content-Type` and `Content-Length` are deliberately not carried: the reshaped body is a different envelope of a different length, and a stale `Content-Length` would truncate it on the wire. Everything that is forwarded is retry metadata and carries no provider identity, so the provider-blind invariant is unchanged. **`GET /v1/models` served two representations for one URL without declaring it.** Nothing on the route sets `Cache-Control` and the route needs a credential, so no correct cache stores it today and no live exploit exists. The declaration is what keeps a later edge cache in front of Caddy, or an intermediary keying on URL alone, from handing an Anthropic-shaped body to Open WebUI and emptying its model picker, which is the regression `TestModelsCompat_OpenAIClientIsUntouched` exists to prevent. `Vary: anthropic-version, x-api-key` is now set on both branches, before either writes. **`Finish` could emit `message_delta` before any `message_start`.** `Translate` calls `Finish` unconditionally and `Finish` did not check `t.started`, so an upstream stream that yielded no parseable chunk at all produced a terminal pair with nothing before it, and the SDK accumulator raises an unexpected-event-order error when a `message_delta` arrives while its snapshot is still nil. Review suggested a follow-up issue for this rather than widening the PR; the fix turned out to be three lines in a file this PR already changes, which is smaller than the issue describing it, so it is taken here. `FeedLine` and `Finish` now share one `ensureStarted`, which emits `message_start` exactly once per stream whichever of them reaches it first. Four more mutations, each applied alone, each restored after: | # | Mutation applied | Test | Result | | --- | --- | --- | --- | | 7 | `reshapeInto`'s header loop forced to skip every key | `TestHandler_CountTokens_RefusalCarriesTheAuthorizersRetryHeaders`, `TestHandler_CountTokens_UpstreamUnavailableCarriesRetryAfter`, `TestModelsCompat_RefusalCarriesTheRetryHeaders` | FAIL: `retry-after: want 30 got ""`, `retry-after: want 5 got ""`, and the three `x-ratelimit-*` assertions on both routes | | 8 | `Content-Length` removed from the skip list, so the delegated body's length is forwarded | `TestModelsCompat_RefusalCarriesTheRetryHeaders` | FAIL: `content-length: the delegated body length must not describe the reshaped one, got "4096"` | | 9 | The `Vary` line deleted | `TestModelsCompat_DeclaresVary` | FAIL on both subtests: `vary: want both request headers that select the representation, got ""` | | 10 | `ensureStarted` removed from `Finish` | `TestSSETranslator_EmptyStreamStillOpensTheMessage` | FAIL: `first event: want message_start got message_delta` | `TestSSETranslator_FinishAfterAStartedStreamDoesNotRepeatMessageStart` is a fifth non-regression guard: it stays green under all ten mutations and exists so that opening the message from `Finish` can never add a second `message_start` to a stream that already carried one. ### Adversarial review of the round-two fixes The round-two commit went back through the review pipeline before this was called done. Two findings, both fixed in `f0e4c56`. **The `Vary` declaration used `Set` where it should use `Add`.** Nothing in edge-api declares a `Vary` on this route today (`grep` finds exactly one writer, the new line itself), so overwriting was harmless in the current tree and wrong the moment an outer middleware declares one of its own, a CORS layer setting `Vary: Origin` being the obvious case. The wrapper is now additive. **The 2xx branch of `ModelsCompat` dropped every header the delegated handler set.** That path re-encodes the body rather than reshaping it, which is exactly why the carry-over the refusal path had just gained was easy to leave out of it. A header set alongside a success is as much part of that response as the body is, and this route is where a 2xx rate-limit budget would surface. The header copy is now its own method on the recorder, `copyHeadersTo`, and both branches call it. | # | Mutation applied | Test | Result | | --- | --- | --- | --- | | 11 | `Vary` back to `Set` | `TestModelsCompat_VaryDoesNotClobberAnExistingDeclaration` | FAIL: `vary: want "Origin" preserved, got "anthropic-version, x-api-key"` | | 12 | `rec.copyHeadersTo(w)` deleted from the success branch | `TestModelsCompat_SuccessCarriesTheDelegatedHeaders` | FAIL: `x-ratelimit-remaining-requests: want 42 got ""` | ### On `display_name` and the provider-blind edge Review confirmed independently that the `ModelsCompat` translation is provider-blind (it reads `id`, `created` and `name` only, so `owned_by` and `description` cannot ride along) and flagged the remaining edge: `display_name` is `model_aliases.display_name`, free text, and one seeded row already reads `Openrouter Auto (Task Aware)`. That row is `visibility='internal'` today, so the `visibility IN ('public', 'preview')` predicate keeps it off both list paths, but the migration that added it describes a later flip to public as a one-line follow-up. Nothing here closes #1284, and this PR never claimed to: the OpenAI branch is unchanged and still ships the `hive-stt` and `hive-tts` descriptions that #1284 reports. The `display_name` edge is recorded as a comment on #1284 rather than a new issue, since it is the same route and the same family, and the guard it asks for (assert no listed alias's `display_name` or `summary` matches a known provider name) belongs where the catalog is listed, not in this translation. ## Effect on #1278 This should flip all four `xfail` markers in `packages/sdk-tests/python/tests/test_anthropic_messages.py` to real assertions. Not touched here, since that branch is not mine to edit. - `test_streaming_event_sequence_integrity` (#1274) - `test_tool_choice_none_forbids_tool_use` (#1260) - `test_count_tokens` (#1261) - `test_models_list` (#1259) One caveat on the last one. `client.models.list()` will now authenticate and return the Anthropic list envelope, but the SDK's typed page model may expect fields this gateway has no source for. If that assertion still fails after this merges, it is a narrower follow-up about specific fields, not the auth and envelope defects #1259 reported. ## Scope deliberately not taken #1260 also observes that a zero-content streaming turn emits `message_start`, `message_delta`, `message_stop` with no `content_block_start`/`stop` pair. That is left alone: the streaming path already emits `"content":[]` in `message_start`, so an SDK folding the stream back into a message gets an empty array rather than null, and Anthropic's own specification does not require a content block on a turn that produced no content. The reported crash is in the non-streaming builder, which is what this changes. The neighbouring case where `message_start` never fires at all is a different defect, was not what #1260 reported, and is now fixed here rather than deferred. See Review round two above. ## Test plan - [x] `go test ./apps/edge-api/... -count=1 -short` through the toolchain container, green, re-run after the review-round-two fixes - [x] `gofmt -l apps/edge-api` clean for every touched file - [x] Twelve-way mutation check, all three tables above - [ ] CI required checks green - [ ] `packages/sdk-tests` Anthropic conformance suite re-run against a live stack once #1278 merges, to confirm the four xfails flip ## Buglog entry ```json {"id":"bug-2026-08-28-anthropic-sdk-wire-conformance","date":"2026-08-28","title":"Anthropic surface broke the real SDK four ways: content_block_start dropped text, empty completions serialized content null, count_tokens 401d API keys, /v1/models rejected x-api-key and answered OpenAI-shaped","error_message":"TypeError: unsupported operand type(s) for +=: 'NoneType' and 'str' in anthropic/lib/streaming/_messages.py accumulate_event; TypeError: 'NoneType' object is not iterable on msg.content; anthropic.AuthenticationError 401 {\"type\":\"error\",\"error\":{\"type\":\"authentication_error\",\"message\":\"missing user\"}} on count_tokens; anthropic.AuthenticationError 401 invalid_api_key on models.list()","root_cause":"Two families. (1) Go zero-value JSON encoding dropped values the Anthropic wire contract requires to be explicitly present: omitempty on StreamContentBlock.Text dropped the required \"text\":\"\" from every text content_block_start, and a nil []ResponseBlock in FromOAIResponse marshaled to null instead of []. Neither is visible to a Go-side assertion, since len() cannot distinguish nil from empty and a \"\" field looks correct until encoding/json runs. (2) The Anthropic surface's auth model only ever covered POST /v1/messages: count_tokens is the only route that does not delegate to the chat chain so it never gained an API-key authority and recognised session principals only, and GET /v1/models never got the leaf APIKeyNormalizer /v1/messages carries, which matters because the global one in authSelectorMiddleware only exists when jwtMW is non-nil.","fix":"StreamContentBlock.Text became *string and gained Input json.RawMessage set to {} on tool_use starts, matching the live streaming spec. FromOAIResponse seeds Content with an empty non-nil slice at construction so both exits are covered. anthropic.Deps.AuthorizeAPIKey added and count_tokens accepts either principal, fail-closed when nil, refusals reshaped from the shared WriteAuthFailure mapping. GET /v1/models registered through modelsHandler applying APIKeyNormalizer at the leaf plus a ModelsCompat wrapper that answers Anthropic-shaped callers with the Anthropic list and error envelopes while leaving OpenAI callers byte-identical.","tags":["anthropic","sdk-conformance","edge-api","json-encoding","omitempty","auth","streaming","provider-blind"],"issues":[1274,1260,1261,1259],"pr":"fix/anthropic-stream-content-block-start"} ```
sakibsadmanshajib
added a commit
that referenced
this pull request
Aug 29, 2026
## Summary This is the batched buglog follow-up for the 21 pull requests merged during the 2026-08-28 session. Its diff is `.wolf/buglog.jsonl` and nothing else. Per `.claude/rules/openwolf.md`, every fixed bug, error, failed test or failed build must be logged, but the line may never be appended on a fix branch: `merge=union` in `.gitattributes` resolves concurrent appends locally and is ignored by GitHub's server side merge, so two branches that both appended land in hard conflict there. An unmergeable pull request gets no `refs/pull/N/merge`, no `pull_request` run and therefore zero checks, and the required status gate then blocks the merge for a reason the page never states (issue #873). Each fix accordingly carried its entry in its own pull request body, and this pull request copies them onto `main` in one batch, which the protocol explicitly prefers over one pull request per entry. ## Source pull requests 1240, 1251, 1253, 1257, 1268, 1276, 1277, 1278, 1281, 1287, 1292, 1293, 1294, 1296, 1300, 1301, 1303, 1305, 1313, 1335, 1337. All merged. Every entry came from a "Buglog entry" heading in one of those bodies. Nothing was invented for a pull request that carried none. ## What landed 36 entries appended, one JSON object per line, append only. The 196 pre-existing lines are byte identical to `origin/main`. | Source | Entries | |---|---| | #1240 | 1 | | #1251 | 1 | | #1253 | 1 | | #1257 | 4 | | #1268 | 5 | | #1276 | 2 | | #1277 | 1 (of 2 in the body) | | #1278 | 0 (merged into #1296) | | #1281 | 2 | | #1287 | 2 | | #1292 | 3 | | #1293 | 1 | | #1294 | 3 | | #1296 | 1 | | #1300 | 1 | | #1301 | 1 | | #1303 | 1 | | #1305 | 3 | | #1313 | 1 | | #1335 | 1 | | #1337 | 1 | Note on #1268: its first "Buglog entry" heading says "None yet" in prose and carries no JSON. Its two later headings, from the CI live lane and from the intermittent tool call failure, carry the five entries taken here. ## Deduplication - **#1278 dropped, folded into #1296.** Both describe the same defect: `omitempty` on `StreamContentBlock.Text` dropped the required `"text":""` from every text `content_block_start`, crashing the real Anthropic SDK's stream accumulator (issue #1274). #1278 is the conformance suite that found it and shipped it marked xfail; #1296 is the fix, and its entry carries the fuller root cause and the actual remedy. One bug, one entry. #1296's entry gains a `discovered_by` field naming #1278 so the discovery is not lost. - **#1277's first entry dropped.** The same body carries a later "Buglog entry (revised)" heading written after the review round found the page's claims did not match what the code enforces. The revised entry is the one taken. - Checked and kept as distinct: #1313 and #1337 are two different hooks (`decision-citation-check.js` and `secrets-scanner.js`) blind to the same MultiEdit payload shape, fixed in two different pull requests, so two entries. #1240 and #1335 are two different `account_not_provisioned` defects, one an observability gap at the edge boundary and one a console mint that should have refused, so two entries. #1305's three entries are three separate rounds of defects in the same money path change, each with its own root cause. ## Corrections against what actually merged Each entry was checked against the merged tree at `origin/main`, not against its own claim. - **#1240.** The entry said the log line went in at `AuthSnapshot.TenantUUID`. On `main` the check is the exported `authz.ParseTenantID(TenantLookup)` that `TenantUUID` delegates to, which the images and audio routing adapters (two further silent call sites found in the same review) also call, and `key_id` is deliberately not logged because CodeQL's clear text logging check flags any field named `*Key*` (alert #31). The `fix` field now says so. - **#1276, first entry.** The entry named `app/console/analytics/page.tsx` as the home of the five fetch helpers and their `Promise.all`. On `main` they live in `apps/web-console/lib/analytics/overview-fetch.ts`, extracted during review. Path corrected. - **#1277.** Its `error_message` was the placeholder `n/a`. Reconstructed from the pull request's own correction narrative: the page as first written published a blanket no content stored claim false for `/v1/batches`, `/v1/files` and `/v1/rag`, a product wide provider blindness claim disproved by catalogue summaries that name vendors (#1284), a metering claim anchored to the console side `UsageEventRow` projection rather than the `usage_events` table, and a 1:1 alias to route claim that is a property of seed data rather than of `SelectRoute`. **#1303 needed no correction.** Its original root cause asserted a live mid stream provider leak on the session chat relay that measurement disproved, and the author had already corrected the body before merge. The corrected version is what was taken, including the sentence recording that the session chat relay did not leak an error frame but silently truncated instead. Every other entry's central claim was verified present in the merged tree, among them `metering.SupportsIncludeUsage`, `sanitize.VariablePriceFrame` in the batch dispatcher, the revoke and regrant in `20260828_01_service_role_public_schema_grant.sql` with the `anon` assertions in `ci-throwaway-db.sh`, `normalizeReasoningUsage` now called from `normalizeChatCompletion`, `signup.SyncTenantMembershipRole`, `TestKeyViewHidesALimitThatIsNotEnforced`, `TestListEventsLatencyCrossesTheWire`, `mask-api-keys.mjs` and `md-table.mjs`, `StreamContentBlock.Text` as `*string`, `redactSnapshot`, `httpx.ReadBody`, `sanitize.ReplaceErrorFrame` with the default deny tail in `provider_blind.go`, `pinCompletionCeiling` and `captureInputTokens` with `applyReasoningHeadroom` gone, `requireBillingTenant`, and `hooks.selfcheck.js` wired into the Repo policy lints check. ## Verification - `node .wolf/hooks/bugstore.selfcheck.js` reports `bugstore selfcheck OK`. - All 232 lines parse as a single JSON object each. - Every appended entry carries `error_message`, `root_cause`, `fix` and `tags`. - Scanned for credentials: no API key, bearer token, JWT, password, AWS key or Postgres DSN with a password appears in any entry. The `hk_` occurrences are prefix descriptions in prose, not keys. - `git diff origin/main...HEAD --name-only` prints `.wolf/buglog.jsonl` and nothing else. No `.wolf/` telemetry was staged. ## Review No adversarial review streams were run, deliberately. This change is records only: it adds no code, no test, no configuration and no behavior, and `.wolf/buglog.jsonl` is on the inert path allowlist in `.github/workflows/ci.yml`, so the six required checks report green without running their heavy steps. If a check does fail here, that is a real signal about the file rather than about the pipeline. https://claude.ai/code/session_01WyEwUxZCArdn1ZUDkTvuQ1 Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
4 of 5 tasks
sakibsadmanshajib
added a commit
that referenced
this pull request
Aug 29, 2026
## Summary This is the batched buglog follow-up for the pull requests merged to `main` on 2026-08-29. Its diff is `.wolf/buglog.jsonl` and nothing else. Per `.claude/rules/openwolf.md`, every fixed bug, error, failed test or failed build must be logged, but the line may never be appended on a fix branch. `merge=union` in `.gitattributes` resolves concurrent appends locally and is ignored by GitHub's server side merge, so two branches that both appended land in hard conflict there. An unmergeable pull request gets no `refs/pull/N/merge`, no `pull_request` run and therefore zero checks, and the required status gate then blocks the merge for a reason the page never states (issue #873). Each fix accordingly carried its entry in its own pull request body, and this pull request copies them onto `main` in one batch, which the protocol explicitly prefers over one pull request per entry. ## Scope examined Fifty nine pull requests merged to `main` on 2026-08-29. Forty eight of them carried at least one entry, for eighty two entries in total. Thirty two of those were already on `main` and are skipped, leaving fifty appended here from thirty four pull requests. The largest block of skips comes from #1342, the equivalent batch for the 2026-08-28 merges, which merged earlier the same day and already landed thirty six entries covering #1257, #1268, #1276, #1277, #1287, #1292, #1293, #1294, #1296, #1301, #1303, #1305, #1313, #1335 and #1337. ## What landed Fifty entries appended, one JSON object per line, append only. The 232 pre-existing lines are byte identical to `origin/main` (verified by hashing the first 232 lines of the result against the base file). Every line in the resulting file parses as JSON and carries `error_message`, `root_cause`, `fix` and `tags`. | Source | Entries | |---|---| | #1083 | 2 | | #1277 | 1 | | #1278 | 1 | | #1298 | 1 | | #1334 | 1 | | #1336 | 3 | | #1343 | 1 | | #1346 | 1 | | #1351 | 1 | | #1365 | 2 | | #1368 | 1 | | #1369 | 1 | | #1371 | 3 | | #1375 | 3 | | #1376 | 1 | | #1378 | 1 | | #1379 | 2 | | #1388 | 5 | | #1389 | 3 | | #1390 | 2 | | #1393 | 1 | | #1394 | 1 | | #1410 | 1 | | #1417 | 1 | | #1421 | 1 | | #1423 | 1 | | #1424 | 1 | | #1426 | 1 | | #1429 | 1 | | #1431 | 1 | | #1433 | 1 | | #1434 | 1 | | #1436 | 1 | | #1439 | 1 | Entries are copied verbatim from their source pull request bodies. Nothing was rewritten, no field was invented, and no field was added. No JSON needed repair: all eighty two extracted entries parsed on the first attempt and all four required fields were present on every one. ## Merged pull requests that carried no entry Eleven of the fifty nine. Recorded here because the gap is itself the useful signal. | Pull request | Title | Assessment | |---|---|---| | #1013 | chore(deps): bump the go-minor-patch group across 1 directory with 4 updates | Dependabot bump, no defect fixed, no entry expected | | #1015 | chore(deps): bump the go-minor-patch group across 1 directory with 6 updates | Dependabot bump, no entry expected | | #1016 | chore(deps): bump golang from 1.26-alpine to 1.27-alpine in /deploy/docker | Dependabot bump, no entry expected | | #1218 | chore(deps): bump postcss from 8.5.19 to 8.5.26 in /apps/desktop | Dependabot bump, no entry expected | | #1219 | chore(deps): bump golang.org/x/crypto from 0.41.0 to 0.52.0 in /apps/control-plane | Dependabot bump, no entry expected | | #1342 | chore: batch buglog entries for the 2026-08-28 merges | The previous batch pull request itself, correctly carries no entry of its own | | #1364 | chore: remove four dead skills and record the patterns that cost time | Protocol gap. The body records patterns that cost time, which is the shape of a buglog entry, but none was written as one | | #1383 | test: retire stale expected-failure markers, restore the ones that are true (#1381, #1382, #1324) | Protocol gap. Stale `it.fails` markers reading as red is a real defect that was fixed here and should have carried an entry | | #1384 | docs: correct D-047, hive-auto reverted to variable pricing (D-059) | Decision ledger correction, arguably a documentation defect, no entry written | | #1387 | chore(deps): bump next from 15.5.23 to 16.3.3 in /apps/agent-console | Dependabot bump, no entry expected | | #1398 | docs: rescue the 2026-08-25 parity captures and add the 2026-08-29 QA matrix evidence | Documentation and evidence rescue, no entry written | Six of the eleven are Dependabot bumps and one is the previous batch, so the genuine protocol gaps are #1364, #1383, #1384 and #1398. Of those, #1383 is the one worth a follow-up: it fixed a real defect class (a stale expected-failure marker reads as a red "Expect test to fail" and gets dismissed as pre-existing) and left no record. ## Entries skipped as already present Thirty two. Thirty of them matched an entry already on `main` on `error_message`, `id` or `fix`. Two more from #1278 are semantic duplicates that an exact match would have missed, and were skipped after reading the landed entries they duplicate: - #1278's `streaming content_block_start omits text field` entry is covered by the consolidated `bug-2026-08-28-anthropic-sdk-wire-conformance` entry landed from #1296, whose root cause names the same `omitempty` on `StreamContentBlock.Text`. - #1278's `GET /v1/models leaked an upstream provider name` entry is covered by `BUG-1284`, landed from #1300, which names the same `public.model_aliases.summary` publication path. #1278's third entry, on `top_k` forwarding producing a 400, is not covered anywhere on `main` and is appended here. #1342 recorded #1278 as fully "merged into #1296", which was accurate for two of its three entries. ## Note on entry quality One appended entry is thin: #1277's parity re-score record carries `error_message` of `n/a` and a root cause of "console had no privacy/data-policy surface at all". It is a parity gap record rather than a defect record. It is included exactly as written rather than embellished, per the protocol's preference for the author's own words. ## Test plan - [x] Branch cut fresh from `origin/main`, diff is `.wolf/buglog.jsonl` and nothing else - [x] First 232 lines byte identical to the base file (md5 match) - [x] All 282 resulting lines parse as JSON and carry `error_message`, `root_cause`, `fix` and `tags` - [x] No `.wolf/` telemetry (`anatomy.md`, `memory.md`, `token-ledger.json`, `hooks/_session.json`, `buglog.json`) in the commit - [ ] The six required checks report green via the inert path allowlist in `.github/workflows/ci.yml` --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a committed, runnable Anthropic Messages API conformance suite that drives the genuine, PyPI-published
anthropicPython SDK (1.2.0, never a hand-rolled HTTP request) againstedge-api's/v1/messagessurface:packages/sdk-tests/python/tests/test_anthropic_messages.py, alongside the existing OpenAI-compatible suites in the same package and the same--profile testcompose job.No Claude model exists in Hive and none will be added (owner decision 2026-08-26, DeepSeek and Groq only). The suite exercises Hive's own aliases (
hive-free,deepseek-v4-flash) through the Anthropic wire shape; aclaude-*model name answering 404 is asserted as expected, correct behavior, not a gap (test_invalid_model_is_not_found_not_a_defect).What it covers
messages.create: non-streaming and streamingsystemas a plain string and as a typed content-block array withcache_controlcontentarraystools+ everytool_choiceform (auto,any,tool,none), a full tool-use -> tool-result round trip,disable_parallel_tool_usemax_tokens,stop_sequences,metadata.user_idas typed kwargs;temperature/top_p/top_kviaextra_body, see note below),thinking(documented as silently ignored, no configured provider supports it)messages.count_tokens,models.listNo test is skipped for missing credentials (
HIVE_API_KEYfalls back to a literal that simply fails loudly with a realAuthenticationError, matching every other file in this package).HIVE_TOOLS_MODELis added to thesdk-tests-pycompose service (previously onlysdk-tests-jshad it) so the tool-use tests have atools_supportedalias to call;tools/lint-sdk-test-env-propagation.mjscovers this.A note on the real, current Anthropic SDK
Running the actual SDK (not reasoning from memory) surfaced that
messages.create()onanthropic==1.2.0no longer exposestemperature,top_p, ortop_kas typed keyword arguments at all -- they raiseTypeError: unexpected keyword argument. The suite reaches them through the SDK's documentedextra_bodyescape hatch instead, andpyproject.tomlpinsanthropic>=1.2.0,<2.0.0so this stays version-matched to what was actually exercised.Four real defects found, each filed and tracked as
xfail(strict=False)(never silently skipped)GET /v1/modelsdoesn't accept the SDK's defaultx-api-keyheader (onlyAuthorization: Bearerworks) and answers 401 with the OpenAI error envelope instead of the Anthropic one when it does reject.tool_choice: {"type":"none"}) serializes"content":nullinstead of Anthropic's documented"content":[]; any client that assumescontentis iterable breaks. Root cause: a nil Go slice with noomitemptyinFromOAIResponse.POST /v1/messages/count_tokens401s every request authenticated with a valid API key ("missing user"), while the sibling/v1/messagesaccepts the identical key fine.handleCountTokensonly recognizes a session principal.content_block_startomits an emptytextfield (json:"text,omitempty"), which crashes the real SDK's ownmessages.stream()accumulator withTypeErroron the very firstcontent_block_deltaof any plain-text response. This is not an edge case; it reproduces on a plain "count to five" prompt every time.Each
xfailcarries the SDK call, the exact exception, and the issue number, and will fail the suite loudly (not silently regress) if the underlying bug is ever fixed without updating the test.Two things specifically asked about
ResponseUsage/StreamUsageuse Anthropic-native EXCLUSIVE accounting (input_tokensexcludes cache;freshInputTokenssubtracts cache read/write from the upstream's inclusiveprompt_tokens, clamped and alarmed on going negative).message_delta.usageis emitted exactly once per stream and is cumulative (final totals), never incremental -- asserted directly intest_streaming_event_sequence_integrity.message_start's usage fields are0/omitted rather than Anthropic's real 1-3 placeholderoutput_tokens; harmless either way since both must never be treated as final, but noted as a minor wire-shape difference.maxBodyBytesinhandler.go) truncates the raw body viaio.LimitReaderbeforejson.Unmarshalever runs, so a body over that ceiling genuinely is invalid JSON once truncated -- the "invalid JSON body" message is accurate for what was actually parsed, not misleading for a well-formed request that merely arrived too large.test_oversized_body_is_bad_request_not_a_hang_or_500asserts the SDK gets a typed 400, not a hang or a 500.Verification
Built and ran the full containerized
sdk-tests-pysuite (docker compose --profile test run --rm sdk-tests-py pytest -v, the exact shapeci.yml's live-integration job uses) against a throwaway Postgres seeded viascripts/ci-throwaway-db.sh+scripts/ci-seed-api-key.sh(same recipe CI uses) on a liveedge-api/control-plane/litellmstack, in my own worktree with an isolatedCOMPOSE_PROJECT_NAME. Result: 32 passed, 4 xfailed, zero regressions to the 14 pre-existing OpenAI-suite tests in the same run.Live-gateway verification against
https://api-hive.scubed.cowith a self-minted console key (the method requested) was attempted but blocked by tooling available to this session: the console's sign-up flow requires solving a live Cloudflare Turnstile challenge in a real browser, and no browser-automation tool was available to this agent; the root.env'sSUPABASE_URL/SUPABASE_JWT_ISSUERalso point at the Supabase Cloud project that was decommissioned in the self-hosted cutover, so GoTrue-admin session minting (live-auth.mjs) is unreachable from this sandbox too. The throwaway-stack verification above runs the identicaledge-apibinary/handler code the live gateway serves, seeded the same way CI already does, so the findings are substrate-accurate for the code under review even though the literal live hostname was not hit in this pass.Buglog entry
Per
.claude/rules/openwolf.md, not appended to.wolf/buglog.jsonlon this branch. Entry for the follow-up buglog-only PR:{"timestamp":"2026-08-28T20:40:00Z","error_message":"streaming content_block_start omits text field, crashing the real Anthropic SDK's own stream accumulator on every text response","root_cause":"StreamContentBlock.Text carries json:\"text,omitempty\" in apps/edge-api/internal/anthropic/types.go; a text content_block_start always constructs Text:\"\", so omitempty drops the key entirely, leaving the SDK's typed content.text as None before the first += delta append","fix":"tracked as issue #1274, not yet fixed on this branch (test-only PR, marked xfail)","tags":["anthropic","streaming","sdk-conformance","edge-api"]}Second and third entries, from the live run this pull request finally got
The first live run of this suite (33214262199) went red on two real defects, both fixed in commit d52c3c5. Run 33219513264 is green: 26 passed in the JavaScript suite, 32 passed and 4 xfailed in the Python suite, 5324 tokens metered against the 6000 ceiling.
{"timestamp":"2026-08-28T23:20:00Z","error_message":"Anthropic Messages request carrying top_k fails with 400 and the customer-facing message hive-free is not available.","root_cause":"edge-api forwards four non-OpenAI fields verbatim from the Anthropic surface (top_k, thinking, cache_control, session_id) because the OpenAI chat-completions shape has no equivalent. hive-free load balances across providers with different tolerances, so the request succeeded or hard-failed depending on which pool member answered, and the refusal reached the caller as model unavailability rather than as a refused field","fix":"dispatchWithRetry in apps/edge-api/internal/inference/retry.go now classifies a 400 that names one of those four fields, strips that single field, and retries. Bounded to one strip per field and to attempts that have a successor; a field the caller actually sent is never stripped. Known edge filed as issue #1323","tags":["anthropic","edge-api","free-pool","sdk-conformance"]}{"timestamp":"2026-08-28T23:20:00Z","error_message":"GET /v1/models leaked an upstream provider name in the description of two aliases","root_cause":"model_aliases.summary is served verbatim as the description field, which makes it a customer-facing surface under the provider-blind rule, and 20260717_02_voice_groq_stt_tts.sql seeded the two voice aliases with the upstream named in prose","fix":"migration 20260828_01_voice_alias_summary_provider_blind.sql rewrites both summaries, scoped by alias_id and the exact old text so it cannot overwrite an operator retune. Caught by the committed assertion in packages/sdk-tests/js/tests/models/list-models.test.ts","tags":["provider-blind","catalog","migration","sdk-conformance"]}Test plan
docker compose --profile test build sdk-tests-pysucceeds with the newanthropicdependencydocker compose --profile test run --rm sdk-tests-py pytest -v-- 32 passed, 4 xfailed, no regressionsxfailmaps to a filed, linked issue (Anthropic /v1/models incompatible with real Anthropic SDK: wrong auth header accepted, wrong error envelope, OpenAI-shaped body #1259, Anthropic /v1/messages returns content:null instead of content:[] on empty completions, breaking real SDK/typed clients #1260, POST /v1/messages/count_tokens 401s a valid Hive API key: session-only auth, no API-key path #1261, Streaming content_block_start omits text field, crashing the real Anthropic SDK's stream accumulator on every text response #1274)