feat: generate the console's API reference from the OpenAPI contract - #1187
Merged
Merged
Conversation
The console shell rendered a Documentation link pointing at hivegpt.io on every page, and there was no in-product API reference behind it. That is the first surface a developer opens when evaluating a gateway, and it pointed off-product. Adds /console/docs, generated at request time from the two artefacts the repo already produces: packages/openai-contract/generated/hive-openapi.yaml packages/openai-contract/matrix/support-matrix.json Nothing on the page is typed in by hand. It carries a quickstart (base URL, auth header, a curl and an OpenAI-SDK snippet naming a model alias read from the live catalog), then the full endpoint table grouped by support status and resource family. The endpoints that are not supported are listed too, with their counts, because docs that show only the happy path let somebody build against an endpoint that was never implemented. The support matrix's own generation date is printed, so staleness is visible rather than hidden. Where the spec and the matrix disagree, the page says so instead of quietly preferring one of them. Today that is 89 operations: 67 the matrix classifies that the spec never describes, and 22 the spec still annotates planned_for_launch that the matrix classifies supported_now. /api/openapi.yaml serves the raw spec unauthenticated so a developer can point their own tooling at it. It sits outside /console because middleware redirects unauthenticated requests there to sign-in, and a spec a code generator cannot fetch is not useful. Both web-console Dockerfiles now COPY packages/openai-contract/ into the image. Without it the route reads a file that is not there and answers 500 in the image only, which is the exact failure that made this issue non-trivial. The package is read from disk rather than vendored into apps/web-console, because a second copy of a generated artefact is the drift that generating it exists to prevent. Closes #1179
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
Warning Review limit reachedNext included review available in 16 minutes. View limit detailsLimit details: You’ve used the included review currently available. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Review configuration: ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (8)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Owner
Author
Visual proofCaptured against the image built from this branch's Dockerfile.web-console.prod (next build + next start, the same Dockerfile that builds console-hive.scubed.co), signed in against a live self-hosted Supabase and a live control-plane. 1: the shell Documentation link now reads /console/docs, and clicking it lands there. 2: the generated quickstart, naming the hive-auto alias read from the live catalog. 3: all 89 spec-versus-matrix disagreements, reported rather than hidden. 4: the unsupported and out-of-scope endpoints, listed with counts. Log: docs/proof/console-hosted-docs-2026-08-25/capture.log
|
… in the quickstart The capture log for the visual proof on PR #1187, committed here because lint:proof-tokens scans docs/proof/ and nothing else. The images themselves live on the visual-proof-assets release, which no branch deletion can reach. The quickstart now prefers one of Hive's own routing aliases when the catalog offers one. Taking the catalog's first entry meant an upstream model id could land in the snippet, which reads as a recommendation to bypass the aliases.
This was referenced Aug 25, 2026
sakibsadmanshajib
added a commit
that referenced
this pull request
Aug 26, 2026
…eway (#1199) Closes #1192 (part one). Parts two and three below are decisions and findings, deliberately not implemented here. ## What this changes One file: `packages/openai-contract/generated/hive-openapi.yaml`, regenerated by its own generator. No hand edits. The generated spec had been produced from an older support matrix and never regenerated, so it annotated 22 live endpoints `planned_for_launch`. Since #1187 that file is served publicly and unauthenticated at `/api/openapi.yaml` for code generators to consume, which means every integrator who pointed tooling at it was told that `POST /v1/chat/completions`, the gateway's primary endpoint, is merely planned. The same applied to `/v1/embeddings`, `/v1/responses`, and all of `/v1/files/*`, `/v1/batches/*`, `/v1/audio/*`, `/v1/images/*` and `/v1/uploads/*`. Regenerated with `sh packages/openai-contract/scripts/generate-matrix.sh`, which runs `packages/openai-contract/scripts/sync_hive_contract.py` against the committed matrix and the pinned upstream document (`upstream/SPEC_VERSION` records the 2026-03-28 download, so the transform is deterministic and reproducible from the repo alone). The diff is 27 insertions and 23 deletions: the 22 `x-hive-status` flips, plus the `/chat/completions` `x-hive-notes` picking up the Phase 20 conditional tool-support text the matrix already carried. There is no formatting churn, which confirms the local PyYAML matches the version that produced the committed file. Running the generator a second time produces byte-identical output, so it is idempotent. ## Verification Re-running the exact comparison the console performs (`diffSpecAgainstMatrix` in `apps/web-console/lib/api-contract.ts`): | | before | after | |---|---|---| | total disagreements | 89 | 67 | | `status_mismatch` | 22 | **0** | | `missing_from_spec` | 67 | 67 | | `missing_from_matrix` | 0 | 0 | The 67 remaining are the benign ones the issue predicted and are unchanged: 51 are `out_of_scope` and dropped from the spec on purpose by the generator, and 16 are Hive-native endpoints that were never in the upstream OpenAI document (part two below). **The direction guard passes.** `tests/unit/console-docs-contract.test.ts` is green, all 12 tests, both with and without a populated env file. The spec operation count still matches the `x-hive-status` annotation count exactly (97 = 97), so the line scan that reads the spec did not stop matching after regeneration. **The served artifact was verified, not assumed.** `hive-web-console-prod:ci` was built from this branch and run, and `GET /api/openapi.yaml` returned HTTP 200 with `content-type: application/yaml`. The served body hashes to `a1aa0cda...`, byte-identical to the regenerated file in the tree. Parsing the served payload confirms `POST /v1/chat/completions`, `/v1/embeddings`, `/v1/responses`, `/v1/files`, `/v1/batches`, `/v1/audio/speech`, `/v1/images/generations` and `/v1/uploads` all now report `supported_now`. The production image was checked specifically because fixing only the dev image would have shipped a page that 500s in production while every local check stayed green. Container unit-test failures unrelated to this change: `tests/unit/ci-web-e2e-secret-free.test.ts` fails in-container by design (it reads a workflow file the Dockerfile never copies in). A further set of render tests fail in the container and drop from 15 files to 8 once an env file is supplied, so they are environment artifacts rather than code. Only three files in the app read the contract at all (`app/api/openapi.yaml/route.ts`, `app/console/docs/page.tsx`, `tests/unit/console-docs-contract.test.ts`) and none of them are in the failing set. ## Part two, a decision for the owner, not implemented here Sixteen endpoints classified `supported_now` in the matrix have no machine-readable schema anywhere: `/v1/rag/*` (6 operations), `/v1/agent/tasks*` (4), `/v1/artifacts*` (3), `/v1/featuregate`, and the Anthropic-compatible `/v1/messages` and `/v1/messages/count_tokens`. The generator is built purely from OpenAI's upstream document, so nothing Hive added itself can appear in its output by construction. A developer cannot generate a client for any of them. Three ways forward, in ascending cost: 1. **Leave it.** The spec stays a pure OpenAI-compatibility document and Hive's own surface stays undocumented in machine-readable form. Zero work, and the docs page keeps reporting the 16 as a visible disagreement rather than hiding them. 2. **Author them by hand and merge them in the generator.** This is cheaper than it looks, because the pattern already exists in this package and is currently orphaned: `packages/openai-contract/spec/paths/` already contains four hand-authored Hive OpenAPI documents (`spend-alerts.yaml`, `invoices.yaml`, `grants.yaml`, `budgets.yaml`) covering the `/api/v1/*` control-plane surface. Nothing reads them. `sync_hive_contract.py` consumes only `upstream/openapi.yaml` and the matrix, so those four files are authored, committed, reviewed, and consumed by nothing. Option 2 is roughly: write 16 operations in that same style, then teach the generator to merge `spec/paths/*.yaml` into its output. The ongoing cost is keeping hand-written schema in step with Go handlers by review discipline alone. 3. **Generate from the handlers.** Highest fidelity and lowest long-term rot, but the services route with plain `net/http.ServeMux` and carry no schema annotations, so this means adding an annotation layer or adopting a framework. Much the largest change. This is the difference between "an OpenAI-compatible gateway" and "a gateway with its own documented API", so it is a product call rather than an engineering one. While mapping this I also found that **`packages/openai-contract/overlays/hive-support-status.yaml` is a third orphaned artefact**. It is a 148-action OpenAPI Overlay document that sets `x-hive-status` per operation, it is not read by the generator (which takes status straight from the matrix), and it is itself stale in exactly the way the generated spec was: it still declares `/chat/completions` `planned_for_launch`. `.wolf/buglog.jsonl` already records it as "consumed by nothing" as of 2026-07-17. Worth deleting or wiring up, so nobody edits it believing it has an effect. ## Part three, the matrix is itself stale against the code Asked for because the console prints the matrix's own `generated` date. Two problems. **The printed date is wrong.** The matrix declares `generated: 2026-03-28`, but the file was last edited 2026-07-28 (#573), and before that 2026-07-22 (#416) and 2026-07-17 (#352). The field is hand-maintained and was not updated by those edits, so the console currently prints a date roughly four months earlier than the file's real content. **Five route families exist in the code and are absent from the matrix entirely**, four of which shipped after the stated generation date: | route family | service | shipped | PR | |---|---|---|---| | `/v1/agent/schedules`, `/v1/agent/schedules/{id}` | edge-api | 2026-08-23 | #1081 | | `/v1/audio/voices` | edge-api | 2026-08-24 | #1079 | | `/v1/tenants/switch` | control-plane | 2026-05-17 | #140 | | `/v1/admin/credit-grants`, `/v1/admin/credit-grants/{id}` | control-plane | 2026-05-08 | #136 | | `/v1/credit-grants/me` | control-plane | 2026-05-08 | #136 | Per the brief I have not fixed this here, because adding matrix rows changes runtime behaviour (see below), would need its own integration-test additions, and would change this PR's diff again. ### The part that needs a decision quickly The matrix is not documentation. `apps/edge-api/cmd/server/main.go:581` wraps the entire mux in `middleware.UnsupportedEndpointMiddleware(m)`, and `matrix.Lookup` (`apps/edge-api/internal/matrix/types.go:44`) returns `StatusUnknown` for any method and path it cannot match exactly or by template. The middleware answers `StatusUnknown` with a 404, `Unknown endpoint`. Replaying that exact lookup algorithm against the committed matrix, all six operations of the two newest edge-api route families resolve to `StatusUnknown`, and therefore to a 404: ``` GET /v1/audio/voices -> UNKNOWN -> 404 GET /v1/agent/schedules -> UNKNOWN -> 404 POST /v1/agent/schedules -> UNKNOWN -> 404 GET /v1/agent/schedules/{id} -> UNKNOWN -> 404 PUT /v1/agent/schedules/{id} -> UNKNOWN -> 404 DELETE /v1/agent/schedules/{id} -> UNKNOWN -> 404 (control: POST /v1/chat/completions -> supported_now) (control: POST /v1/agent/tasks -> supported_now) ``` This is the same failure recorded in `.wolf/buglog.jsonl` as `matrix-missing-proprietary-endpoints` on 2026-07-17, which was a demo blocker. The guard built in response, `apps/edge-api/internal/middleware/unsupported_integration_test.go`, drives the real middleware against the real committed matrix, but it enumerates only the Waves 2-4 routes. It covers neither `/v1/agent/schedules*` nor `/v1/audio/voices`, so the same class of regression landed again without turning anything red. I have verified this by reading the middleware and `Lookup` and by replaying the algorithm, not against a live stack, so please confirm against a running edge-api before acting. If it holds, scheduled agent tasks (#1081) and the Open WebUI voice roster (#1079) are both dead in any deployment, and the fix is matrix rows plus adding those paths to the integration test's table. ## Why this rotted, and the cheapest guard Nothing regenerates the spec in CI. The only reference to this package in `.github/workflows/ci.yml` is an unrelated lint script, so a matrix edit has never been forced to bring the generated spec with it. The repo already has the right pattern a few lines above, for a different artefact: ```yaml - name: Codegen drift — permissions.generated.ts run: | make gen-permissions git diff --exit-code apps/web-console/lib/control-plane/permissions.generated.ts ``` The same shape would have caught this on the day the matrix changed: ```yaml - name: Codegen drift — hive-openapi.yaml run: | sh packages/openai-contract/scripts/generate-matrix.sh git diff --exit-code packages/openai-contract/generated/hive-openapi.yaml ``` Not added here because `.github/workflows/ci.yml` was explicitly out of scope for this task and other lanes are live in it. Recommended as a small follow-up. ## Two smaller findings **The direction guard is now vacuously true, and its allowance is stale.** `console-docs-contract.test.ts` pins the direction of status mismatches rather than their count, which was the right call while a known-stale direction existed. After this PR there are zero status mismatches, so the loop body never executes. The guard is not dead (any *new* mismatch still enters the loop and fails, including the overstating direction it was written to catch), but the allowed direction it whitelists is exactly the bug this PR just fixed, so that specific regression could return silently. The test's comment also now describes a live defect that no longer exists. Per the brief I have not edited the test, since changing a guard in the same PR that changes what it guards deserves a separate decision. The one-line tightening, if wanted: ```ts expect(disagreements.filter((entry) => entry.kind === "status_mismatch")).toEqual([]); ``` That is strictly louder than what is there now, and because the generator stamps `x-hive-status` directly from the matrix, a mismatch is structurally impossible unless somebody edited the matrix without regenerating. In other words, that assertion is the drift guard, in a place CI already runs. **The generator writes a file the repo deleted.** `sync_hive_contract.py` still renders `docs/support-matrix.md`, but that file was removed from the repo by #315 when planning docs moved to the Obsidian vault, and it is not in `.gitignore`. Every run therefore drops a 21 KB untracked file into the tree, which a later `git add -A` would sweep back in and silently restore a deliberately deleted document. I removed it from my tree rather than committing it. Either drop `render_markdown` from the generator or ignore its output, but somebody should pick one. ## Buglog entry To be appended to `.wolf/buglog.jsonl` on `main` in a separate buglog-only PR after this merges, per `.claude/rules/openwolf.md`. ```json {"id":"openapi-spec-stale-understates-gateway","priority":"P1","date":"2026-08-25","error_message":"Publicly served /api/openapi.yaml annotated 22 live endpoints planned_for_launch, including POST /v1/chat/completions, telling every integrator the gateway's primary endpoint was not shipped","root_cause":"packages/openai-contract/generated/hive-openapi.yaml is generated from matrix/support-matrix.json by scripts/sync_hive_contract.py, but nothing in CI regenerates it or fails on drift. The matrix was edited three times (#352, #416, #573) without the generated spec being regenerated, so the spec kept the statuses of a much older matrix. Invisible until #1187 started serving the file publicly and rendering a spec-vs-matrix diff on /console/docs, which surfaced 89 disagreements.","fix":"Ran packages/openai-contract/scripts/generate-matrix.sh to regenerate the spec from the committed matrix and the pinned upstream document, no hand edits. Status mismatches went 22 to 0 and total disagreements 89 to 67 (the remaining 67 are out_of_scope drops and Hive-native endpoints absent from upstream, both by design). Verified the regenerated bytes reach the served artefact by building hive-web-console-prod:ci and hashing the GET /api/openapi.yaml response body. Follow-up recommended: a CI codegen-drift step mirroring the existing permissions.generated.ts one.","tags":["contract","openapi","codegen-drift","docs","ci-gap"]} ``` Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1179.
What this changes
The console shell rendered a Documentation link pointing at
https://hivegpt.ioon every console page, and there was no in-product API reference behind it. Hosted docs is row 7 of the console similarity scorecard, verdict MISSING, and one of the two surfaces a developer opens first when judging a gateway.This adds
/console/docs, generated at request time from the two artefacts the repo already produces, and repoints both in-app documentation links at it.apps/web-console/lib/api-contract.tspackages/openai-contract/, parses the support matrix, extracts the spec's operations and theirx-hive-status, diffs the two, groups the result by status and resource family.apps/web-console/app/console/docs/page.tsxapps/web-console/app/api/openapi.yaml/route.tsapps/web-console/components/app-shell/console-shell.tsx/console/docs. This is the single highest-visibility line in the PR.apps/web-console/app/console/page.tsxhivegpt.iolink, on the overview page's "Need the API reference?" footer, repointed the same way.deploy/docker/Dockerfile.web-console,.prodCOPY packages/openai-contract/so the route can read it inside the image.apps/web-console/tests/unit/console-docs-contract.test.tsNothing on the page is typed in by hand. Every endpoint, status, count and date comes from the spec or the matrix.
Honesty properties
generateddate is printed next to its version, with a line saying plainly that if the date is old, the classifications are that old too.getCatalogModels()), falling back tohive-default, the alias seeded insupabase/migrations/, if the catalog cannot be read. No invented model name.Spec versus matrix: what the diff actually found
97 operations in the generated spec, 164 in the support matrix, 89 disagreements, all rendered on the page:
out_of_scope, whichscripts/sync_hive_contract.pydrops from the generated spec deliberately. The other 16 aresupported_nowHive-native endpoints the spec has never described:/v1/rag/*(6),/v1/agent/tasks*(4),/v1/artifacts*(3),/v1/featuregate, and the Anthropic-compatible/v1/messagesand/v1/messages/count_tokens. The generated spec is built from OpenAI's upstream document, so nothing Hive added itself is in it. A developer generating a client from the YAML gets no schema at all for 16 live endpoints.planned_for_launch, the matrix classifies itsupported_now. IncludingPOST /v1/chat/completions.The finding worth carrying out of this PR: the stale artefact is the spec, not the matrix. The matrix says
generated: 2026-03-28and is the more current of the two;generated/hive-openapi.yamlwas produced from an older matrix and still tells any tool that reads it that chat completions, embeddings, files, batches, audio, images, responses and uploads are merely planned. Regenerating it is apackages/openai-contract/change and out of scope here, so it is reported rather than papered over. Worth its own issue.The Docker blocker, and proof the fix works
A route importing from
packages/compiles locally and then breaks the image, because neither web-console Dockerfile copied that tree. Both now do.Dockerfile.web-console.prodis included deliberately: it is what actually buildsconsole-hive.scubed.co, so fixing only the dev image would have shipped a docs page that 500s in production while looking green everywhere else.The package is read from disk rather than vendored into
apps/web-console/, since a second copy of a generated artefact reintroduces exactly the drift that generating from a spec avoids.Verified in the image, with
--build, which is the point:docker compose run --rm --build web-console npm run build— exit 0./console/docsand/api/openapi.yamlboth appear in the route manifest, so the route resolves and typechecks inside the image.docker compose run --rm --build web-console npm run test:unit— 691 passed, 12 of them new. The new tests callloadSupportMatrix()andloadSpecOperations()against the real files, so they would ENOENT inside the image without the COPY.tests/unit/ci-web-e2e-secret-free.test.ts, which reads.github/workflows/ci.ymlthat the Dockerfile never copies in. Known, pre-existing, container-only, passes in CI.What the tests guard
The three ways a generated docs page goes quietly wrong:
x-hive-status:annotation count, which is an independent count of the same set, so that goes red instead of rendering an empty page.API_BASE_URL's host is asserted againstdeploy/cloudflare/tunnel-ingress.json, the file that calls itself the single source of truth for which hive hostnames are public, and its path against the spec's ownservers[0].url. Both have moved before.href="https://hivegpt.io"and to link/console/docs.Plus a shape test per disagreement kind, an unknown-status test proving an unrecognised classification gets its own section rather than vanishing from the counts, and a parser test proving a malformed matrix throws rather than rendering an empty table.
The real-contract test pins the direction of the status mismatches rather than their count, so regenerating the spec makes it pass rather than cry wolf, but a mismatch in a new direction (say, the spec claiming supported for something the matrix calls unsupported) fails loudly.
Buglog entry
{"id":"console-docs-hivegpt-dead-link","date":"2026-08-25","title":"Console Documentation link pointed off-product at hivegpt.io on every page, and no in-product API reference existed","error_message":"Shell header link href=\"https://hivegpt.io\" rendered on every /console/* page; second copy on the overview page footer","root_cause":"No hosted docs surface was ever built. The repo generates an OpenAPI contract and a 164-endpoint support matrix under packages/openai-contract/, but deploy/docker/Dockerfile.web-console and .prod copy only apps/web-console/, deploy/, .env.example and supabase/migrations/, so any console route importing from packages/ compiles locally and then breaks the Docker build. That blocker is why the cheap fix was never taken.","fix":"Added /console/docs generated from the spec and matrix at request time, an unauthenticated /api/openapi.yaml route serving the raw spec, COPY packages/openai-contract/ in both web-console Dockerfiles, and repointed both hivegpt.io links. 12 unit tests guard the extraction count, the base URL against tunnel-ingress.json, and the absence of the off-product link.","tags":["web-console","docs","openapi","docker","dead-link","issue-1179"]}Visual proof
Posted as a comment on this PR, images on the permanent
visual-proof-assetsrelease. Captured against the image built from this branch'sDockerfile.web-console.prod(next build+next start, the same Dockerfile that buildsconsole-hive.scubed.co), running against a live self-hosted Supabase and a live control-plane, signed in throughtests/e2e/support/live-auth.mjs. No password was set or rotated.Recorded in
docs/proof/console-hosted-docs-2026-08-25/capture.log:shell header Documentation link href = /console/docs, and clicking it lands on/console/docs.endpoint rows rendered = 164, all four status sections present.GET /api/openapi.yaml (no session) -> 200 application/yaml; charset=utf-8,spec bytes served = 1165460. Served from the prod image, which is the COPY working end to end./console,/console/catalog,/console/docsall load clean with no page errors.The signed-in address is covered with Playwright's screenshot mask, which paints over the region without touching the DOM, so it reaches neither the pixels nor the log.
npm run lint:proof-tokenspasses over the committed log.