chore: migrate planning docs and specs to Obsidian vault - #315
Merged
Merged
Conversation
Moved .planning/, docs/, and website/sovereign/SPEC.md (299 files) to the Obsidian vault at hive/ for cross session reference. These were working docs (plans, specs, research, execution logs) that belong in the vault per the project's documentation convention, not duplicated in the repo.
|
Too many files changed for review. ( Bypass the limit by tagging |
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
Important Review skippedToo many files! This PR contains 303 files, which is 153 over the limit of 150. To get a review, narrow the scope: Upgrade to a paid plan to raise the limit. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (303)
You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
sakibsadmanshajib
added a commit
that referenced
this pull request
Aug 26, 2026
…eway (#1199) Closes #1192 (part one). Parts two and three below are decisions and findings, deliberately not implemented here. ## What this changes One file: `packages/openai-contract/generated/hive-openapi.yaml`, regenerated by its own generator. No hand edits. The generated spec had been produced from an older support matrix and never regenerated, so it annotated 22 live endpoints `planned_for_launch`. Since #1187 that file is served publicly and unauthenticated at `/api/openapi.yaml` for code generators to consume, which means every integrator who pointed tooling at it was told that `POST /v1/chat/completions`, the gateway's primary endpoint, is merely planned. The same applied to `/v1/embeddings`, `/v1/responses`, and all of `/v1/files/*`, `/v1/batches/*`, `/v1/audio/*`, `/v1/images/*` and `/v1/uploads/*`. Regenerated with `sh packages/openai-contract/scripts/generate-matrix.sh`, which runs `packages/openai-contract/scripts/sync_hive_contract.py` against the committed matrix and the pinned upstream document (`upstream/SPEC_VERSION` records the 2026-03-28 download, so the transform is deterministic and reproducible from the repo alone). The diff is 27 insertions and 23 deletions: the 22 `x-hive-status` flips, plus the `/chat/completions` `x-hive-notes` picking up the Phase 20 conditional tool-support text the matrix already carried. There is no formatting churn, which confirms the local PyYAML matches the version that produced the committed file. Running the generator a second time produces byte-identical output, so it is idempotent. ## Verification Re-running the exact comparison the console performs (`diffSpecAgainstMatrix` in `apps/web-console/lib/api-contract.ts`): | | before | after | |---|---|---| | total disagreements | 89 | 67 | | `status_mismatch` | 22 | **0** | | `missing_from_spec` | 67 | 67 | | `missing_from_matrix` | 0 | 0 | The 67 remaining are the benign ones the issue predicted and are unchanged: 51 are `out_of_scope` and dropped from the spec on purpose by the generator, and 16 are Hive-native endpoints that were never in the upstream OpenAI document (part two below). **The direction guard passes.** `tests/unit/console-docs-contract.test.ts` is green, all 12 tests, both with and without a populated env file. The spec operation count still matches the `x-hive-status` annotation count exactly (97 = 97), so the line scan that reads the spec did not stop matching after regeneration. **The served artifact was verified, not assumed.** `hive-web-console-prod:ci` was built from this branch and run, and `GET /api/openapi.yaml` returned HTTP 200 with `content-type: application/yaml`. The served body hashes to `a1aa0cda...`, byte-identical to the regenerated file in the tree. Parsing the served payload confirms `POST /v1/chat/completions`, `/v1/embeddings`, `/v1/responses`, `/v1/files`, `/v1/batches`, `/v1/audio/speech`, `/v1/images/generations` and `/v1/uploads` all now report `supported_now`. The production image was checked specifically because fixing only the dev image would have shipped a page that 500s in production while every local check stayed green. Container unit-test failures unrelated to this change: `tests/unit/ci-web-e2e-secret-free.test.ts` fails in-container by design (it reads a workflow file the Dockerfile never copies in). A further set of render tests fail in the container and drop from 15 files to 8 once an env file is supplied, so they are environment artifacts rather than code. Only three files in the app read the contract at all (`app/api/openapi.yaml/route.ts`, `app/console/docs/page.tsx`, `tests/unit/console-docs-contract.test.ts`) and none of them are in the failing set. ## Part two, a decision for the owner, not implemented here Sixteen endpoints classified `supported_now` in the matrix have no machine-readable schema anywhere: `/v1/rag/*` (6 operations), `/v1/agent/tasks*` (4), `/v1/artifacts*` (3), `/v1/featuregate`, and the Anthropic-compatible `/v1/messages` and `/v1/messages/count_tokens`. The generator is built purely from OpenAI's upstream document, so nothing Hive added itself can appear in its output by construction. A developer cannot generate a client for any of them. Three ways forward, in ascending cost: 1. **Leave it.** The spec stays a pure OpenAI-compatibility document and Hive's own surface stays undocumented in machine-readable form. Zero work, and the docs page keeps reporting the 16 as a visible disagreement rather than hiding them. 2. **Author them by hand and merge them in the generator.** This is cheaper than it looks, because the pattern already exists in this package and is currently orphaned: `packages/openai-contract/spec/paths/` already contains four hand-authored Hive OpenAPI documents (`spend-alerts.yaml`, `invoices.yaml`, `grants.yaml`, `budgets.yaml`) covering the `/api/v1/*` control-plane surface. Nothing reads them. `sync_hive_contract.py` consumes only `upstream/openapi.yaml` and the matrix, so those four files are authored, committed, reviewed, and consumed by nothing. Option 2 is roughly: write 16 operations in that same style, then teach the generator to merge `spec/paths/*.yaml` into its output. The ongoing cost is keeping hand-written schema in step with Go handlers by review discipline alone. 3. **Generate from the handlers.** Highest fidelity and lowest long-term rot, but the services route with plain `net/http.ServeMux` and carry no schema annotations, so this means adding an annotation layer or adopting a framework. Much the largest change. This is the difference between "an OpenAI-compatible gateway" and "a gateway with its own documented API", so it is a product call rather than an engineering one. While mapping this I also found that **`packages/openai-contract/overlays/hive-support-status.yaml` is a third orphaned artefact**. It is a 148-action OpenAPI Overlay document that sets `x-hive-status` per operation, it is not read by the generator (which takes status straight from the matrix), and it is itself stale in exactly the way the generated spec was: it still declares `/chat/completions` `planned_for_launch`. `.wolf/buglog.jsonl` already records it as "consumed by nothing" as of 2026-07-17. Worth deleting or wiring up, so nobody edits it believing it has an effect. ## Part three, the matrix is itself stale against the code Asked for because the console prints the matrix's own `generated` date. Two problems. **The printed date is wrong.** The matrix declares `generated: 2026-03-28`, but the file was last edited 2026-07-28 (#573), and before that 2026-07-22 (#416) and 2026-07-17 (#352). The field is hand-maintained and was not updated by those edits, so the console currently prints a date roughly four months earlier than the file's real content. **Five route families exist in the code and are absent from the matrix entirely**, four of which shipped after the stated generation date: | route family | service | shipped | PR | |---|---|---|---| | `/v1/agent/schedules`, `/v1/agent/schedules/{id}` | edge-api | 2026-08-23 | #1081 | | `/v1/audio/voices` | edge-api | 2026-08-24 | #1079 | | `/v1/tenants/switch` | control-plane | 2026-05-17 | #140 | | `/v1/admin/credit-grants`, `/v1/admin/credit-grants/{id}` | control-plane | 2026-05-08 | #136 | | `/v1/credit-grants/me` | control-plane | 2026-05-08 | #136 | Per the brief I have not fixed this here, because adding matrix rows changes runtime behaviour (see below), would need its own integration-test additions, and would change this PR's diff again. ### The part that needs a decision quickly The matrix is not documentation. `apps/edge-api/cmd/server/main.go:581` wraps the entire mux in `middleware.UnsupportedEndpointMiddleware(m)`, and `matrix.Lookup` (`apps/edge-api/internal/matrix/types.go:44`) returns `StatusUnknown` for any method and path it cannot match exactly or by template. The middleware answers `StatusUnknown` with a 404, `Unknown endpoint`. Replaying that exact lookup algorithm against the committed matrix, all six operations of the two newest edge-api route families resolve to `StatusUnknown`, and therefore to a 404: ``` GET /v1/audio/voices -> UNKNOWN -> 404 GET /v1/agent/schedules -> UNKNOWN -> 404 POST /v1/agent/schedules -> UNKNOWN -> 404 GET /v1/agent/schedules/{id} -> UNKNOWN -> 404 PUT /v1/agent/schedules/{id} -> UNKNOWN -> 404 DELETE /v1/agent/schedules/{id} -> UNKNOWN -> 404 (control: POST /v1/chat/completions -> supported_now) (control: POST /v1/agent/tasks -> supported_now) ``` This is the same failure recorded in `.wolf/buglog.jsonl` as `matrix-missing-proprietary-endpoints` on 2026-07-17, which was a demo blocker. The guard built in response, `apps/edge-api/internal/middleware/unsupported_integration_test.go`, drives the real middleware against the real committed matrix, but it enumerates only the Waves 2-4 routes. It covers neither `/v1/agent/schedules*` nor `/v1/audio/voices`, so the same class of regression landed again without turning anything red. I have verified this by reading the middleware and `Lookup` and by replaying the algorithm, not against a live stack, so please confirm against a running edge-api before acting. If it holds, scheduled agent tasks (#1081) and the Open WebUI voice roster (#1079) are both dead in any deployment, and the fix is matrix rows plus adding those paths to the integration test's table. ## Why this rotted, and the cheapest guard Nothing regenerates the spec in CI. The only reference to this package in `.github/workflows/ci.yml` is an unrelated lint script, so a matrix edit has never been forced to bring the generated spec with it. The repo already has the right pattern a few lines above, for a different artefact: ```yaml - name: Codegen drift — permissions.generated.ts run: | make gen-permissions git diff --exit-code apps/web-console/lib/control-plane/permissions.generated.ts ``` The same shape would have caught this on the day the matrix changed: ```yaml - name: Codegen drift — hive-openapi.yaml run: | sh packages/openai-contract/scripts/generate-matrix.sh git diff --exit-code packages/openai-contract/generated/hive-openapi.yaml ``` Not added here because `.github/workflows/ci.yml` was explicitly out of scope for this task and other lanes are live in it. Recommended as a small follow-up. ## Two smaller findings **The direction guard is now vacuously true, and its allowance is stale.** `console-docs-contract.test.ts` pins the direction of status mismatches rather than their count, which was the right call while a known-stale direction existed. After this PR there are zero status mismatches, so the loop body never executes. The guard is not dead (any *new* mismatch still enters the loop and fails, including the overstating direction it was written to catch), but the allowed direction it whitelists is exactly the bug this PR just fixed, so that specific regression could return silently. The test's comment also now describes a live defect that no longer exists. Per the brief I have not edited the test, since changing a guard in the same PR that changes what it guards deserves a separate decision. The one-line tightening, if wanted: ```ts expect(disagreements.filter((entry) => entry.kind === "status_mismatch")).toEqual([]); ``` That is strictly louder than what is there now, and because the generator stamps `x-hive-status` directly from the matrix, a mismatch is structurally impossible unless somebody edited the matrix without regenerating. In other words, that assertion is the drift guard, in a place CI already runs. **The generator writes a file the repo deleted.** `sync_hive_contract.py` still renders `docs/support-matrix.md`, but that file was removed from the repo by #315 when planning docs moved to the Obsidian vault, and it is not in `.gitignore`. Every run therefore drops a 21 KB untracked file into the tree, which a later `git add -A` would sweep back in and silently restore a deliberately deleted document. I removed it from my tree rather than committing it. Either drop `render_markdown` from the generator or ignore its output, but somebody should pick one. ## Buglog entry To be appended to `.wolf/buglog.jsonl` on `main` in a separate buglog-only PR after this merges, per `.claude/rules/openwolf.md`. ```json {"id":"openapi-spec-stale-understates-gateway","priority":"P1","date":"2026-08-25","error_message":"Publicly served /api/openapi.yaml annotated 22 live endpoints planned_for_launch, including POST /v1/chat/completions, telling every integrator the gateway's primary endpoint was not shipped","root_cause":"packages/openai-contract/generated/hive-openapi.yaml is generated from matrix/support-matrix.json by scripts/sync_hive_contract.py, but nothing in CI regenerates it or fails on drift. The matrix was edited three times (#352, #416, #573) without the generated spec being regenerated, so the spec kept the statuses of a much older matrix. Invisible until #1187 started serving the file publicly and rendering a spec-vs-matrix diff on /console/docs, which surfaced 89 disagreements.","fix":"Ran packages/openai-contract/scripts/generate-matrix.sh to regenerate the spec from the committed matrix and the pinned upstream document, no hand edits. Status mismatches went 22 to 0 and total disagreements 89 to 67 (the remaining 67 are out_of_scope drops and Hive-native endpoints absent from upstream, both by design). Verified the regenerated bytes reach the served artefact by building hive-web-console-prod:ci and hashing the GET /api/openapi.yaml response body. Follow-up recommended: a CI codegen-drift step mirroring the existing permissions.generated.ts one.","tags":["contract","openapi","codegen-drift","docs","ci-gap"]} ``` Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Merged
6 tasks done
sakibsadmanshajib
added a commit
that referenced
this pull request
Aug 26, 2026
… guard from the mux (#1203) ## Summary Live authenticated probe against hive-demo-cf confirmed the claim in the dispatch: `GET /v1/audio/voices` and every `/v1/agent/schedules` operation return 404 `{"code":"unknown_endpoint"}` in production, even with a real Supabase session, because they were never added to `packages/openai-contract/matrix/support-matrix.json`. `UnsupportedEndpointMiddleware` wraps the whole `/v1/` mux and answers `StatusUnknown` with 404 before the request ever reaches the feature gate or the handler, so both shipped features (#1079, #1081) have been dead on the live box since they merged. Reading the two handlers turned up two more unlisted operations in the same family while auditing: `GET /v1/agent/tasks/{task_id}/events` and `.../files`, real routes registered on the mux with no matrix entry at all. All eight are added with `supported_now`. ### How I verified (before writing any code) 1. Read `UnsupportedEndpointMiddleware` and `matrix.Lookup`: confirmed the mechanism (default case on `StatusUnknown` returns 404) and that auth (`authSelectorMiddleware`) wraps *outside* the middleware, so an unauthenticated probe cannot distinguish "unlisted" from "unauthorized" — which is exactly why the earlier unauthenticated probe was inconclusive. 2. Minted a real session via the admin one-time-token flow (magic-link mint, no password touched) against the box's self-hosted Supabase, from inside the `control-plane` container (it's the only container holding `SUPABASE_SERVICE_ROLE_KEY`), for `e2e-verified@scubed.com.bd`. 3. Sent raw authenticated HTTP requests to `edge-api:8080` from inside the docker network on the box. `GET /v1/models` (control) returned 200. `GET /v1/audio/voices` and `GET /v1/agent/schedules` both returned: ``` HTTP/1.1 404 Not Found {"error":{"message":"Unknown endpoint: GET /v1/audio/voices","type":"invalid_request_error","param":null,"code":"unknown_endpoint"}} ``` This is the literal `default:` branch body from `UnsupportedEndpointMiddleware`, i.e. `matrix.Lookup` returned `StatusUnknown`. Confirmed, not inferred. 4. Confirmed both endpoints are genuinely registered on the mux (`grep mux.Handle`), so this is purely a matrix data gap, not a routing gap. ## What changed **Data:** 8 new `supported_now` entries in `support-matrix.json`: `GET /v1/audio/voices`, the 5 `/v1/agent/schedules` operations, and `GET /v1/agent/tasks/{task_id}/events` / `.../files`. **Guard, two layers (this is a recurrence — buglog entry `matrix-missing-proprietary-endpoints`, 2026-07-17, was the same class and the guard built then only covers a hand-typed case list nobody extended for these two new families):** - Kept and extended `unsupported_integration_test.go`'s hand-list test with the 8 new cases. This is the only mechanism that can see a new suffix inside a handler's own internal path dispatch (`routeItem`/`routeTaskByID`-style switch) — a raw mux pattern alone cannot. - Added a mux-derived guard that needs no such list. `route_recorder.go` wraps the real `*http.ServeMux`, recording every pattern registered through it. `main()` now refuses to start (`log.Fatal`) if any `/v1/` pattern it actually registered has zero `support-matrix.json` coverage (`assertMatrixCoverage`), and `route_matrix_guard_test.go` exercises the identical check in CI by calling the real registration functions (`registerRAGRoutes`, `registerAgentTaskRoutes`, `registerAgentScheduleRoutes`, `registerInfraRoutes`, `registerMediaFileBatchRoutes`, the newly-extracted `registerAudioVoicesRoute`, and `artifacts.Handler.Register`) with lightweight fakes, mirroring the existing `gated_routes_test.go` pattern of building a real mux from real registration code. - Verified the new guard actually would have caught this bug: replayed it against the pre-fix matrix via `HIVE_MATRIX_PATH_FOR_TEST` and it fails with exactly `/v1/audio/voices, /v1/agent/schedules, /v1/agent/schedules/`. `registerInfraRoutes`, `registerMediaFileBatchRoutes`, the three `gated_routes.go` functions, and `artifacts.Handler.Register` now accept a small local interface (`httpMux` / `muxHandleFunc`) instead of the concrete `*http.ServeMux`, so the recorder can be passed to them with zero behavior change. Existing tests that build a plain `http.NewServeMux()` (`gated_routes_test.go`, `artifacts/handler_test.go`) compile unchanged — `*http.ServeMux` already satisfies both interfaces structurally. **Stated, known limit:** the mux-derived guard cannot see past a registered subtree prefix into a handler's own internal method/suffix switch — that's why the hand-list test stays rather than being deleted. Closing that fully would mean rewriting these proprietary handlers onto Go 1.22+ method+wildcard mux patterns (`mux.HandleFunc("GET /v1/agent/schedules/{id}", ...)`) instead of one prefix registration plus manual dispatch inside. Out of scope here; noted for anyone picking this up further. ## Not fixed here (noted per dispatch instructions) - Nothing in CI regenerates the spec/matrix from source, which is the root cause of the drift. The repo already has the right pattern for this in the `permissions.generated.ts` drift step; applying that pattern to the matrix is a separate change. - `packages/openai-contract/scripts/sync_hive_contract.py` still writes `docs/support-matrix.md`, deleted by #315 and not gitignored, so every run drops an untracked file that a later `git add -A` would silently restore. Also separate. ## Deploy status Deploys to the box are currently blocked by an unrelated failing migration on another branch. This fix will not reach production until that clears — merging this PR alone does not fix the live 404s. ## Test plan - [x] `go build ./apps/edge-api/...` - [x] `gofmt -l` clean on every changed/new file - [x] `go vet ./apps/edge-api/...` clean - [x] `go test ./apps/edge-api/... -count=1` — all packages green, including the extended hand-list test and the new guard test - [x] New guard test replayed against the pre-fix matrix via `HIVE_MATRIX_PATH_FOR_TEST` — fails on exactly the two originally-reported routes, confirming it would have caught this - [x] Live authenticated probe against hive-demo-cf (see Summary) — this is what confirmed the bug in the first place 🤖 Generated with [Claude Code](https://claude.com/claude-code) ## Buglog entry Per `.claude/rules/openwolf.md`, carried here for the follow-up buglog-only PR (not appended to `.wolf/buglog.jsonl` on this branch): ```json {"date":"2026-08-25","tags":["matrix","edge-api","routing","recurrence"],"error_message":"GET /v1/audio/voices and the whole /v1/agent/schedules family returned 404 unknown_endpoint on the live box despite being fully implemented and registered on the mux","root_cause":"UnsupportedEndpointMiddleware 404s any /v1/ path with no support-matrix.json entry, checked before auth/gate/handler; the two route families shipped (#1079, #1081) without matrix entries, and the existing regression guard (unsupported_integration_test.go, added for the same defect on 2026-07-17) only covered a hand-typed case list that nobody extended for these","fix":"Added the 8 missing matrix entries (including two more found while auditing: GET /v1/agent/tasks/{task_id}/events and .../files); added a mux-derived boot-time+CI guard (route_recorder.go, assertMatrixCoverage) that fails on any /v1/ pattern the mux actually registers with zero matrix coverage, so a new route can no longer ship unlisted without a human remembering to update a list"} ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added support for audio voice listing and agent task event, file, and schedule endpoints. * Added startup validation to detect API routes missing from the support matrix. * **Bug Fixes** * Improved route coverage checks to recognize exact paths, normalized paths, and route subtrees. * **Tests** * Added coverage checks for registered routes, including validation that unsupported routes are rejected. * Expanded regression coverage for agent and audio voice routes. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
.planning/(286 files) anddocs/(12 files), pluswebsite/sovereign/SPEC.md(1 file), for a total of 299 markdown files, out of the repository and into the Obsidian vault athive/(flat folder, one file per source doc) for cross session reference.type,date,source,tags) recording where it came from; original content is otherwise byte-for-byte unchanged below the frontmatter./replaced by-, lowercased, with the leading.planning/docsdot stripped. Example:.planning/phases/05-api-keys-hot-path-enforcement/05-01-PLAN.mdbecameplanning-phases-05-api-keys-hot-path-enforcement-05-01-plan.mdin the vault..planning/were intentionally left in the repository because they were never in scope for migration and have no vault copy:.planning/config.json(GSD tooling config) and.planning/staging/cloud-init.yml(a live OCI staging VM bootstrap script, not documentation). Four empty.gitkeepplaceholders in now-empty phase folders were removed along with their now-content-less directories.docs/is now fully removed from the repository (all 12 files were markdown and migrated).Why
These were working docs (plans, specs, research notes, execution logs, verification evidence) that belong in the Obsidian vault per the project's documentation convention rather than duplicated inside the repository.
Not in scope
This is a documentation cleanup only. It does not change any application behavior, code, or tests.
.wolf/, rootCLAUDE.md, rootREADME.md, and all other README files acrossapps/,deploy/,scripts/,services/,supabase/, andtools/were left untouched.Test plan
.planning/*.md+ 12docs/*.md+ 1website/sovereign/SPEC.md= 299 source files; 299 files written to the vault with zero collisions..planning/phases/, one underdocs/superpowers/specs/, one plan document) by diffing source content against vault content below the frontmatter; all matched exactly.git statusaftergit rmshowed no unexpected staged files, and that the two non-markdown files under.planning/were restored before committing.