Repository navigation
feat: AI-first surface — llm() factory, PgVectorStore, MCP-driven authz - #32
Merged
Merged
Conversation
`llm()` already returns an `action` (packages/ai/src/llm.ts) and the ruling is recorded in root CLAUDE.md, docs/idea/09-ai-first.md and packages/ai/CLAUDE.md. What was missing was axiom 3: nothing failed when the rule was broken. - PRIMITIVE_KINDS is now the single source `PrimitiveKind` derives from, so the runtime list and the type cannot drift; registrar.test.ts pins it at eight, so inventing a ninth kind is a failing test rather than a silent compile. - llm.test.ts pins `.client()` and `.as`, the two projections the ruling names but nothing exercised, and asserts the returned value declares itself as one of core's canonical eight rather than a bare 'action' literal. Both negative-proved: a ninth kind reddens registrar.test.ts; dropping `.client` from the action facade reddens the llm projection pin. Co-Authored-By: Claude <noreply@anthropic.com>
PgVectorStore is the only production vector path, and every test it had asserted the statement TEXT a method compiles to against a recording client. That cannot catch a statement Postgres rejects, nor a filter that compiles cleanly and excludes nothing — which is exactly how metadata shipped bound `::jsonb`, JSON-encoded twice, making every `metadata ->>` filter match nothing while reading back correctly. - pg-vector.live.test.ts runs the whole chain: ddl() -> a live server -> upsert -> cosine, FTS and the RRF fusion in SQL -> decoded hit. Tenant isolation, the policy allow-list on BOTH halves of the fusion, the (tenant, id) key and scoped delete are asserted against real rows. - The plan is pinned too: the distance ordering sits in a subquery because that is the only shape hnsw answers, and the `1 - distance desc` alternative is a sequential scan. Only a planner can say which shipped. - It REFUSES to skip when TEST_DATABASE_URL names a Postgres without the extension. A suite that stands down reports green for the one store that runs in front of real traffic — so CI's service container is now pgvector/pgvector:pg17. Negative-proved: reverting `::text::jsonb`, dropping the scope from the lexical CTE, dropping the tenant from deleteSql, and inverting the distance ordering each turn the matching case red. Co-Authored-By: Claude <noreply@anthropic.com>
The two references were already shipped — stream() reads SSE through readSse/MessageStream, and RemoteEmbedder moved to remote-embedder.ts. The real gaps were in what the provider sends and how it reads a reply. - new models.ts: the catalogue plus the reasoning controls each model accepts. `output_config.effort` and adaptive thinking arrived with 4.6, so one body for the whole catalogue was a guaranteed 400 on claude-haiku-4-5 — a blessed model that could not be called at all. A control the model lacks is now refused locally with a real code rather than silently dropped, and the opus-only cap on disabling thinking no longer refuses a legal sonnet request. - generate() above STREAM_ONLY_MAX_TOKENS (16k) runs the streaming transport and assembles the result. The ceiling is the transport's and llm() has no streaming path, so refusing would make a legal maxTokens undeclarable. - a refusal is a 200 with no answer in it: X_LLM_REFUSED at the llm() seam before the schema runs, carrying stop_details.category from both transports. Truncation that fails its schema is X_LLM_TRUNCATED — the ceiling does not move, so the repair turn cannot succeed. Neither is cached. +11 tests, all four fixes negative-proved. verify 17/17. Co-Authored-By: Claude <noreply@anthropic.com>
`x verify` gated on the drop from a committed baseline and on every prompt having an eval, but two doors left it green over nothing: - `ULTIMATE_EVAL_RECORD` in the environment made every eval write the numbers it had just measured and pass — a gate run inheriting the flag reported green and rewrote the committed baselines on its way through. The step now refuses with `X_EVAL_RECORDING` *before* the suite runs, since a finding after the fact does not put the baselines back. Proved live: red step, baseline byte identical. - A `defineEval` proves a prompt is named, never that anything measured it. An eval whose baseline was never recorded — one no test asserts, one whose `baseline:` is a cwd-relative string — satisfied `X_EVAL_MISSING` while gating on nothing. The step now reads each declared baseline through `@ultimat3/ai`'s own reader: absent is `X_EVAL_BASELINE_MISSING`, unreadable is `X_EVAL_BASELINE_INVALID`. New code `X_EVAL_RECORDING` owned by `@ultimat3/ai`, registered and documented. verify 17/17 · reference-app ratchet unchanged at 10/17, eval green bun test 3970/26 — +6 = exactly the new tests, failure set unchanged Co-Authored-By: Claude <noreply@anthropic.com>
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 44 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yml Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (9)
📝 WalkthroughWalkthroughThe PR adds model-specific AI controls, refusal and truncation errors, refusal-aware caching, streaming metadata, eval baseline safeguards, live pgvector tests, and an executable eight-primitive registry contract. ChangesAI runtime behavior
Verification and runtime contracts
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant LLM
participant Gateway
participant AnthropicProvider
participant MessageStream
LLM->>Gateway: request generation
Gateway->>AnthropicProvider: generate request
AnthropicProvider->>MessageStream: stream large request
MessageStream-->>AnthropicProvider: result with stop details
AnthropicProvider-->>Gateway: GenerateResult
Gateway-->>LLM: refusal error or cached result
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 9
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/ci.yml:
- Line 91: Pin both pgvector service image references in
.github/workflows/ci.yml at lines 91-91 and 160-160 to
pgvector/pgvector@sha256:7ae6051efd0e60444282c27c7e141af07f322ce033300e727a49c3dd11075e38,
replacing the mutable pg17 tags at both sites.
In `@packages/ai/src/errors.ts`:
- Line 147: Update LlmRefusedError at packages/ai/src/errors.ts:147-147 and
LlmTruncatedError at packages/ai/src/errors.ts:164-164 so each fix value is an
executable, supported command or pasteable call expression rather than prose;
preserve the stable X_* code and cause fields on every throw while providing
exact runnable remediation for rephrasing/model selection and max-token
configuration.
In `@packages/ai/src/models.ts`:
- Around line 1-12: Shorten the file headers in packages/ai/src/models.ts lines
1-12 and packages/ai/src/provider.ts lines 1-6 to no more than four lines each.
Each header must state the module’s responsibility and why the module exists,
while retaining the essential context from the existing comments.
- Around line 124-132: Update the effort handling around the rules.effort branch
so output_config is added only when effort is explicitly defined; do not default
an undefined effort to 'high'. Preserve 'high' only in the local
disabled-thinking validation path, and retain the existing AiRequestInvalidError
behavior for unsupported explicitly provided effort.
In `@packages/ai/src/pg-vector.live.test.ts`:
- Around line 1-10: Reduce the file-level header above the live tests to one
through four lines describing that the file validates PgVectorStore against a
real pgvector/Postgres server. Remove the detailed implementation rationale and
regression history from the header; retain only the module’s single
responsibility.
- Line 13: Remove the direct `@ultimat3/db` import from the live test and route
createPostgresClient, PostgresClient, raw, and sql through an approved
lower-tier AI boundary. Update the live-client setup in the test to use that
boundary while preserving its existing behavior and types.
- Around line 69-76: Update the pgvector setup catch block around the extension
creation in the test setup to throw a registered AI-specific UltimateError
instead of silently returning. Add the appropriate error subclass, pass the
caught database exception through sourceError, include the failure message via
cause, and provide an executable fix while preserving the existing
unavailable-extension handling.
In `@packages/cli/src/app-evals.test.ts`:
- Around line 200-207: Update the whileRecording helper to capture the existing
Bun.env[RECORD_ENV] value before setting recording mode, then restore that exact
value in finally; only delete the environment entry when it was previously
unset.
In `@wiki/Error-Codes.md`:
- Line 296: The X_AI_REQUEST_INVALID guidance currently directs callers to edit
prompt content instead of the llm() request. Update the error-code entry to
instruct removing unsupported effort or thinking controls from the llm()
declaration/request, or selecting a compatible model; keep model-specific
controls defined through MODELS and clarify that unsupported controls are
rejected locally.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 309cba80-c182-46fb-b383-d56c8006f0f6
📒 Files selected for processing (30)
.github/workflows/ci.ymlCLAUDE.mddocs/architecture/14-testing-internals.mdexamples/dummy/CLAUDE.mdframework.manifest.jsonpackages/ai/CLAUDE.mdpackages/ai/README.mdpackages/ai/src/errors.tspackages/ai/src/gateway.test.tspackages/ai/src/gateway.tspackages/ai/src/index.tspackages/ai/src/llm.test.tspackages/ai/src/llm.tspackages/ai/src/models.tspackages/ai/src/pg-vector.live.test.tspackages/ai/src/prompt.tspackages/ai/src/provider.test.tspackages/ai/src/provider.tspackages/ai/src/wire.tspackages/cli/CLAUDE.mdpackages/cli/src/app-evals.test.tspackages/cli/src/app-evals.tspackages/cli/src/verify-tests.tspackages/core/CLAUDE.mdpackages/core/src/index.tspackages/core/src/registrar.test.tspackages/core/src/registrar.tswiki/CLI-Reference.mdwiki/Error-Codes.mdwiki/Testing.md
- reasoningBody omits output_config when no effort was requested, matching the invariant its own docstring states — a default sent as a request is indistinguishable on the wire from a declaration that asked for it - X_LLM_REFUSED / X_LLM_TRUNCATED fixes are now pasteable: a real blessed model id computed at the throw site, and a concrete maxTokens number - fix lines and the wiki row name definePrompt / the llm() declaration rather than "the prompt", which reads as the template text - file headers in models.ts, provider.ts and pg-vector.live.test.ts cut to the 1-4 line convention; the detail moves onto the declarations it covers - the live pgvector setup keeps the caught error as `cause` — "not available" and "permission denied" want different fixes - whileRecording restores ULTIMATE_EVAL_RECORD instead of deleting it Co-Authored-By: Claude <noreply@anthropic.com>
Contributor
|
This PR looks ready — CI is green, there's an approval, and the risk is routine (minor semver impact, experienced contributor). However, there's an active CodeRabbit review in progress. Once that completes, this PR can be merged. 🤖 Posted by developerz.ai — the maintainer agent, not a human. |
Contributor
|
🎉 Released: v0.2.0 🤖 Posted by developerz.ai — the maintainer agent, not a human. |
3 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
llm()action factory,PgVectorStore(live-proved against real pgvector), Anthropic provider per-model requests, and theevalgate step (session commits already on branch).examples/dummy/apps/web/app/posts/mcp-drive.contract.test.tsproves authz parity: an agent callingcreatePost/createCommentover the wire MCP protocol (mcp.server.handle()) gets the identicalX_FORBIDDENdenial a direct call gets — same entry point as the UI (packages/mcp/src/exposed.ts→action.run).examples/dummy/apps/web/app/posts/prompts/summarize.eval.test.tsgreen (2/2).Test plan
bun run verify— 17/17 framework gate greencd examples/dummy && bun test apps/web/app/posts/mcp-drive.contract.test.ts apps/web/app/posts/prompts/summarize.eval.test.ts— 6/6 passscripts/reference-app-gate.ts'sEXPECTED_RED, not this task's scope)🤖 Generated with Claude Code
Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.Summary by CodeRabbit
New Features
Bug Fixes
Documentation