Repository navigation
feat: provider visibility for devs + block reasons - #3335
Merged
Merged
Conversation
Provider-key and custom-model list endpoints now scope to all active org memberships instead of owner/admin only, so project-scoped developers can browse which providers and custom models are available. Tokens stay masked and every mutation remains owner/admin-gated; the custom-models page disables its management controls for developers and shows a read-only badge. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The compliance predicates in @llmgateway/models are now backed by reason-returning variants (getProviderComplianceFailures, getProviderRefPolicyListFailures, getAttestationComplianceFailures) so callers can see which requirement a provider fails. The Provider Impact card lists those reasons in a tooltip on every blocked chip (catalogue and custom providers alike) and shows a banner when an allowed-providers list is active, since that blocks every other provider regardless of certifications. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
Contributor
WalkthroughCustom-model and provider-key reads now allow active organization members. Custom-model mutations remain restricted to enterprise owners and admins. Compliance APIs return structured failure reasons, which the UI displays for blocked providers. Provider selectors show policy status and filtering controls. ChangesCompliance and organization access
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant Member
participant ComplianceClient
participant ProviderCompliance
participant ProviderSelector
Member->>ComplianceClient: open compliance impact view
ComplianceClient->>ProviderCompliance: evaluate catalogue and custom providers
ProviderCompliance-->>ComplianceClient: return failure reasons
ComplianceClient->>ProviderSelector: provide policy status and notes
ProviderSelector-->>Member: show shields, tooltips, and compatible filter
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
## Summary Stacked on #3335 (base branch is `developer-models-compliance-visibility`; GitHub will retarget to `main` when that PR merges — only the last two commits are new here). Completes the remaining asks from the enterprise customer report that #3335 addressed: **1. "How can I tell which providers meet my compliance requirements when adding an allowed provider?"** The provider dropdowns on the compliance page are now policy-aware: - A **green shield** marks providers that meet every active certification, data-policy, and headquarters requirement; a **red shield** marks providers that don't, with the exact failing requirements listed under the name (e.g. "May log prompts · Headquartered in China, which is not an allowed country"). - A **"Only providers that meet policy requirements"** toggle at the top of the dropdown hides incompatible providers — the customer's "show only compatible providers" request. - The indicators deliberately evaluate the requirements only and **ignore the allowed/blocked lists themselves**, so an active allow list (the customer's exact situation: only their custom provider allow-listed) no longer makes every candidate look blocked while choosing what to add. - The org's custom providers are evaluated against their self-attested posture; ones without an attestation show "No compliance attestation on file". - Selected chips that fail the policy get a small warning shield. **2. "What do the different colors represent?"** The unexplained dot in the picker was the provider's *brand color* and carried no compliance meaning. In the compliance context it is replaced by the meaningful shield indicators with an in-dropdown legend, and the docs now spell out what every indicator/color means (impact chips, shields, brand dots elsewhere). **3. Developer-role discoverability (follow-up to #3335's read access)** #3335 made the custom-models catalog readable for developers, but their sidebar had no path to it. Developers now get an **Organization → Custom Models** entry pointing at the read-only catalog, and the docs recommend this as the way for developers to see available providers/models without extra permissions. Also fixes the Provider Impact counter to include custom providers ("0 of 45 providers meet this policy… Requests will be blocked" previously showed even when the org's allow-listed custom provider was fully compliant; it now reads "0 of 45 catalogue providers meet this policy. 1 of 2 custom providers comply." and the empty state says only compliant custom providers can serve requests). The `MultiProviderSelector` changes are backward-compatible: options without `meetsPolicy` render exactly as before (brand-color dot, no toggle/legend), so the API-keys and IAM usages are unaffected. ## Testing - `pnpm test:unit` — 208 files, 3457 tests pass. - `pnpm build` — all 17 tasks pass. - `pnpm format` clean. - Verified end-to-end against a seeded enterprise org reproducing the customer scenario (policy with no-training/no-logging + FR/GB/US countries + only a custom provider allow-listed): recorded a demo video showing the owner flow (blocked reasons, picker shields, compatible-only filter, allow-listing a compliant provider) and the developer flow (new sidebar entry → read-only catalog). https://claude.ai/code/session_01Vd3toRu4u6fU9quL7XjbC3
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@apps/docs/content/features/compliance.mdx`:
- Line 69: Update the compliance documentation sentence describing the “Only
providers that meet policy requirements” toggle to clarify that it hides
unselected incompatible providers while retaining selected incompatible
providers so users can remove them.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro Plus
Run ID: c23c8f8f-4f5f-4a4f-8516-1babaae99743
📒 Files selected for processing (5)
apps/docs/content/features/compliance.mdxapps/docs/content/features/custom-providers.mdxapps/ui/src/app/dashboard/[orgId]/org/compliance/compliance-client.tsxapps/ui/src/components/dashboard/dashboard-sidebar.tsxpackages/shared/src/components/multi-provider-selector.tsx
🚧 Files skipped from review as they are similar to previous changes (1)
- apps/ui/src/app/dashboard/[orgId]/org/compliance/compliance-client.tsx
|
|
||
| - A **green shield** marks a provider that meets every active certification, data-policy, and headquarters requirement. | ||
| - A **red shield** marks a provider that does not; the requirements it misses (for example "May log prompts" or "Headquartered in China, which is not an allowed country") are listed under its name. | ||
| - The **"Only providers that meet policy requirements"** toggle at the top of the dropdown hides incompatible providers entirely. |
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Describe the retained incompatible selections.
Line 69 says that the filter hides incompatible providers entirely. MultiProviderSelector keeps selected incompatible providers visible so users can remove them. State that the filter hides unselected incompatible providers.
Proposed documentation fix
-- The **"Only providers that meet policy requirements"** toggle at the top of the dropdown hides incompatible providers entirely.
+- The **"Only providers that meet policy requirements"** toggle at the top of the dropdown hides unselected incompatible providers. Selected incompatible providers remain visible so you can remove them.📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| - The **"Only providers that meet policy requirements"** toggle at the top of the dropdown hides incompatible providers entirely. | |
| - The **"Only providers that meet policy requirements"** toggle at the top of the dropdown hides unselected incompatible providers. Selected incompatible providers remain visible so you can remove them. |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@apps/docs/content/features/compliance.mdx` at line 69, Update the compliance
documentation sentence describing the “Only providers that meet policy
requirements” toggle to clarify that it hides unselected incompatible providers
while retaining selected incompatible providers so users can remove them.
smakosh
added this pull request to the merge queue
Jul 31, 2026
pull Bot
pushed a commit
to soitun/llmgateway
that referenced
this pull request
Aug 1, 2026
## Summary Revamps the org **Custom Models** page into a full **Models** directory at `/dashboard/<orgId>/org/models` (the old `org/custom-models` URL redirects, preserving query params). Follow-up to theopenco#3335. ### Directory (all roles) - Reuses the shared `AllModels` table component from the public `/models` page — same search, capability toggles, provider/price/context filters, table/grid views, sorting and pagination — instead of the old hand-rolled table. - Shows **all catalog models plus the org's custom models in one table**. Custom models are addressed as `<provider>/<model>`, carry a `Custom` badge, and don't link to (nonexistent) public model pages; the copy button copies the exact model string to request. - New **Source** filter (Catalog & custom / Catalog / Custom), shown only when the org has custom models. - **Compliance eligibility, fully implemented**: when the org is enterprise and its provider compliance policy is enabled, every provider mapping is evaluated client-side with the exact same `@llmgateway/models` predicates the gateway enforces at request time (`getProviderComplianceFailures`, `getAttestationComplianceFailures`, provider/model allow/deny lists, custom-provider attestations). Ineligible rows render greyed out with a Ban icon whose tooltip lists the precise failure reasons (e.g. "No SOC 2 Type 2 report"); grid cards get a "Not eligible" badge. A new **Eligible only** toggle filters them out. - The shared component changes are opt-in (`source`/`blockedReasons` fields + `hideHeader` prop): the public `/models` page and DevPass directory are unchanged (verified in-browser). ### Management (owner/admin only) - The custom-model catalog management (attestation card, catalog-only switch, add/edit/delete) moved below the directory, unchanged in behavior, and stays hidden for developer-role members — the page is purely read-only for them (from the first commit of this PR). - Sidebar entry renamed to **Models**; compliance/provider-keys links updated. - `failureLabel` extracted to `apps/ui/src/lib/compliance-failure-labels.ts`, shared with the compliance page. ## Testing - Drove the full stack via Playwright as `admin@example.com` and `developer@example.com` on the seeded enterprise org with an enabled policy (`requireSoc2Type2` + `blockApiTraining`) and a custom provider (passing attestation) with two custom models: - 33/50 visible rows correctly marked ineligible; custom rows eligible; tooltip reasons correct; "Eligible only" removes all blocked rows; Source=Custom shows only the 2 custom models; old URL redirects. - Developer: Read-only badge, no management UI at all. - Public `/models`: no new filters leak, header intact (pre-existing dev-mode hydration warning confirmed present on main too). - `pnpm format`, full `pnpm build`, and unit tests (compliance + model-categories specs) pass. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a unified Models directory for catalog and custom models. * Added custom-model badges, eligibility indicators, blocked-reason tooltips, and filtering options. * Added read-only access for users without management permissions. * Restricted model creation, editing, deletion, attestations, and management controls to authorized administrators. * Non-administrators now see catalog-only status instead of an interactive setting. * **Improvements** * Legacy custom-model links now redirect to the Models directory while preserving search parameters. * Compliance labels and provider-related navigation are now consistent across the experience. <!-- end of auto-generated comment: release notes by coderabbit.ai --> ## Docs & changelog - New feature page `features/models-directory.mdx` and Knowledge base page `learn/models` (light/dark screenshots), registered in the learn index/meta. - Stale "Custom Models" page references in the compliance and custom-providers docs renamed to "Models". - Changelog entry `org-models-directory` with generated OG image. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Addresses two issues reported by an enterprise customer:
1. Developer-role members saw an empty Custom Models page and had no way to see which providers/models their org makes available. The
GET /keys/provider,GET /keys/provider/active, andGET /custom-modelsread endpoints scoped orgs throughgetAdminOrganizationIds(), sodevelopermemberships silently got200 [].getActiveUserOrganizationIds), so project-scoped developers can browse the org's providers and custom-model catalog. Provider-key tokens remain masked, and every mutation stays owner/admin-gated server-side (POST /keys/providerrole check,getManageableProviderKey, admin-scoped PATCH/DELETE).useTeamMembersrole lookup as the compliance page.2. The compliance page showed every provider as blocked with no explanation. Selecting only a custom provider in
allowedProvidersblocks all catalogue providers by allow-list semantics, and the Provider Impact card only rendered a boolean Ban icon.@llmgateway/modelscompliance predicates are now backed by reason-returning variants —getProviderComplianceFailures,getDataPolicyComplianceFailures,getProviderRefPolicyListFailures,getAttestationComplianceFailures(+ComplianceFailureReasontype). The boolean predicates are reimplemented on top of them, so gateway enforcement behavior is unchanged.Testing
packages/models/src/compliance.spec.ts).pnpm test:unit— 208 files, 3457 tests pass.pnpm buildandpnpm formatpass.🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Access Control
Bug Fixes