feat(inference): always-seed managed profile triplet (Phase 1.2 PR-D)#30232
Conversation
Managed profile seeding becomes idempotent and provider-agnostic on the caller side: - The 3 managed profiles (balanced / quality-optimized / cost-optimized) seed on fresh installs only. Existing on-disk entries are never overwritten, so user edits via the Providers UI and platform-overlay fragments both stay authoritative across reboots. - Every managed profile carries provider_connection: "anthropic-managed". Dispatch resolves auth from that connection (seeded as platform-auth by seedCanonicalConnections); users not logged in fail soft at call time rather than at boot. - Custom-* profiles no longer auto-materialize on non-Anthropic defaults. The Providers UI is responsible for creating user-defined connection + profile pairs on demand. - llm.default.provider rewrite + llm.default.model sync are gone. Routing is owned by the active profile's provider_connection now; the legacy default block is read-only fallback for backfill of pre-1.2 configs. Test churn drops the four scenarios that exercised the removed code paths (non-Anthropic seeding, openai→anthropic re-hatch, unknown-provider rewrite, default.model sync).
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
There was a problem hiding this comment.
🔴 Stale docstring describes removed custom- profile behavior, violating code comment rule*
The JSDoc for seedInferenceProfiles (lines 87-90) still describes the custom-* profile seeding behavior that this PR removes: "When the resolved default provider is non-Anthropic, the 3 custom-* profiles are also materialized using resolveModelIntent against that provider, and custom-balanced becomes the active profile for fresh hatches." This behavior no longer exists in the function body — the CUSTOM_PROFILE_TEMPLATES constant, the non-Anthropic seeding loop, and the custom-balanced active-profile fallback were all deleted. This violates the assistant/AGENTS.md rule: "When writing or updating comments, do not reference code that has been removed. Comments should describe the current state of the codebase, not narrate its history."
(Refers to lines 87-90)
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
Docstring rewritten in commit 2nd-push-pending. Now describes the current behavior: managed-only seed, fresh-install-only writes, provider_connection wiring, and the new active-profile resolution rules (options.preserveActiveProfile + provider-mismatch reset).
| if (!shouldPreserveActiveProfile && !requestedActiveExists) { | ||
| llm.activeProfile = "balanced"; | ||
| } |
There was a problem hiding this comment.
🚩 Behavioral change: active profile with provider mismatch no longer auto-corrected
The old code had logic (removed lines 198-219) that detected when the active profile's provider mismatched the resolved default provider (e.g., custom-balanced pointing at openai after re-hatching to anthropic) and would reset the active profile. The new code at lines 137-139 only resets when the active profile doesn't exist at all. This means an existing workspace with activeProfile: "custom-balanced" and a custom-balanced profile on disk will keep that active profile forever, even if the user re-hatches to a different provider. This is likely intentional (the PR removes the concept of provider-directed profile switching), but reviewers should confirm this is the desired UX for re-hatch scenarios.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
Restored the auto-correct path under the simpler model: when options.preserveActiveProfile === false and the on-disk active profile's provider no longer matches llm.default.provider, reset to balanced. Regression test re-hatch from openai to anthropic resets stale active profile to balanced locks the OpenAI→Anthropic case and asserts the legacy custom-balanced profile is preserved on disk so the user can switch back via the Providers UI.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 66f7074783
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| if (!shouldPreserveActiveProfile && !requestedActiveExists) { | ||
| llm.activeProfile = "balanced"; |
There was a problem hiding this comment.
This branch now preserves any existing llm.activeProfile as long as the profile exists, which regresses re-hatch flows where only llm.default.provider is updated (for example, legacy overlays that switch OpenAI → Anthropic without supplying llm.activeProfile). In that case a stale legacy profile like custom-balanced remains active, and resolveCallSiteConfig("mainAgent", ...) continues routing by that profile (active profile overrides llm.default), so the user's provider change is silently ignored across restarts.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Same fix as Devin's ANALYSIS finding — when preserveActiveProfile is false and the active profile's provider no longer matches the (now-overlay-updated) llm.default.provider, the active profile resets to balanced. Covered by the new re-hatch from openai to anthropic resets stale active profile to balanced test in config-loader-backfill.test.ts.
|
Codex Review: Didn't find any major issues. Keep them coming! ℹ️ About Codex in GitHubCodex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback". |
…file on re-hatch Addresses PR-D cycle-1 review feedback: - Docstring on seedInferenceProfiles described the deleted custom-* seeding and the resolveModelIntent branch. Replaced with current-state copy: managed-only seed, fresh-install-only writes, provider_connection wiring, and the new active-profile resolution rules. - The active profile resolution preserved any existing on-disk choice as long as the named profile existed, which let a legacy OpenAI→Anthropic re-hatch (overlay updates llm.default.provider only, no activeProfile) keep a stale custom-balanced active. Now: when options.preserveActiveProfile is false and the on-disk active profile's provider no longer matches llm.default.provider, reset to 'balanced'. - Restored a regression test covering the OpenAI→Anthropic re-hatch reset under the new rules (and asserting the legacy custom-balanced profile is preserved on disk for user recovery via the Providers UI).
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 351c55b80b
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| const defaultProvider = readString(readObject(llm.default)?.provider); | ||
| activeProviderMismatch = | ||
| activeProvider !== undefined && | ||
| defaultProvider !== undefined && | ||
| activeProvider !== defaultProvider; |
There was a problem hiding this comment.
This mismatch check treats any llm.activeProfile whose provider differs from llm.default.provider as stale, then forces activeProfile = "balanced" on boot. That now clobbers legitimate user choices: the macOS settings flow persists active-profile changes by patching only llm.activeProfile (see clients/macos/vellum-assistant/Features/Settings/SettingsStore.swift lines 3345-3349), so llm.default.provider can remain anthropic while the selected profile is openai/gemini. After restart, this code will silently revert those selections to balanced, making user-selected providers non-persistent.
Useful? React with 👍 / 👎.
… managed profiles upserted every boot - Managed Anthropic profiles (balanced, quality-optimized, cost-optimized) are overwritten on every off-platform boot so Vellum can push model/config updates. On-platform, insert-if-not-exists (platform overlays are authoritative). - Off-platform hatches now create a personal provider connection (e.g. openai-personal backed by CES credential) and 3 user profiles (custom-balanced, custom-quality-optimized, custom-cost-optimized) pointing to it. activeProfile defaults to custom-balanced. - seedCanonicalConnections changed from insert-if-not-exists to upsert so managed connections can also receive updates in new releases. - DefaultWorkspaceConfigMergeResult gains hadOverlay flag so seedInferenceProfiles distinguishes hatch from regular boot. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 participant
Summary
Phase 1.2 PR-D: simplify managed inference profile seeding to align with the new
provider_connectionsmodel.What changes
balanced/quality-optimized/cost-optimized) seed on fresh installs only. Existing on-disk entries are never overwritten, so user edits via the Providers UI and platform-overlay fragments both stay authoritative across reboots.provider_connection: "anthropic-managed". Dispatch resolves auth from that connection (already seeded asplatform-auth byseedCanonicalConnections). Users not logged in to Vellum fail soft at call time rather than at boot.CUSTOM_PROFILE_TEMPLATESauto-materialize is gone. The Providers UI (PR-E / PR-F) creates user-defined connection + profile pairs on demand.llm.default.providerrewrite +llm.default.modelsync logic removed. Routing is owned by the active profile'sprovider_connection; the legacy default block stays as a read-only fallback for backfill of pre-1.2 configs.Test churn
Drops 4 obsolete scenarios that exercised the removed code paths:
non-Anthropic hatch overlay seeds custom-* profilesre-hatch from openai to anthropic resets stale custom-balancedunknown overlay provider falls back to Anthropic seedingsyncs llm.default.model to active profileplatform-provided profile fragments are not polluted by managed seedsstill passes (the new fresh-install-only check covers it).Diff
assistant/src/config/seed-inference-profiles.ts: 199 lines removed, 35 added.assistant/src/__tests__/config-loader-backfill.test.ts: 175 lines of dead-path tests removed.Sequence
Phase 1.2 PR-A #30229 and PR-B #30228 merged earlier today. PR-E #30231 (macOS) and PR-F #6405 (web in
vellum-assistant-platform) are open in parallel. This PR has no client-side dependency.