fix(macos): emit llm.activeProfile=balanced for managed-inference hatch#30343
Closed
vellum-apollo-bot[bot] wants to merge 1 commit into
Closed
fix(macos): emit llm.activeProfile=balanced for managed-inference hatch#30343vellum-apollo-bot[bot] wants to merge 1 commit into
vellum-apollo-bot[bot] wants to merge 1 commit into
Conversation
The macOS onboarding flow has two distinct local-hosting paths that share
the same hatch step:
1. Managed inference - user signs in with their Vellum account, picks
Local hosting, never enters a provider API key. Chat traffic routes
through the managed Anthropic proxy via a platform-injected
'assistant_api_key' credential.
2. BYOK - user provides their own provider API key, hatches without a
Vellum account.
Both paths were emitting the same hatch-time config overlay
('llm.default.provider: <provider>'), which the daemon seeder
unconditionally interpreted as 'user has supplied their own API key'.
The seeder materialized an 'anthropic-personal' provider connection and
- after #30232 (Phase 1.2 PR-D) flipped the activeProfile default -
set the active profile to 'custom-balanced'. That profile reads
'credential/anthropic/api_key' from CES, which doesn't exist in the
managed flow, so message sends 500'd.
Fix: branch on state.skippedAPIKeyEntry (set exclusively when the user
is authenticated and skipped the API key entry step) and emit
'llm.activeProfile: balanced' instead of 'llm.default.provider' on the
managed path. The daemon seeder already tracks this via
'providedLlmActiveProfile' from mergeDefaultWorkspaceConfig and skips
its 'isHatch ? custom-balanced : balanced' branch when
preserveActiveProfile is true and the named profile exists - so
'balanced' (managed Anthropic) stays selected through the hatch.
The logic moves into a pure top-level helper
('onboardingHatchConfigOverlay') so it can be unit-tested without
spinning up OnboardingState, matching the existing convention
('OnboardingHostingModeResolver', 'OnboardingManagedAuthState').
Fixes the 'hello-world-managed-inference' Playwright test that's been
red on every scheduled run since 2026-05-10.
Contributor
Author
|
Parking this pending the test-side approach. Vargas redirected: the fix should land in |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The macOS managed-inference onboarding path was emitting
llm.default.provider=anthropicto the daemon hatch overlay - same shape as BYOK. The daemon'sseedInferenceProfiles(assistant/src/config/seed-inference-profiles.ts) readsllm.default.provider, materializes ananthropic-personalconnection backed bycredential/anthropic/api_key, and (after #30232 flipped the default) setsllm.activeProfile = custom-balanced. In managed inference that credential never exists - the user signed in with their Vellum account and gets a platform-injectedassistant_api_keyinstead - so message sends 500'd.Fix: in
HatchingStepView.buildOnboardingConfigValues, branch onstate.skippedAPIKeyEntry(set exclusively inAPIKeyStepView.handleContinuewhen the user is authenticated and skipped the provider API key step). Managed: emitllm.activeProfile=balanced. BYOK: emitllm.default.provider=<provider>as before.The daemon's
mergeDefaultWorkspaceConfigalready tracksprovidedLlmActiveProfileand passespreserveActiveProfile=trueto the seeder. WithpreserveActiveProfile=trueandprofiles["balanced"]already seeded (it's a managed profile, always materialized),shouldPreserveActiveProfileis true and the seeder skips itsuserConnectionName ? "custom-balanced" : "balanced"branch entirely. With nollm.default.providerin the overlay, the seeder also doesn't materialize the personal connection.Why
hello-world-managed-inference(Playwright) has been red on every scheduled run since 2026-05-10 (#30232 landed May 10). Last green pre-flip: run 25569878332 on May 8.The defect predates #30232 - the macOS app has been emitting the same wrong overlay for the managed path the whole time. Before the PR-D flip, the seeder defaulted
activeProfiletobalancedeven whenanthropic-personalwas materialized, so chat still worked. PR-D flipped that default so the seeder now points at the user connection when one exists, which exposed the underlying onboarding-flow bug.How
onboardingHatchConfigOverlay(skippedAPIKeyEntry:selectedProvider:defaultProvider:)inclients/macos/vellum-assistant/Features/Onboarding/OnboardingHatchConfigOverlay.swift. Matches the existingOnboardingHostingModeResolver/OnboardingManagedAuthStateconvention of "extract logic into a top-level function so it can be unit-tested without spinning up state."HatchingStepView.buildOnboardingConfigValuesbecomes a one-line delegation.OnboardingHatchConfigOverlayTests.swiftcover both paths (managed-inference and BYOK), the empty-selectedProviderfallback, and the structural invariants (managed doesn't emitdefault.provider; BYOK doesn't emitactiveProfile).Trace through the daemon
For the managed path with this change:
{"llm": {"activeProfile": "balanced"}}to$VELLUM_DEFAULT_WORKSPACE_CONFIG_PATH.mergeDefaultWorkspaceConfigreturns{hadOverlay: true, providedLlmProfileNames: {}, providedLlmActiveProfile: true}.lifecycle.ts:522callsseedInferenceProfiles({isHatch: true, preserveActiveProfile: true, ...}).balanced).hatchProvider = readString(readObject(llm.default)?.provider)isundefined(nodefault.providerin overlay), souserConnectionNamestaysundefinedand theanthropic-personalconnection is NOT created.requestedActiveProfile = "balanced",requestedActiveExists = true,shouldPreserveActiveProfile = true. The entireif (options.isHatch)branch is skipped.llm.activeProfilestays"balanced".Test plan
OnboardingHatchConfigOverlayTestscovers the helper.hello-world-managed-inferencePlaywright run is the integration check.llm.default.provider=<X>still emitted whenskippedAPIKeyEntry == false.Related