Skip to content

fix(macos): emit llm.activeProfile=balanced for managed-inference hatch#30343

Closed
vellum-apollo-bot[bot] wants to merge 1 commit into
mainfrom
apollo/managed-inference-active-profile
Closed

fix(macos): emit llm.activeProfile=balanced for managed-inference hatch#30343
vellum-apollo-bot[bot] wants to merge 1 commit into
mainfrom
apollo/managed-inference-active-profile

Conversation

@vellum-apollo-bot
Copy link
Copy Markdown
Contributor

@vellum-apollo-bot vellum-apollo-bot Bot commented May 12, 2026

What

The macOS managed-inference onboarding path was emitting llm.default.provider=anthropic to the daemon hatch overlay - same shape as BYOK. The daemon's seedInferenceProfiles (assistant/src/config/seed-inference-profiles.ts) reads llm.default.provider, materializes an anthropic-personal connection backed by credential/anthropic/api_key, and (after #30232 flipped the default) sets llm.activeProfile = custom-balanced. In managed inference that credential never exists - the user signed in with their Vellum account and gets a platform-injected assistant_api_key instead - so message sends 500'd.

Fix: in HatchingStepView.buildOnboardingConfigValues, branch on state.skippedAPIKeyEntry (set exclusively in APIKeyStepView.handleContinue when the user is authenticated and skipped the provider API key step). Managed: emit llm.activeProfile=balanced. BYOK: emit llm.default.provider=<provider> as before.

The daemon's mergeDefaultWorkspaceConfig already tracks providedLlmActiveProfile and passes preserveActiveProfile=true to the seeder. With preserveActiveProfile=true and profiles["balanced"] already seeded (it's a managed profile, always materialized), shouldPreserveActiveProfile is true and the seeder skips its userConnectionName ? "custom-balanced" : "balanced" branch entirely. With no llm.default.provider in the overlay, the seeder also doesn't materialize the personal connection.

Why

hello-world-managed-inference (Playwright) has been red on every scheduled run since 2026-05-10 (#30232 landed May 10). Last green pre-flip: run 25569878332 on May 8.

The defect predates #30232 - the macOS app has been emitting the same wrong overlay for the managed path the whole time. Before the PR-D flip, the seeder defaulted activeProfile to balanced even when anthropic-personal was materialized, so chat still worked. PR-D flipped that default so the seeder now points at the user connection when one exists, which exposed the underlying onboarding-flow bug.

How

  • New pure top-level helper onboardingHatchConfigOverlay(skippedAPIKeyEntry:selectedProvider:defaultProvider:) in clients/macos/vellum-assistant/Features/Onboarding/OnboardingHatchConfigOverlay.swift. Matches the existing OnboardingHostingModeResolver / OnboardingManagedAuthState convention of "extract logic into a top-level function so it can be unit-tested without spinning up state."
  • HatchingStepView.buildOnboardingConfigValues becomes a one-line delegation.
  • 8 new unit tests in OnboardingHatchConfigOverlayTests.swift cover both paths (managed-inference and BYOK), the empty-selectedProvider fallback, and the structural invariants (managed doesn't emit default.provider; BYOK doesn't emit activeProfile).

Trace through the daemon

For the managed path with this change:

  1. CLI writes {"llm": {"activeProfile": "balanced"}} to $VELLUM_DEFAULT_WORKSPACE_CONFIG_PATH.
  2. Daemon boots. mergeDefaultWorkspaceConfig returns {hadOverlay: true, providedLlmProfileNames: {}, providedLlmActiveProfile: true}.
  3. lifecycle.ts:522 calls seedInferenceProfiles({isHatch: true, preserveActiveProfile: true, ...}).
  4. Seeder materializes managed profiles (incl. balanced).
  5. hatchProvider = readString(readObject(llm.default)?.provider) is undefined (no default.provider in overlay), so userConnectionName stays undefined and the anthropic-personal connection is NOT created.
  6. requestedActiveProfile = "balanced", requestedActiveExists = true, shouldPreserveActiveProfile = true. The entire if (options.isHatch) branch is skipped. llm.activeProfile stays "balanced".

Test plan

  • OnboardingHatchConfigOverlayTests covers the helper.
  • Scheduled hello-world-managed-inference Playwright run is the integration check.
  • BYOK path is unchanged in shape - llm.default.provider=<X> still emitted when skippedAPIKeyEntry == false.

Related


Open in Devin Review

The macOS onboarding flow has two distinct local-hosting paths that share
the same hatch step:

1. Managed inference - user signs in with their Vellum account, picks
   Local hosting, never enters a provider API key. Chat traffic routes
   through the managed Anthropic proxy via a platform-injected
   'assistant_api_key' credential.

2. BYOK - user provides their own provider API key, hatches without a
   Vellum account.

Both paths were emitting the same hatch-time config overlay
('llm.default.provider: <provider>'), which the daemon seeder
unconditionally interpreted as 'user has supplied their own API key'.
The seeder materialized an 'anthropic-personal' provider connection and
- after #30232 (Phase 1.2 PR-D) flipped the activeProfile default -
set the active profile to 'custom-balanced'. That profile reads
'credential/anthropic/api_key' from CES, which doesn't exist in the
managed flow, so message sends 500'd.

Fix: branch on state.skippedAPIKeyEntry (set exclusively when the user
is authenticated and skipped the API key entry step) and emit
'llm.activeProfile: balanced' instead of 'llm.default.provider' on the
managed path. The daemon seeder already tracks this via
'providedLlmActiveProfile' from mergeDefaultWorkspaceConfig and skips
its 'isHatch ? custom-balanced : balanced' branch when
preserveActiveProfile is true and the named profile exists - so
'balanced' (managed Anthropic) stays selected through the hatch.

The logic moves into a pure top-level helper
('onboardingHatchConfigOverlay') so it can be unit-tested without
spinning up OnboardingState, matching the existing convention
('OnboardingHostingModeResolver', 'OnboardingManagedAuthState').

Fixes the 'hello-world-managed-inference' Playwright test that's been
red on every scheduled run since 2026-05-10.
Copy link
Copy Markdown
Contributor

@devin-ai-integration devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 2 additional findings.

Open in Devin Review

@vellum-apollo-bot
Copy link
Copy Markdown
Contributor Author

Parking this pending the test-side approach. Vargas redirected: the fix should land in qa/agent/fixtures.ts setManagedInferenceMode so the managed-inference fixture is correct for all consumers, not in the macOS onboarding overlay. See vellum-ai/vellum-assistant-platform#TBD (replacement PR). Leaving open in case we want both layers belt-and-suspenders after the fixture change ships and we see how hello-world-managed-inference lands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant