feat(assistant): gate activation rail on web-assigned variation (JARVIS-1102)#33421
feat(assistant): gate activation rail on web-assigned variation (JARVIS-1102)#33421alex-nork wants to merge 2 commits into
Conversation
…IS-1102) Remove the independent LD flag evaluation for the activation flow experiment in the web pre-chat context. The activation rail now gates exclusively on the variation delivered by the platform onboarding recipe endpoint (PR 5 in vellum-assistant-platform): - Treatment users: platform returns recipe with cohort=experiment-activation-flow-2026-06-03 and bootstrapTemplate=BOOTSTRAP-ACTIVATION-RAIL.md. The web client propagates these via the existing recipe path (onboarding.cohort → daemon). - Control/unassigned users: platform returns 204; recipe is null; cohort absent. This eliminates the independent second evaluation that could disagree with the warehouse-logged assignment. No second LD flag eval means the assignment the user received is exactly the experience they get — required for the retention-by-cohort analysis to be valid. Changes: - prechat-context.ts: remove activationFlowEnabled from BuildPreChatContextInput and the override block; remove the input.activationFlowEnabled ? null : recipe conditional in resolveInitialMessage call - pre-chat-flow.tsx: remove experimentActivationFlow20260603 from flag store reads and from buildPreChatContext call - feature-flag-registry.json (meta + apps/web): change scope from 'client' to 'assistant' (daemon resolver picks it up, web client no longer needs it); update description to match string-variation contract - Tests: rewrite activation-rail tests to use recipe input instead of activationFlowEnabled flag; update feature-flag-catalog test for new scope Companion to vellum-assistant-platform PRs: #8166 (flag string variations) #8173 (web eval + log + persist) #8174 (cohort delivery via recipe) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fed36a6a4e
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fed36a6a4e
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
The activation rail is gated by the onboarding recipe cohort
(control/treatment) assigned vid-keyed on the platform, not by a boolean
feature flag — nothing in apps/web reads experiment-activation-flow-2026-06-03
as a flag anymore. Declaring it in the registry put it in
ASSISTANT_FLAG_DEFAULTS, so Settings → Feature Flags rendered it as a
manual toggle that could PATCH an {enabled} override and diverge from the
vid-keyed assignment. Remove the registry entry entirely (canonical meta/
+ synced apps/web copy) and assert it is absent from both flag stores.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Codex Review: Didn't find any major issues. Keep it up! ℹ️ About Codex in GitHubCodex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback". |
|
Codex Review: Didn't find any major issues. Bravo. ℹ️ About Codex in GitHubCodex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback". |
|
Closing: the marketing-experimentation owner is building first-class post-signup experiment-assignment support into the rail he owns, which supersedes the emission/eval/persistence work here. We'll restart with a minimal assistant-side consumer once his mechanism + the handoff contract exist. Context preserved in the plan: vellum-assistant .private/plans/activation-experiment-retention-rail.md. Ref JARVIS-1102. |
1 participant
Summary
Gates the activation flow experiment rail exclusively on the variation delivered by the platform's onboarding recipe endpoint, eliminating the independent LD flag re-evaluation in the web pre-chat context.
Why this matters
The
experiment-activation-flow-2026-06-03flag was previously gated client-side viauseClientFeatureFlagStore.use.experimentActivationFlow20260603()— a boolean LD eval that happened independently of the warehouse-logged vid-keyed assignment. This created a risk of the logged assignment diverging from the user experience, which would corrupt retention-by-cohort analysis.How it works after this PR
(app)/layout.tsxcallsgetActivationAssignment()which logssource:"experiment"+ persistsAssistantExperimentAssignmentto Django (platform PR fix: add --require to Ruby per-interpreter value flags #8173)fetchOnboardingRecipe()returns a recipe withcohort=experiment-activation-flow-2026-06-03andbootstrapTemplate=BOOTSTRAP-ACTIVATION-RAIL.mdfor treatment users (platform PR fix: gate wake word coordinator by enabled setting and add logout teardown #8174)buildPreChatContextpropagatescohort + bootstrapTemplateintoonboarding.cohort→ daemon reads itChanges
prechat-context.ts: removeactivationFlowEnabledfromBuildPreChatContextInputand the override block; remove the conditional that passednullrecipe toresolveInitialMessagefor activation userspre-chat-flow.tsx: removeexperimentActivationFlow20260603flag store read and theactivationFlowEnabledprop frombuildPreChatContextcallfeature-flag-registry.json(meta + apps/web): scopeclient→assistant; update description to document the string-variation contractDependency chain
Wave 1: platform #8166 (flag string variations) + #8167 (dbt experiments) + #8168 (sink widening)
Wave 2: platform #8173 (web eval + log + persist)
Wave 3: platform #8174 (cohort delivery via recipe)
Wave 4: this PR (assistant — gate rail on delivered variation)
Closes JARVIS-1102