Log A/B test buckets for Face/Touch recommend visited#11496
Merged
Conversation
changelog: Internal, A/B Tests, Log A/B test buckets for Face/Touch recommend visited
jmdembe
approved these changes
Nov 12, 2024
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🎫 Ticket
Supports LG-14191
🛠 Summary of changes
Updates allowlist of events for Face/Touch Unlock recommendation to SMS users to ensure that the A/B test buckets are logged with the
webauthn_platform_recommended_visitedevent.Per the test document (linked from ticket), the primary metric to consider with this test is the rate at which users go on to successfully set up Face or Touch Unlock after being presented the recommendation. Since this needs to be broken down between account creation and sign-in, we need to include the bucket to differentiate.
Example planned CloudWatch query:
📜 Testing Plan
Verify that A/B test buckets are logged when presented recommendation, and when finishing setup.
Configure recommendation to always show in local development:
make watch_eventsin a separate terminal processwebauthn_platform_recommended_visitedincludesproperties.ab_tests.recommend_webauthn_platform_for_sms_user.bucketof"recommend_for_account_creation"webauthn_setup_submittedincludesproperties.ab_tests.recommend_webauthn_platform_for_sms_user.bucketof"recommend_for_account_creation"