Skip to content

Log A/B test buckets for Face/Touch recommend visited#11496

Merged
aduth merged 2 commits intomainfrom
aduth-ft-recommend-ab-logging
Nov 12, 2024
Merged

Log A/B test buckets for Face/Touch recommend visited#11496
aduth merged 2 commits intomainfrom
aduth-ft-recommend-ab-logging

Conversation

@aduth
Copy link
Copy Markdown
Contributor

@aduth aduth commented Nov 12, 2024

🎫 Ticket

Supports LG-14191

🛠 Summary of changes

Updates allowlist of events for Face/Touch Unlock recommendation to SMS users to ensure that the A/B test buckets are logged with the webauthn_platform_recommended_visited event.

Per the test document (linked from ticket), the primary metric to consider with this test is the rate at which users go on to successfully set up Face or Touch Unlock after being presented the recommendation. Since this needs to be broken down between account creation and sign-in, we need to include the bucket to differentiate.

Example planned CloudWatch query:

filter name in ['webauthn_platform_recommended_visited', 'webauthn_setup_submitted'] and properties.ab_tests.recommend_webauthn_platform_for_sms_user.bucket = 'recommend_for_account_creation'
| filter name != 'webauthn_setup_submitted' or (properties.event_properties.platform_authenticator and properties.event_properties.success)
| stats sum(name = 'webauthn_platform_recommended_visited') > 0 as visited, sum(name = 'webauthn_setup_submitted') > 0 as setup by properties.user_id
| stats sum(setup)/sum(visited)

📜 Testing Plan

Verify that A/B test buckets are logged when presented recommendation, and when finishing setup.

Configure recommendation to always show in local development:

# config/application.yml
recommend_webauthn_platform_for_sms_ab_test_account_creation_percent: 100
  1. Run make watch_events in a separate terminal process
  2. On a device eligible for Face or Touch Unlock, go to http://localhost:3000
  3. Click "Create an account"
  4. Continue account creation up to MFA selection screen
  5. Choose and setup phone as your sole MFA
  6. Observe that you see Face/Touch recommendation after setting up SMS
  7. See logged event for webauthn_platform_recommended_visited includes properties.ab_tests.recommend_webauthn_platform_for_sms_user.bucket of "recommend_for_account_creation"
  8. Click "Set up Face or Touch Unlock"
  9. Setup Face or Touch Unlock
  10. See logged event webauthn_setup_submitted includes properties.ab_tests.recommend_webauthn_platform_for_sms_user.bucket of "recommend_for_account_creation"

changelog: Internal, A/B Tests, Log A/B test buckets for Face/Touch recommend visited
@aduth aduth requested a review from a team November 12, 2024 14:54
@aduth aduth merged commit 47594d2 into main Nov 12, 2024
@aduth aduth deleted the aduth-ft-recommend-ab-logging branch November 12, 2024 15:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants