LG-14655: A/B test to recommend platform authenticator to SMS users

[aduth]

🎫 Ticket

LG-14655

🛠 Summary of changes

Implements new A/B test to recommend Face or Touch Unlock as an authenticator option after either...

• The user chooses only SMS as their MFA methods when creating an account
• The user signs in and MFAs with SMS and doesn't already have Face or Touch Unlock associated with their account

📜 Testing Plan

Force A/B test by adding configuration to config/application.yml:

recommend_webauthn_platform_for_sms_ab_test_percent: 100

Verify you are presented recommendation when creating account with only SMS:

1. On a device eligible to set up Face or Touch Unlock (iPhone, Android), go to http://localhost:3000
   • To simplify, you can use Chrome DevTools device mode to simulate iPhone
2. Click "Create an account"
3. Continue account creation until MFA selection screen
4. Choose "Text or voice message" as your only selected MFA
5. Continue setting up phone
6. Observe that you arrive at the "Set up face or touch unlock for a more secure sign in" screen
7. Click either "Set up face or touch unlock" or "Skip"
8. Observe that you arrive where you expect
   • "Set up face or touch unlock": Face or touch unlock setup screen
   • "Skip": After-sign-in path, account page in this scenario

Verify you are presented recommendation when creating account with only SMS:

1. On main branch, repeat Steps 1-5 above
2. Click "Skip for now" when offered to set up second MFA
3. Click "Forget all browsers" on account dashboard and confirm prompt
4. Click "Sign out"
5. Switch back to aduth-lg-14655-sms-ft-recommend-ab branch
6. Sign in with the account you created
7. Authenticate with SMS
8. Repeat Steps 6-8 above

Also consider testing with some edge-case scenarios, where users should not be able to opt in...

• ... unless they're browsing in English
• ... if they're on an unsupported device for setup (e.g. desktop)
• ... if they've seen the recommendation already
• ... if they've already been recommended to set up PIV/CAC as a user with a email address from a federal domain
• ... if they create or authenticate with phone using voice channel instead of SMS
• ... if they already have platform authenticator set up

👀 Screenshots

[aduth]

Removed since:

• context is for a scenario which no longer exists (a feature flag that's been removed)
• The written test description is the opposite of what it asserts (describes being sent for second MFA, asserts being sent to account page)
• There's already a test case above that asserts the expected behavior for a single selection in the multi-MFA setup flow (source)

[jmdembe]

Should there be a test to support the creation of the RECOMMEND_WEBAUTHN_PLATFORM_FOR_SMS_USER bucket?

[aduth]

@jmdembe Ah, yeah, there is some precedent for that in ab_tests_spec.rb, I can add some test cases there.

[aduth]

@jmdembe Ah, yeah, there is some precedent for that in ab_tests_spec.rb, I can add some test cases there.

Added in 7851915.

[jmdembe]

For my understanding: does this differentiate between users in the bucket vs. not?

[aduth]

ab_test_bucket returns either one of the buckets they've been assigned to if they're in the X% group, otherwise it returns :default. So this checks that they're part of the group.

[aduth]

Request from product to be able to support different configured percentage for recommendation during sign-in vs. account creation.

Could probably handle this by creating separate buckets:

Suggested change

	buckets: {
	recommend: IdentityConfig.store.recommend_webauthn_platform_for_sms_ab_test_percent,
	},
	buckets: {
	recommend_for_account_creation: IdentityConfig.store.recommend_webauthn_platform_for_sms_ab_test_account_creation_percent,
	recommend_for_authentication: IdentityConfig.store.recommend_webauthn_platform_for_sms_ab_test_authentication_percent,
	},

[mdiarra3]

Looks like theres a merge conflict @aduth

[mdiarra3]

Everything LGTM, tested and worked as expected locally. Do we have a followup ticket to ensure set this test up to work for non english users once we get translations.

[mdiarra3]

Nice like having this split out

[aduth]

Everything LGTM, tested and worked as expected locally. Do we have a followup ticket to ensure set this test up to work for non english users once we get translations.

That's a good question. I'm assuming the current plan is to wait and see how the test performs to decide whether we want to make this permanent, and only then get the translations.

But the immediate follow-on is LG-14191.

[aduth]

Noting that having separate configurations per buckets makes local testing a little cumbersome, since they can't both be maxed out at the same time, otherwise initialization fails with "buckets exceeding 100". The solution is to set one or the other to 100 at a time and test each individually. In production, we'd have a smaller percentage, and using buckets this way allows for test candidates which are mutually exclusive from each other, which isn't strictly a requirement, but also not an issue.