A/B testing infrastructure updates

[matthinz]

🎫 Ticket

Relates to work for LG-12451

🛠 Summary of changes

This PR proposes a few changes to our A/B testing system. I think we are in a lull between A/B tests right now, so it seemed like a good opportunity.

The goals here are:

1. Make it easier to add a new A/B test
2. Make it easier to understand how different A/B tests work
3. Make A/B test analytics logging less error-prone

Here's how I'm proposing to do those things:

Rename AbTestBucket to AbTest

Not to get too bikesheddy, but I think what we're describing in the config/initializers/ab_test.rb file are tests, not buckets, so I think this name is more appropriate.

Move discriminator calculation into the A/B test definition

When determining which bucket to put a user in for an A/B test, we first need to calculate a discriminator. Examples of common discriminators:

• uuid Ensures that the same user is always placed in the same bucket
• document_capture_session_uuid ensures that, for a given identity verification attempt, the user is always placed in the same bucket. If they restart IdV, they may be placed in a new bucket.

It is sometimes hard to figure out what is being used as a discriminator for a particular A/B test. This PR updates AbTest to accept a block that can be used to determine the discriminator:

MY_AB_TEST = AbTest.new(
  experiment_name: 'My fun A/B test',
  buckets: {
    a: 50,
    b: 50,
  }
) do |request:, service_provider:,  session:, user:, user_session:|
  user.uuid
end

I believe this makes it clearer what discriminator is actually in use, and makes it so you don't have to go digging through controller files to find out.

If no block is provided, the user's UUID is used as a discriminator. If the user is not logged in, no bucket will be assigned by default.

Automatically add A/B test buckets to analytics events

This PR updates the Analytics class to augment analytics events with bucket assignments for the user for any running A/B tests. Here is what that looks like:

{
  "name": "IdV: doc auth hybrid handoff visited",
  "properties": {
    "event_properties": { /* ... */  },
    /* ... */
    "ab_tests": {
      "acuant_sdk": {
        "bucket": "use_alternate_sdk"
      }
    }
  }

Importantly:

• If the user is not assigned to any buckets (for example, non-authenticated users can't be bucketed for IdV purposes), then no information will be logged for that test
• If the user is not assigned to buckets for any tests, then no ab_tests key will be logged for the event.

Warning

This is a breaking change. If you were previously counting on this data being in a certain place for a dashboard, now it will be in another place.

Limiting what events get annotated with A/B test bucket data

The should_log option can be set to a Regexp to limit what events get augmented with A/B test data. For example, this event is limited to identity-verification related events (events that, by convention, start with IdV, Idv, or idv):

AbTest.new(
  experiment_name: "my test"
  buckets: { a: 50, b: 50 },
  should_log: /idv/i
)

📜 Testing Plan

Test with no A/B tests enabled

1. Create an account and run through identity verification
2. Verify that any events logged in your logs/events.log do not include an ab_tests property

• Hint: tail -f log/events.log | jq '{name, ab_tests: .properties.ab_tests}'

Test with DOC_AUTH_VENDOR A/B test enabled

1. Update application.yml:

  doc_auth_vendor: 'mock'
  doc_auth_vendor_randomize: true
  doc_auth_vendor_randomize_alternate_vendor: 'something_else'
  doc_auth_vendor_randomize_percent: 50

2. Run through identity verification
3. Verify that "IdV" events logged in log/events.log include an ab_tests property (you may get an error that something_else is not a valid doc auth vendor since... it isn't)

Test with ACUANT_SDK A/B test enabled

1. Update application.yml:

  idv_acuant_sdk_upgrade_a_b_testing_enabled: false
  idv_acuant_sdk_upgrade_a_b_testing_percent: 50
  idv_acuant_sdk_version_alternate: '11.9.2'
  idv_acuant_sdk_version_default: '11.9.3'

2. Run through identity verification
3. Verify that "IdV" events logged in log/events.log include an ab_tests property.

[zachmargolis]

so as long as we're here, an idea for the future. Doesn't have to be this PR but something on my mind

I think it would be nice for future, new AB tests to not use SHA256 because it's slow, we could find some other faster stable digest and use that. (maybe something like this)

To maintain legacy compatibility and not redo existing AB tests, we would need to add a use_legacy_digest: that defaults to false and opt in our existing tests

[matthinz]

Yeah, I like this idea. Captured as LG-14187

[matthinz]

I updated the PR description to include some manual testing steps. Out of that I realized that there were no tests for the AbTest initializers, so I added those in f5f1d18.

[amirbey]

👋🏿 @matthinz ... assuming you're getting rid of this b/c it's noisy?

[matthinz]

Yep, bucket will now be present on ab_tests.acuant_sdk.bucket

[amirbey]

LGTM ... I ran through the tests and worked as expected

also tested with idv_acuant_sdk_upgrade_a_b_testing_enabled: true