- AB Testing
- Terms
- Package API References
- Configuration File Reference
- Credits
The AB Testing library implements multi-variance testing with a deterministic algorithm. It doesn't require any complex backend or database. Supporting both Javascript and Python.
The AB Testing library segmentating users under different "cohorts", so you could use an if-else-like statement to implement different features base on the segmentation.
There're 2 ways of segmentation:
- Percentage based segmentation: Randomly allocate certain percent of user to a "cohort". The allocation algorithm guarantees to produce stable results within a single experiment, and produce a different set of users for another experiment in a random way. You can also further filter the user set using the key
allocation_criteria. Take a look at this example:first 25% users, it will always result into the same group of users in experimentA, but results in another completely different group of users for experimentB. - User profile based segmentation: Manually assign a specific group of users to a cohort. This can be used to achieve features like "expose a new feature to the company emplyees for internal testing". Note the profile based segmentation works independently from the percentage based method. They doesn't interfere each other.
The segmentation decisions are made base on a configuration file. The file should be json formated and hosted centralized place. More details about the configuration file below.
The config file can be optionaly hashed so the content is not reversable. In this case you can host the file publicly, for example, on you CDN server, or in a publich S3 bucket.
Experiment: A subject that you want to do ab-testing with. You can have multiple experiments on going at the same time, the library can handle them independently.
Cohort: Inside an experiment, users are classified into groups. The groups are called Cohorts.
control: The name of the default cohort. Which is, the cohort returned when the user doesn't matche any rule on any cohorts, or the experiment just doesn't exists.
React binding for the vanilla Javascript package, featuring similar functionality.
npm install @appannie/react-ab-testing
# or
yarn add @appannie/react-ab-testingWrap your app with the ABTestingController
import { ABTestingController } from '@appannie/react-ab-testing';
const MyApp = ({ user }) => {
const profile = {
persona: user.persona,
employee: user.isEmployee,
};
return (
<ABTestingController config={testConfig} userId={user.id} userProfile={profile}>
<App />
</ABTestingController>
);
};The required props are:
config: the configuration object.userId: a unique identifier for your current user. This ID should be the same across visits to make sure your user always end up in the same cohorts. It can astringor anumber.userProfile: a key/value map used to force include a user in given cohorts.
Then within your app, check the cohort a user is assigned to using the useCohortOf hook.
import { useCohortOf } from '@appannie/react-ab-testing';
const Component = () => {
const cohort = useCohortOf('experiment-name');
switch (cohort) {
case 'blue':
return <BlueButton />;
case 'red':
return <RedButton />;
// 'control' is the default cohort. All experiments have a control cohort.
case 'control':
default:
return <Default />;
}
};
Core Javascript package that implementing ab-testing SDK APIs.
npm install @appannie/ab-testing
# or
yarn add @appannie/ab-testingimport { Experiments } from '@appannie/ab-testing';
const profile = {
persona: user.persona,
employee: user.isEmployee,
};
const experiments = new Experiments(config, user.id, profile);
const cohort = experiments.getCohort('experiment-name');
switch (cohort) {
case 'blue':
console.log('user in the blue cohort');
break;
case 'red':
console.log('user in the red cohort');
break;
// 'control' is the default cohort. All experiments have a control cohort.
case 'control':
default:
console.log('user in the control (default) cohort');
break;
}
An helper library that implements hashing for the configurations (user profile and the configuration force_include section) in case you want to host the config file publicly but still keep the content private, or just don't want to leave PIIs been transmitted in plain text.
The hashing algorithm used is sha256, and we're using a universal implementation (create-hash) that'll work both in Node and the browser.
npm install @appannie/ab-testing-hash-object
# or
yarn add @appannie/ab-testing-hash-objectimport { hashObject } from '@appannie/ab-testing-hash-object';
const profile = hashObject(
{
persona: user.persona,
employee: user.isEmployee,
},
salt
);
const forceIncludeCondition = hashObject(
{
persona: ['data analyst'],
employee: ['yes'],
},
salt
);You'll want to make sure the salt is the same on your server and your client for the encoding result to be the same.
Python implementation for the same APIs in @appannie/ab-testing and @appannie/ab-testing-hash-object
pip install py-ab-testing
# or
pipenv install py-ab-testingNote: The config variable holds an dict with configuration file format that documented here.
from ABTesting import ABTestingController
user_profile = {
'persona': user.persona,
'employee': user.isEmployee,
}
controller = ABTestingController(config, user.id, user_profile)
cohort = controller.get_cohort('experiment-name')
if cohort == 'blue':
do_something()
elif cohort == 'red':
do_something_else()
else:
do_default_behavior()Similar to the Javascript SDKs, the package comes with an optional util for hashing private information with sha256.
from ABTesting.utils import hash_dict
config['salt'] = salt
for experiment in config['experiments']:
for cohort in experiment['cohorts']:
cohort['force_include'] = hash_dict(cohort['force_include'], salt)from ABTesting.utils import hash_dict
hashed_user_profile = hash_dict(
{
'persona': user.persona,
'employee': user.isEmployee,
},
salt
)
# Make sure config is hashed with the same salt
controller = ABTestingController(config, user.id, hashed_user_profile){
// The version isn't used right now, it is used to allow potentially breaking configuration
// change in the future. Current version is "1.0"
version: "1.0",
experiments: [
{
// Unique name used to refer to an experiment; used within `useCohortOf`
name: 'experiment-name',
cohorts: [
{
// Name of a cohort. These are the values returned by `useCohortOf`
name: 'blue',
// The force_include section is used to force users into given cohorts if their
// userProfile key. In many keys are used, *any* match will force include the user in
// the cohort.
// The force_include rules are checked in the order of the cohorts in the array and
// the first match wins.
// This section is optional.
force_include: {
persona: [
'data analyst'
]
},
// The users are allocated to values in a range of 0 to 100. The allocation config
// represents the slice of users allocated to a cohort. You can have multiple
// allocation ranges.
// Every range needs to be unique and not overlap other ranges in any other cohort.
allocation: [
[0, 25]
],
// The allocation_criteria key allows us to further filter the set of users by enforcing
// a criteria that must be valid before the cohort is approved. In this case,
// the user must have an email domain of data.ai or appannie.com.
allocation_criteria: {
email_domain: ['appannie.com', 'data.ai']
}
},
{
// "control" is the default cohort. All experiments always have a control cohort.
// All users not allocated to other cohorts will be assigned to "control" by default.
name: 'control'
},
]
}
]
}At Data AI, we're maintaining this configuration in yaml format to reduce the syntax noise. We have an automated CI task to encode/encrypt and push the final configuration to a public S3 bucket from where our SDK retrieve the configuration.
You shouldn't list user private information (PII) within the configuration file as these are most likely public.
If you need to segment users using emails or information you wish/need to keep private, we suggest you to securely encrypt them.
We're providing an extra module @appannie/ab-testing-hash-object aiming to simplify this flow. This module will encode the keys and the values of your user profile. You then need to call it with the same salt while preprocessing your configuration and once when defining the user profile:
import { hashObject } from '@appannie/ab-testing-hash-object';
const profile = hashObject(
{
persona: user.persona,
employee: user.isEmployee,
},
config.salt
);You can do this by force including a user or a group of users in the control cohort.
experiments:
- name: my_experiment
cohorts:
- name: control
force_include:
# Here we'd prevent any google.com users from seeing the experiment
email_domain:
- google.com
- name: cohort_A
allocation:
- [0, 10]Force include users with email_domain equal to appannie.com under the desired cohort.
experiments:
- name: my_experiment
cohorts:
- name: control
- name: cohort_A
force_include:
# Force all Data AI user under the given cohort
email_domain:
- appannie.comMade with ❤️ by Zhang Tao and Simon Boudrias from the Data AI Beijing office.
Available for public use under the MIT license.