This gem is a fork of Split gem, with several adjustments to our needs. Split is a rack based ab testing framework designed to work with Rails, Sinatra or any other rack based app. Split is heavily inspired by the Abingo and Vanity rails ab testing plugins and Resque in its use of Redis. Split is designed to be hacker friendly, allowing for maximum customisation and extensibility.
Funtestic doesn't use a database, it uses redis as a datastore.
Funtestic only supports redis 2.0 or greater.
If you're on OS X, Homebrew is the simplest way to install Redis:
$ brew install redis
$ redis-server /usr/local/etc/redis.conf
You now have a Redis daemon running on 6379.
Add Funtestic to your Gemfile:
gem 'funtestic', :git => '[email protected]:playedonline/Funtestic.git', :require => 'Funtestic/dashboard'
Then run:
$ bundle install
Funtestic is autoloaded when rails starts up, as long as you've configured redis it will 'just work'.
To begin your ab test use the ab_test
method, naming your experiment with the first argument and then the different alternatives which you wish to test on as the other arguments (or define them in Configuration as explained later on).
ab_test
returns one of the alternatives, if a user has already seen that test they will get the same alternative as before, which you can use to split your code on.
It can be used to render different templates, show different text or any other case based logic.
report_attempt
is used to track a conversion attempt - an action which is performed prior to making a conversion that might lead to a conversion.
finished
is used to count a conversion.
Conversion rate can then be calculated as conversion divided by converstion attempts.
Example: View
<% ab_test("login_button", "/images/button1.jpg", "/images/button2.jpg") do |button_file| %>
<%= img_tag(button_file, :alt => "Login!") %>
<% end %>
Example: Controller
def register_new_user
# See what level of free points maximizes users' decision to buy replacement points.
@starter_points = ab_test("new_user_free_points", '100', '200', '300')
end
Example: Boolean alternatives:
@logo_as_thumb = ab_test("logo_as_thumb_ab_test_17-9-13", "true", "false") == "true"
Example: Conversion tracking (in a controller!)
def buy_new_points
# some business logic
finished("new_user_free_points")
end
Example: Conversion tracking (in a view)
Thanks for signing up, dude! <% finished("signup_page_redesign") %>
You can find more examples, tutorials and guides on the Split wiki.
Note that in some tests we may want to allow multiple conversions & conversion attempts per user, whereas in others we don't. To disable multiple conversion pass in the following option:
report_attempt("logo_as_thumb_ab_test_17-9-13", {:multiple_conversion => false})
finished("logo_as_thumb_ab_test_17-9-13", {:multiple_conversion => false})
When multiple conversion is disabled, calling report_attempt
or finished
more than once per user will simply be ignored.
In some cases you may want a user to join an ab test only if he answers a certain criteria -
for example, only add new visitors to a test. If the user isn't new you want to retrieve his selected alternative (if he joined the test when he was a new visitor) or use a default value in case he never participated.
For that purpose you can use get_alternative_for_user_or_default
.
Example: Getting user's alternative without adding him to a test
if @is_new_visitor
@logo_as_thumb = ab_test("logo_as_thumb_ab_test_17-9-13") == "true"
else
@logo_as_thumb = get_alternative_for_user_or_default("logo_as_thumb_ab_test_17-9-13", "false") == "true"
...
Channels are just additional information which can be assigned to an alternative and later be retrieved by the Alternative.get_all_historical_alternatives
method (which we use to send lookup data to Splunk).
Example: Setting channels for alternatives
config.experiments = {
"abtest_test_19-8-13" => {
:alternatives => [
{ :name => "true", :percent => 50, :channel => 123 },
{ :name => "false", :percent => 50, :channel => 456 }
]
},
Each alternative also keeps a hash of custom counters called events. You can increment the values of your custom events using abtest_event
, which will increment the specified counter by the specified amount for the alternative of the user (if the user isn't a participant nothing will happen).
Example: Incrementing counts of custom events
abtest_event(abtest_name, {:install_click => count_of_installs})
Perhaps you only want to show an alternative to 10% of your visitors because it is very experimental or not yet fully load tested.
To do this you can pass a weight with each alternative in the following ways:
ab_test('homepage design', {:name => 'Old', :weight => 20}, {:name => 'New', :weight => 2})
ab_test('homepage design', 'Old', {:name => 'New', :weight => 0.1})
ab_test('homepage design', {:name => 'Old', :weight => 10}, 'New')
This will only show the new alternative to visitors 1 in 10 times, the default weight for an alternative is 1.
Another way to control the chosen alternative for user is to use percentage.
Example: Setting percentage for alternatives
Funtestic.configuration.experiments[:my_experiment] = {
:alternatives => [
{ :name => "control_opt", :percent => 67 },
{ :name => "second_opt", :percent => 10 },
{ :name => "third_opt", :percent => 23 },
],
}
For development and testing, you may wish to force your app to always return an alternative. You can do this by passing it as a parameter in the url.
If you have an experiment called button_color
with alternatives called red
and blue
used on your homepage, a url such as:
http://myawesomesite.com?button_color=red
will always have red buttons. This won't be stored in your session or count towards to results, unless you set the store_override
configuration option.
By default Funtestic will avoid users participating in multiple experiments at once. This means you are less likely to skew results by adding in more variation to your tests.
To stop this behaviour and allow users to participate in multiple experiments at once enable the allow_multiple_experiments
config option like so:
Funtestic.configure do |config|
config.allow_multiple_experiments = true
end
Funtestic comes with two built-in persistence adapters for storing users and the alternatives they've been given for each experiment.
By default Funtestic will store the tests for each user in the session.
You can optionally configure Funtestic to use a cookie or any custom adapter of your choosing.
Funtestic.configure do |config|
config.persistence = :cookie
end
Note: Using cookies depends on ActionDispatch::Cookies
or any identical API
Your custom adapter needs to implement the same API as existing adapters.
See Funtestic::Persistance::CookieAdapter
or Funtestic::Persistence::SessionAdapter
for a starting point.
Funtestic.configure do |config|
config.persistence = YourCustomAdapterClass
end
You can define methods that will be called when a user's alternative is selected and when a conversion & conversion attempt are reported to Funtestic.
For example:
Funtestic.configure do |config|
config.on_trial_choose = :log_trial_choice
config.on_attempt_reported = :handle_experiment_conversion_attempt
config.on_finish_reported = :handle_experiment_conversion
end
Set these attributes to a method name available in the same context as the
ab_test
method (i.e application_controller).
The first method should accept one argument, a Trial
instance, the other two accept an alternative_id (int) which is the unique id of the alternative of the user to which a conversion/attempt were reported.
def log_trial_choose(trial)
logger.info "experiment=%s alternative=%s user=%s" %
[ trial.experiment.name, trial.alternative, current_user.id ]
end
def handle_experiment_conversion_attempt(alternative_id)
# Report conversion attempt to Splunk
EventLog.log_event ExperimentConversionAttemptEvent.new({:group_ids => alternative_id, :user_id => @user_id})
end
def handle_experiment_conversion(alternative_id)
# Report conversion to Splunk
EventLog.log_event ExperimentConversionEvent.new({:group_ids => alternative_id, :user_id => @user_id})
end
Funtestic comes with a Sinatra-based front end to get an overview of how your experiments are doing.
If you are using Rails 3: You can mount this inside your app routes by first adding this to the Gemfile:
gem 'funtestic', :git => '[email protected]:playedonline/Funtestic.git', :require => 'Funtestic/dashboard'
Then adding this to config/routes.rb
mount Funtestic::Dashboard, at: 'funtestic'
You may want to password protect that page, you can do so with Rack::Auth::Basic
(in your funtestic initializer file)
Funtestic::Dashboard.use Rack::Auth::Basic do |username, password|
username == 'admin' && password == 'p4s5w0rd'
end
You can even use Devise or any other Warden-based authentication method to authorize users. Just replace mount Funtestic::Dashboard, :at => 'funtestic'
in config/routes.rb
with the following:
mount Funtestic::Dashboard, at: 'funtestic', :constraints => lambda { |request|
request.env['warden'].authenticated?
# or even check any other condition such as request.env['warden'].user.is_admin?
}
More information on this here
You can override the default configuration options of Funtestic like so:
Funtestic.configure do |config|
config.db_failover = true # handle redis errors gracefully
config.db_failover_on_db_error = proc{|error| Rails.logger.error(error.message) }
config.allow_multiple_experiments = true
config.enabled = true
config.persistence = Funtestic::Persistence::SessionAdapter
end
In most scenarios you don't want to have AB-Testing enabled for web spiders, robots or special groups of users. Funtestic provides functionality to filter this based on a predefined, extensible list of bots, IP-lists or custom exclude logic.
Funtestic.configure do |config|
# bot config
config.robot_regex = /my_custom_robot_regex/ # or
config.bots['newbot'] = "Description for bot with 'newbot' user agent, which will be added to config.robot_regex for exclusion"
# IP config
config.ignore_ip_addresses << '81.19.48.130' # or regex: /81\.19\.48\.[0-9]+/
# or provide your own filter functionality, the default is proc{ |request| is_robot? || is_ignored_ip_address? }
config.ignore_filter = proc{ |request| CustomExcludeLogic.excludes?(request) }
end
Instead of providing the experiment options inline, you can store them in a hash. This hash can control your experiment's alternatives, weights and algorithm:
Funtestic.configure do |config|
config.experiments = {
"my_first_experiment" => {
:alternatives => ["a", "b"]
},
"my_second_experiment" => {
:algorithm => 'Funtestic::Algorithms::Whiplash',
:alternatives => [
{ :name => "a", :percent => 67 },
{ :name => "b", :percent => 33 }
]
}
}
end
You can also store your experiments in a YAML file:
Funtestic.configure do |config|
config.experiments = YAML.load_file "config/experiments.yml"
end
You can then define the YAML file like:
my_first_experiment:
alternatives:
- a
- b
my_second_experiment:
alternatives:
- name: a
percent: 67
- name: b
percent: 33
This simplifies the calls from your code:
ab_test("my_first_experiment")
and:
finished("my_first_experiment")
You might wish to track generic metrics, such as conversions, and use
those to complete multiple different experiments without adding more to
your code. You can use the configuration hash to do this, thanks to
the :metric
option.
Funtestic.configure do |config|
config.experiments = {
"my_first_experiment" => {
:alternatives => ["a", "b"],
:metric => :my_metric,
}
}
end
Your code may then track a completion using the metric instead of the experiment name:
finished(:my_metric)
You can also create a new metric by instantiating and saving a new Metric object.
Funtestic::Metric.new(:my_metric)
Funtestic::Metric.save
You might wish to allow an experiment to have multiple, distinguishable goals. The API to define goals for an experiment is this:
ab_test({"link_color" => ["purchase", "refund"]}, "red", "blue")
or you can you can define them in a configuration file:
Funtestic.configure do |config|
config.experiments = {
"link_color" => {
:alternatives => ["red", "blue"],
:goals => ["purchase", "refund"]
}
}
end
To complete a goal conversion, you do it like:
finished("link_color" => "purchase")
Due to the fact that Redis has no automatic failover mechanism, it's
possible to switch on the db_failover
config option, so that ab_test
and finished
will not crash in case of a db failure. ab_test
always
delivers alternative A (the first one) in that case.
It's also possible to set a db_failover_on_db_error
callback (proc)
for example to log these errors via Rails.logger.
You may want to change the Redis host and port Funtestic connects to, or set various other options at startup.
Funtestic has a redis
setter which can be given a string or a Redis
object. This means if you're already using Redis in your app, Funtestic
can re-use the existing connection.
String: Funtestic.redis = 'localhost:6379'
Redis: Funtestic.redis = $redis
For our rails apps we have a config/initializers/funtestic.rb
file where
we set the Redis information appropriately.
Here's our initializer config/initializers/funtestic.rb
:
Funtestic.configure do |config|
config.db_failover = true # handle redis errors gracefully
config.db_failover_on_db_error = proc { |error| Rails.logger.error(error.message) }
config.allow_multiple_experiments = true
config.enabled = true
config.persistence = Funtestic::Persistence::CookieAdapter
config.on_attempt_reported = :handle_experiment_conversion_attempt
config.on_finish_reported = :handle_experiment_conversion
config.experiments = {...}
end
if Rails.env.production?
redis_config = YAML.load_file(Rails.root + 'config/redis.yml')[Rails.env]
redis_config_hash = {host: redis_config['host'], port: redis_config['port']}
else
redis_config_hash ={:host => "localhost", :port => 6379}
end
Funtestic.redis = "#{redis_config_hash[:host]}:#{redis_config_hash[:port]}"
If you're running multiple, separate instances of Funtestic you may want to namespace the keyspaces so they do not overlap. This is not unlike the approach taken by many memcached clients.
This feature is provided by the redis-namespace library, which Funtestic uses by default to separate the keys it manages from other keys in your Redis server.
Simply use the Funtestic.redis.namespace
accessor:
Funtestic.redis.namespace = "funtestic:blog"
We recommend sticking this in your initializer somewhere after Redis is configured.
Funtestic provides the Helper module to facilitate running experiments inside web sessions.
Alternatively, you can access the underlying Metric, Trial, Experiment and Alternative objects to conduct experiments that are not tied to a web session.
# create a new experiment
experiment = Funtestic::Experiment.find_or_create('color', 'red', 'blue')
# create a new trial
trial = Funtestic::Trial.new(:experiment => experiment)
# run trial
trial.choose!
# get the result, returns either red or blue
trial.alternative.name
# if the goal has been achieved, increment the successful completions for this alternative.
if goal_acheived?
trial.complete!
end
By default, Funtestic ships with an algorithm that randomly selects from possible alternatives for a traditional a/b test.
An implementation of a bandit algorithm is also provided.
Users may also write their own algorithms. The default algorithm may be specified globally in the configuration file, or on a per experiment basis using the experiments hash of the configuration file.
- Split::Export - easily export ab test data out of Split
- Split::Analytics - push test data to google analytics
- Split::Mongoid - store data in mongoid instead of redis
Ryan bates has produced an excellent 10 minute screencast about split on the Railscasts site: A/B Testing with Split
Source hosted at GitHub. Report Issues/Feature requests on GitHub Issues. Discussion at Google Groups
Tests can be ran with rake spec
Copyright (c) 2013 Andrew Nesbitt. See LICENSE for details.