Skip to content

Commit

Permalink
Add additional variant for Related Links A/B test v3
Browse files Browse the repository at this point in the history
This commit adds an additional `C` variant for the upcoming related links A/B test iteration #3. In the previous test we saw some users who had the control variant cookie, yet navigated via the related links which are only shown on the `B` variant.

In order to check if there are any problems with our A/B testing framework, in addition to making the data collection easier for later analysis, we are introducing the new `C` variant which will become our test variant. How the variants now work are as follows:

- `A` variant: No users should ever end up in this variant, as our CDN will be configured it to assign 0% of users. If users do end up in this variant, we may have issues with our A/B testing framework which will require further investigation.
- `B` variant: Our control variant, where related links will not be changed in any way.
- `C` variant: Our test variant, where related links will be changed to use our generated links for the sample involved.
  • Loading branch information
Karl Baker committed Mar 12, 2019
1 parent 3b61a18 commit ea2819b
Show file tree
Hide file tree
Showing 4 changed files with 16 additions and 6 deletions.
2 changes: 1 addition & 1 deletion app/controllers/concerns/ab_testable.rb
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ def related_links_test
@related_links_test ||= GovukAbTesting::AbTest.new(
"RelatedLinksABTest3",
dimension: RELATED_LINKS_DIMENSION,
allowed_variants: %w(A B),
allowed_variants: %w(A B C),
control_variant: "A"
)
end
Expand Down
2 changes: 1 addition & 1 deletion app/controllers/content_items_controller.rb
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ def set_guide_draft_access_token
def load_content_item
content_item = Services.content_store.content_item(content_item_path)

if related_links_variant.variant?('B') && content_item.dig('links')
if related_links_variant.variant?('C') && content_item.dig('links')
content_item['links']['ordered_related_items'] = content_item['links'].fetch('suggested_ordered_related_items', [])
end

Expand Down
16 changes: 13 additions & 3 deletions test/controllers/content_items_controller_test.rb
Original file line number Diff line number Diff line change
Expand Up @@ -130,7 +130,7 @@ class ContentItemsControllerTest < ActionController::TestCase
assert_equal content_item['title'], assigns[:content_item].title
end

test "gets item from the content store and keeps ordered_related_items when running RelatedLinksABTest3 control variant" do
test "gets item from the content store and keeps ordered_related_items when running RelatedLinksABTest3 misclassification variant" do
with_variant RelatedLinksABTest3: 'A' do
content_item = content_store_has_schema_example('case_study', 'case_study')

Expand All @@ -140,18 +140,28 @@ class ContentItemsControllerTest < ActionController::TestCase
end
end

test "gets item from the content store and replaces ordered_related_items when running RelatedLinksABTest3 test variant" do
test "gets item from the content store and keeps ordered_related_items when running RelatedLinksABTest3 control variant" do
with_variant RelatedLinksABTest3: 'B' do
content_item = content_store_has_schema_example('case_study', 'case_study')

get :show, params: { path: path_for(content_item) }
assert_response :success
assert_equal content_item['links']['ordered_related_items'], assigns[:content_item].content_item['links']['ordered_related_items']
end
end

test "gets item from the content store and replaces ordered_related_items when running RelatedLinksABTest3 test variant" do
with_variant RelatedLinksABTest3: 'C' do
content_item = content_store_has_schema_example('case_study', 'case_study')

get :show, params: { path: path_for(content_item) }
assert_response :success
assert_equal assigns[:content_item].content_item['links']['ordered_related_items'], assigns[:content_item].content_item['links']['suggested_ordered_related_items']
end
end

test "gets item from the content store and replaces ordered_related_items when empty array when RelatedLinksABTest3 test variant has no suggestions" do
with_variant RelatedLinksABTest3: 'B' do
with_variant RelatedLinksABTest3: 'C' do
content_item = content_store_has_schema_example('guide', 'guide')

get :show, params: { path: path_for(content_item) }
Expand Down
2 changes: 1 addition & 1 deletion test/controllers/development_controller_test.rb
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
class DevelopmentControllerTest < ActionController::TestCase
include GovukAbTesting::MinitestHelpers

%w(A B).each do |test_variant|
%w(A B C).each do |test_variant|
test "RelatedLinksABTest3 works correctly for each variant (variant: #{test_variant})" do
with_variant RelatedLinksABTest3: test_variant do
get :index
Expand Down

0 comments on commit ea2819b

Please sign in to comment.