Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: [automl/epsilon_decay] integrate robust confidence sequences #4377

Merged
merged 28 commits into from
Dec 27, 2022
Merged

refactor: [automl/epsilon_decay] integrate robust confidence sequences #4377

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
076f99c
refactor: [automl/epsilon_decay] integrate robust confidence sequences
bassmang Dec 20, 2022
d6abd37
clang
bassmang Dec 20, 2022
97a05b7
Merge remote-tracking branch 'upstream/master' into integ_csr
bassmang Dec 20, 2022
7a93756
merge
bassmang Dec 21, 2022
e98b868
clang
bassmang Dec 21, 2022
edc36ff
Merge branch 'master' into integ_csr
bassmang Dec 21, 2022
123c62d
new flag
bassmang Dec 22, 2022
0f44f84
clang
bassmang Dec 22, 2022
a6db10e
reorder
bassmang Dec 22, 2022
d598f5e
more printing
bassmang Dec 22, 2022
4845fd0
clang
bassmang Dec 22, 2022
c41ecaf
audit fix
bassmang Dec 22, 2022
3d2c125
merge
bassmang Dec 22, 2022
cc15675
more audit
bassmang Dec 22, 2022
4593b95
clang
bassmang Dec 22, 2022
b00c106
Merge branch 'master' into integ_csr
bassmang Dec 22, 2022
fbba83a
tests
bassmang Dec 22, 2022
063a1dd
Merge branch 'master' into integ_csr
jackgerrits Dec 23, 2022
fdb8e87
move to gtest
bassmang Dec 23, 2022
8a09608
clang
bassmang Dec 23, 2022
a1da91d
clean
bassmang Dec 23, 2022
b83ec10
comments
bassmang Dec 27, 2022
025c40e
clang
bassmang Dec 27, 2022
5535a4f
change seed
bassmang Dec 27, 2022
90806da
capture
bassmang Dec 27, 2022
ca964e1
assert
bassmang Dec 27, 2022
739f29b
capture
bassmang Dec 27, 2022
c20e0f0
capture
bassmang Dec 27, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
103,162 changes: 51,581 additions & 51,581 deletions test/pred-sets/ref/aml_spinoff.inv
View file
Open in desktop

Large diffs are not rendered by default.

36 changes: 18 additions & 18 deletions test/train-sets/automl_spin_off.txt
View file
Open in desktop

Large diffs are not rendered by default.

244 changes: 72 additions & 172 deletions test/train-sets/ref/automl_readable.txt
View file
Open in desktop

Large diffs are not rendered by default.

244 changes: 72 additions & 172 deletions test/train-sets/ref/automl_readable_cubic.txt
View file
Open in desktop

Large diffs are not rendered by default.

12 changes: 6 additions & 6 deletions test/train-sets/ref/cb_similar_aml_spinoff_save.stderr
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,15 +13,15 @@ Input label = CB
Output pred = ACTION_PROBS
average since example example current current current
loss last counter weight label predict features
0.333333 0.333333 1 1.0 0:1:1 0:0.33 8436
0.175000 0.016667 2 2.0 1:1:1 2:0.97 18319
0.333333 0.491667 4 4.0 1:1:1 0:0.97 23278
0.293750 0.254167 8 8.0 0:1:1 2:0.97 13862
0.273958 0.254167 16 16.0 0:1:1 2:0.97 2249
-0.33333 -0.33333 1 1.0 0:-1:1 0:0.33 8436
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why would loss change on the first example if the CS is not used for loss?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had to change the rewards because the new estimator can only use rewards between 0 and 1, implying a costs between -1 and 0

-0.17500 -0.01666 2 2.0 1:-1:1 0:0.97 18319
-0.09583 -0.01666 4 4.0 1:-1:1 2:0.97 23278
-0.17500 -0.25416 8 8.0 0:-1:1 1:0.97 13862
-0.27395 -0.37291 16 16.0 0:-1:1 1:0.97 2249

finished run
number of examples = 18
weighted example sum = 18.000000
weighted label sum = 0.000000
average loss = 0.298148
average loss = -0.245370
total feature number = 293853
244 changes: 72 additions & 172 deletions test/train-sets/ref/cbadf_automl_readable.txt
View file
Open in desktop

Large diffs are not rendered by default.

12 changes: 6 additions & 6 deletions test/train-sets/ref/spin_off_aml_load.stderr
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,15 +12,15 @@ Input label = CB
Output pred = ACTION_PROBS
average since example example current current current
loss last counter weight label predict features
0.016667 0.016667 1 1.0 0:1:1 2:0.97 8436
0.016667 0.016667 2 2.0 1:1:1 2:0.97 18319
0.016667 0.016667 4 4.0 1:1:1 0:0.97 23278
0.016667 0.016667 8 8.0 0:1:1 2:0.97 13862
0.016667 0.016667 16 16.0 0:1:1 2:0.97 2249
-0.96666 -0.96666 1 1.0 0:-1:1 0:0.97 8436
-0.96666 -0.96666 2 2.0 1:-1:1 1:0.97 18319
-0.49166 -0.01666 4 4.0 1:-1:1 2:0.97 23278
-0.37291 -0.25416 8 8.0 0:-1:1 1:0.97 13862
-0.43229 -0.49166 16 16.0 0:-1:1 0:0.97 2249

finished run
number of examples = 18
weighted example sum = 18.000000
weighted label sum = 0.000000
average loss = 0.069444
average loss = -0.438889
total feature number = 293853
12 changes: 6 additions & 6 deletions test/train-sets/ref/spin_off_aml_save.stderr
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,15 +12,15 @@ Input label = CB
Output pred = ACTION_PROBS
average since example example current current current
loss last counter weight label predict features
0.333333 0.333333 1 1.0 0:1:1 0:0.33 8436
0.175000 0.016667 2 2.0 1:1:1 2:0.97 18319
0.333333 0.491667 4 4.0 1:1:1 0:0.97 23278
0.293750 0.254167 8 8.0 0:1:1 2:0.97 13862
0.273958 0.254167 16 16.0 0:1:1 2:0.97 2249
-0.33333 -0.33333 1 1.0 0:-1:1 0:0.33 8436
-0.17500 -0.01666 2 2.0 1:-1:1 0:0.97 18319
-0.09583 -0.01666 4 4.0 1:-1:1 2:0.97 23278
-0.17500 -0.25416 8 8.0 0:-1:1 1:0.97 13862
-0.27395 -0.37291 16 16.0 0:-1:1 1:0.97 2249

finished run
number of examples = 18
weighted example sum = 18.000000
weighted label sum = 0.000000
average loss = 0.298148
average loss = -0.245370
total feature number = 293853
85 changes: 45 additions & 40 deletions test/unit_test/automl_test.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
#include "reductions/details/automl_impl.h"
#include "simulator.h"
#include "test_common.h"
#include "vw/core/confidence_sequence.h"
#include "vw/core/confidence_sequence_robust.h"
#include "vw/core/metric_sink.h"
#include "vw/core/vw_fwd.h"

Expand Down Expand Up @@ -36,8 +36,9 @@ std::ostream& operator<<(std::ostream& o, std::vector<T> const& vec)
namespace aml_test
{
template <typename T>
void check_interactions_match_exclusions(VW::reductions::automl::automl<
interaction_config_manager<VW::reductions::automl::config_oracle<T>, VW::estimators::confidence_sequence>>* aml)
void check_interactions_match_exclusions(
VW::reductions::automl::automl<interaction_config_manager<VW::reductions::automl::config_oracle<T>,
VW::estimators::confidence_sequence_robust>>* aml)
{
for (const auto& estimator : aml->cm->estimators)
{
Expand Down Expand Up @@ -72,8 +73,9 @@ void check_interactions_match_exclusions(VW::reductions::automl::automl<
}

template <typename T>
void check_config_states(VW::reductions::automl::automl<
interaction_config_manager<VW::reductions::automl::config_oracle<T>, VW::estimators::confidence_sequence>>* aml)
void check_config_states(
VW::reductions::automl::automl<interaction_config_manager<VW::reductions::automl::config_oracle<T>,
VW::estimators::confidence_sequence_robust>>* aml)
{
// No configs in the index queue should be live
auto index_queue = aml->cm->_config_oracle.index_queue;
Expand All @@ -94,7 +96,7 @@ void check_config_states(VW::reductions::automl::automl<

template <typename T>
VW::reductions::automl::automl<
interaction_config_manager<VW::reductions::automl::config_oracle<T>, VW::estimators::confidence_sequence>>*
interaction_config_manager<VW::reductions::automl::config_oracle<T>, VW::estimators::confidence_sequence_robust>>*
get_automl_data(VW::workspace& all)
{
std::vector<std::string> e_r;
Expand All @@ -106,34 +108,36 @@ get_automl_data(VW::workspace& all)

VW::LEARNER::multi_learner* automl_learner = as_multiline(all.l->get_learner_by_name_prefix("automl"));

return (VW::reductions::automl::automl<
interaction_config_manager<VW::reductions::automl::config_oracle<T>, VW::estimators::confidence_sequence>>*)
return (VW::reductions::automl::automl<interaction_config_manager<VW::reductions::automl::config_oracle<T>,
VW::estimators::confidence_sequence_robust>>*)
automl_learner->get_internal_type_erased_data_pointer_test_use_only();
}
template VW::reductions::automl::automl<
interaction_config_manager<VW::reductions::automl::config_oracle<VW::reductions::automl::oracle_rand_impl>,
VW::estimators::confidence_sequence>>*
VW::estimators::confidence_sequence_robust>>*
get_automl_data(VW::workspace& all);
template VW::reductions::automl::automl<interaction_config_manager<
VW::reductions::automl::config_oracle<VW::reductions::automl::one_diff_impl>, VW::estimators::confidence_sequence>>*
template VW::reductions::automl::automl<
interaction_config_manager<VW::reductions::automl::config_oracle<VW::reductions::automl::one_diff_impl>,
VW::estimators::confidence_sequence_robust>>*
get_automl_data(VW::workspace& all);
template VW::reductions::automl::automl<
interaction_config_manager<VW::reductions::automl::config_oracle<VW::reductions::automl::champdupe_impl>,
VW::estimators::confidence_sequence>>*
VW::estimators::confidence_sequence_robust>>*
get_automl_data(VW::workspace& all);
template VW::reductions::automl::automl<
interaction_config_manager<VW::reductions::automl::config_oracle<VW::reductions::automl::one_diff_inclusion_impl>,
VW::estimators::confidence_sequence>>*
VW::estimators::confidence_sequence_robust>>*
get_automl_data(VW::workspace& all);

using aml_rand = VW::reductions::automl::automl<
interaction_config_manager<VW::reductions::automl::config_oracle<VW::reductions::automl::oracle_rand_impl>,
VW::estimators::confidence_sequence>>;
using aml_onediff = VW::reductions::automl::automl<interaction_config_manager<
VW::reductions::automl::config_oracle<VW::reductions::automl::one_diff_impl>, VW::estimators::confidence_sequence>>;
VW::estimators::confidence_sequence_robust>>;
using aml_onediff = VW::reductions::automl::automl<
interaction_config_manager<VW::reductions::automl::config_oracle<VW::reductions::automl::one_diff_impl>,
VW::estimators::confidence_sequence_robust>>;
using aml_onediff_inclusion = VW::reductions::automl::automl<
interaction_config_manager<VW::reductions::automl::config_oracle<VW::reductions::automl::one_diff_inclusion_impl>,
VW::estimators::confidence_sequence>>;
VW::estimators::confidence_sequence_robust>>;
} // namespace aml_test

// Need to add save_load functionality to multiple structs in automl reduction including
Expand Down Expand Up @@ -358,9 +362,9 @@ BOOST_AUTO_TEST_CASE(automl_clear_configs_w_iterations)
BOOST_CHECK_EQUAL(aml->cm->current_champ, 0);
BOOST_CHECK_EQUAL(aml->cm->_config_oracle.valid_config_size, 4);
BOOST_CHECK_EQUAL(clear_champ_switch - 1, aml->cm->total_learn_count);
BOOST_CHECK_EQUAL(aml->cm->estimators[0].first.live_interactions.size(), 3);
BOOST_CHECK_EQUAL(aml->cm->estimators[1].first.live_interactions.size(), 2);
BOOST_CHECK_EQUAL(aml->cm->estimators[2].first.live_interactions.size(), 2);
BOOST_CHECK_EQUAL(aml->cm->estimators[0].first.live_interactions.size(), 2);
BOOST_CHECK_EQUAL(aml->cm->estimators[1].first.live_interactions.size(), 3);
BOOST_CHECK_EQUAL(aml->cm->estimators[2].first.live_interactions.size(), 1);
BOOST_CHECK(aml->current_state == VW::reductions::automl::automl_state::Experimenting);
return true;
});
Expand Down Expand Up @@ -431,9 +435,9 @@ BOOST_AUTO_TEST_CASE(automl_clear_configs_one_diff_w_iterations)
{
aml_test::aml_onediff* aml = aml_test::get_automl_data<VW::reductions::automl::one_diff_impl>(all);
BOOST_CHECK_EQUAL(aml->cm->estimators.size(), 3);
BOOST_CHECK_EQUAL(aml->cm->estimators[0].first.live_interactions.size(), 3);
BOOST_CHECK_EQUAL(aml->cm->estimators[1].first.live_interactions.size(), 2);
BOOST_CHECK_EQUAL(aml->cm->estimators[2].first.live_interactions.size(), 2);
BOOST_CHECK_EQUAL(aml->cm->estimators[0].first.live_interactions.size(), 2);
BOOST_CHECK_EQUAL(aml->cm->estimators[1].first.live_interactions.size(), 3);
BOOST_CHECK_EQUAL(aml->cm->estimators[2].first.live_interactions.size(), 1);
return true;
});

Expand Down Expand Up @@ -479,7 +483,8 @@ BOOST_AUTO_TEST_CASE(one_diff_impl_unittest_w_iterations)
auto rand_state = all.get_random_state();

std::map<VW::namespace_index, uint64_t> ns_counter;
std::vector<std::pair<aml_estimator<VW::estimators::confidence_sequence>, VW::estimators::confidence_sequence>>
std::vector<std::pair<aml_estimator<VW::estimators::confidence_sequence_robust>,
VW::estimators::confidence_sequence_robust>>
estimators;

config_oracle<one_diff_impl> oracle(aml->cm->default_lease, co.calc_priority, co._interaction_type,
Expand All @@ -495,7 +500,7 @@ BOOST_AUTO_TEST_CASE(one_diff_impl_unittest_w_iterations)
BOOST_CHECK_EQUAL(estimators.size(), 0);
BOOST_CHECK_EQUAL(prio_queue.size(), 0);
interaction_config_manager<config_oracle<one_diff_impl>,
VW::estimators::confidence_sequence>::insert_starting_configuration(estimators, oracle,
VW::estimators::confidence_sequence_robust>::insert_starting_configuration(estimators, oracle,
aml->cm->automl_significance_level);
BOOST_CHECK_EQUAL(configs.size(), 1);
BOOST_CHECK_EQUAL(estimators.size(), 1);
Expand Down Expand Up @@ -540,7 +545,7 @@ BOOST_AUTO_TEST_CASE(one_diff_impl_unittest_w_iterations)
for (size_t i = 1; i < configs.size(); ++i)
{
interaction_config_manager<config_oracle<one_diff_impl>,
VW::estimators::confidence_sequence>::apply_config_at_slot(estimators, oracle.configs, i,
VW::estimators::confidence_sequence_robust>::apply_config_at_slot(estimators, oracle.configs, i,
config_oracle<one_diff_impl>::choose(oracle.index_queue), aml->cm->automl_significance_level, 1);
auto& temp_exclusions = oracle.configs[estimators[i].first.config_index];
auto& temp_interactions = estimators[i].first.live_interactions;
Expand All @@ -551,8 +556,8 @@ BOOST_AUTO_TEST_CASE(one_diff_impl_unittest_w_iterations)
BOOST_CHECK_EQUAL(estimators.size(), 4);

// excl_2 is now champ
interaction_config_manager<config_oracle<one_diff_impl>, VW::estimators::confidence_sequence>::apply_new_champ(
oracle, 2, estimators, 0, ns_counter);
interaction_config_manager<config_oracle<one_diff_impl>,
VW::estimators::confidence_sequence_robust>::apply_new_champ(oracle, 2, estimators, 0, ns_counter);

BOOST_CHECK_EQUAL_COLLECTIONS(
configs[0].elements.begin(), configs[0].elements.end(), excl_2.begin(), excl_2.end());
Expand All @@ -576,7 +581,7 @@ BOOST_AUTO_TEST_CASE(one_diff_impl_unittest_w_iterations)
for (size_t i = 2; i < 4; ++i)
{
interaction_config_manager<config_oracle<one_diff_impl>,
VW::estimators::confidence_sequence>::apply_config_at_slot(estimators, oracle.configs, i,
VW::estimators::confidence_sequence_robust>::apply_config_at_slot(estimators, oracle.configs, i,
config_oracle<one_diff_impl>::choose(oracle.index_queue), aml->cm->automl_significance_level, 1);
auto& temp_config = oracle.configs[estimators[i].first.config_index];
auto& temp_interactions = estimators[i].first.live_interactions;
Expand All @@ -586,8 +591,8 @@ BOOST_AUTO_TEST_CASE(one_diff_impl_unittest_w_iterations)
BOOST_CHECK_EQUAL(prio_queue.size(), 0);

// excl_4 is now champ
interaction_config_manager<config_oracle<one_diff_impl>, VW::estimators::confidence_sequence>::apply_new_champ(
oracle, 3, estimators, 0, ns_counter);
interaction_config_manager<config_oracle<one_diff_impl>,
VW::estimators::confidence_sequence_robust>::apply_new_champ(oracle, 3, estimators, 0, ns_counter);

BOOST_CHECK_EQUAL_COLLECTIONS(
configs[0].elements.begin(), configs[0].elements.end(), excl_4.begin(), excl_4.end());
Expand Down Expand Up @@ -650,15 +655,15 @@ BOOST_AUTO_TEST_CASE(automl_insertion_champ_change_w_iterations)
BOOST_CHECK_EQUAL(aml->cm->current_champ, 0);
BOOST_CHECK_EQUAL(aml->cm->_config_oracle.valid_config_size, 4);
BOOST_CHECK_EQUAL(clear_champ_switch - 1, aml->cm->total_learn_count);
BOOST_CHECK_EQUAL(aml->cm->estimators[0].first.live_interactions.size(), 0);
BOOST_CHECK_EQUAL(aml->cm->estimators[1].first.live_interactions.size(), 1);
BOOST_CHECK_EQUAL(aml->cm->estimators[2].first.live_interactions.size(), 1);
BOOST_CHECK_EQUAL(aml->cm->estimators[0].first.live_interactions.size(), 1);
BOOST_CHECK_EQUAL(aml->cm->estimators[1].first.live_interactions.size(), 0);
BOOST_CHECK_EQUAL(aml->cm->estimators[2].first.live_interactions.size(), 2);
BOOST_CHECK_EQUAL(aml->cm->_config_oracle.configs.size(), 4);
BOOST_CHECK_EQUAL(aml->cm->_config_oracle.configs[0].elements.size(), 0);
BOOST_CHECK_EQUAL(aml->cm->_config_oracle.configs[1].elements.size(), 1);
BOOST_CHECK_EQUAL(aml->cm->_config_oracle.configs[2].elements.size(), 1);
BOOST_CHECK_EQUAL(aml->cm->_config_oracle.configs[3].elements.size(), 1);
BOOST_CHECK_EQUAL(aml->cm->total_champ_switches, 0);
BOOST_CHECK_EQUAL(aml->cm->_config_oracle.configs[0].elements.size(), 1);
BOOST_CHECK_EQUAL(aml->cm->_config_oracle.configs[1].elements.size(), 0);
BOOST_CHECK_EQUAL(aml->cm->_config_oracle.configs[2].elements.size(), 2);
BOOST_CHECK_EQUAL(aml->cm->_config_oracle.configs[3].elements.size(), 2);
BOOST_CHECK_EQUAL(aml->cm->total_champ_switches, 1);
BOOST_CHECK(aml->current_state == VW::reductions::automl::automl_state::Experimenting);
return true;
});
Expand All @@ -670,7 +675,7 @@ BOOST_AUTO_TEST_CASE(automl_insertion_champ_change_w_iterations)
aml_test::check_config_states(aml);
BOOST_CHECK_EQUAL(aml->cm->current_champ, 0);
BOOST_CHECK_EQUAL(clear_champ_switch, aml->cm->total_learn_count);
BOOST_CHECK_EQUAL(aml->cm->estimators.size(), 2);
BOOST_CHECK_EQUAL(aml->cm->estimators.size(), 3);
BOOST_CHECK_EQUAL(aml->cm->_config_oracle.valid_config_size, 4);
BOOST_CHECK_EQUAL(aml->cm->estimators[0].first.live_interactions.size(), 1);
BOOST_CHECK_EQUAL(aml->cm->estimators[1].first.live_interactions.size(), 0);
Expand Down
Loading