Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Script for complementary experiment on functions with few successes #2208

Closed
nikohansen opened this issue Jun 24, 2023 · 2 comments
Closed

Comments

@nikohansen
Copy link
Contributor

nikohansen commented Jun 24, 2023

Following up on #1117 and #1978 to account for functions with few or zero successes (see also #2117).

A better approach may be to check the success rates in the previous data and run a complementary repetition subexperiment which only uses those function & dimension where the number of successes is below some threshold (like 7 or 10 or 15).

Caveat: when complementing experiments that used with-trial independent restarts, the budget limit (timeout budget) of the previous experiment remains as an artifact in the data. That is, complementary repetitions should better be used alternatively to independent restarts within a single run (and with the same timeout budget as the original experiment).

Caveat: using all-zeros as first initial solution (for the first trial) now breaks the experimental procedure because not all trials are conducted identically. A possible remedy could be to have all-zeros with some probability? Having all-zeros always in the first trial/instance improves reproducibility (however the reproducibility of a possible artifact).

nikohansen added a commit that referenced this issue Nov 25, 2023
which is meant to be used in an example experiment.
This partly addresses issues #2117 and #2208.
@nikohansen
Copy link
Contributor Author

nikohansen commented Jan 15, 2024

When running several batches adding trials based on the success number, all trials for a single function+dimension must be accessible to the batch where this function+dimension is run and all instances must be run within one batch. Otherwise, we cannot guaranty to have the same number of repetitions for each instance. Unfortunately, this makes a uniform time distribution between batches much more difficult (however the data organization less confusing).

@nikohansen
Copy link
Contributor Author

For the time being we don't complement previous data, but instead we add trials depending the observed successes in the current experiment until the budget is exhausted. As we can combine data from different experiments, ignoring previous data for the choice should not be a very relevant limitation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant