feat!: update experimentalists grid_pool, random_pool, random_sample to use the new State mechanism

[hollandjg]

Updates:

• Breaking Change, Refactor: remove structural pooler vs. sampler split in autora.experimentalist
• Breaking Change, Feature: Update standard way of setting up experimentalists so that every function should be wrapped by the end-user, reducing load on contributors and hopefully making reuse easier
• Breaking Change, Refactor: Gather standard State and wrapper functions in a single submodule to allow from autora.state.standard import StandardState, Delta, on_state, state_fn_from_estimator
• Breaking Change, Feature: convert existing example experimentalists grid_pool, random_pool, and random_sample to use the State + Delta mechanism with the standard field names from StandardState
• CI: Update Pre-Commit Hooks

Resolutions:

• resolves grid pooler and random pooler expect different arguments #9
• requires feat: add StandardState #32
• resolves refactor: ensure experimentalists have a consistent interface autora#231
• resolves clarify in documentation the inputs and outputs of theorists/experimentalists/cycle autora#462

See also:

• feat: add support for new State mechanism (suggestion) autora-experimentalist-model-disagreement#10

Discussion – moving poolers/samplers to top-level of experimentalist

There are several reasons for moving the poolers and samplers into files at the top level of experimentalist:

poolers and samplers are now less relevant

The strict difference between poolers and samplers now has less meaning

Conceptually: Now we just have functions which sit in a pipeline or cycle and operate on the state. Some of them still act like poolers and samplers of old, but they look more similar.

Technically: In the old setup, poolers took no positional arguments by design, whereas samplers took one (conditions). Now all of the poolers and samplers might take any number of arguments, like the experiment_data, model, conditions from a step before, and we achieve that using only named arguments in the functions which act on State.

It makes sense to gather related functions in one file

Some of the poolers and samplers are obviously related – like the random pooler/sampler, falsification pooler/sampler.

It is simpler to have these in the same file, so that you can share functions more easily between them.

This will also remove one level of complexity for contributors – they'll just be able to make an experimentalist with full freedom to use whatever they want.

This will also remove one chunk of complexity from the cookiecutter template.

Allows for backward compatibility whilst freeing up the good names we want

There are two things we really want to achieve here:

• no breaking changes in this release
• free up names like grid_pool, random_sample, random_pool for our new paradigm

The proposed method was to make the existing functions into singledispatch functions which had different behavior depending on what the inputs were. However, this relies on there always being at least one positional argument to the function, an assumption which is broken for the old poolers which had no positional arguments by design.

An attempt was made to remedy this, but it was super complicated and didn't work because of the positional arguments thing.

Both in this case, and in future cases where we'll want to update all of the existing contributions to use the new setup, it's just much easier to start over in a new file than to try to make it work in the same file.

[hollandjg]

Hi all, esp @musslick @younesStrittmatter @benwandrew @blinodelka @chadcwilliams @benwandrew
I've updated this PR to include the core changes from the mini-hackathon. I think this might be ready to merge, so please let me know if there's something broken.

If anything is missing, I suggest we make that a new PR, because this one is pretty large and already does a lot of stuff.

• I didn't include the align_dataframe_to_ivs function from https://github.com/AutoResearch/autora-core/blob/mini-hackaton-before-refactor-state/src/autora/utils/conversion.py because I don't think this should be part of the on_state function.
  • The functions in autora.state.delta are by design completely agnostic about the contents of the state objects, other than assuming that they can be treated as State objects and are fundamentally dataclasses. That means that they don't do any transformations on the data beyond extending DataFrames and renaming things according to the alias rules.
  • I think it's best to include the align_dataframe_to_ivs in the places where you need it in your experimentalist, for instance, or to define a new function wrapper which you can use if you need it.
  • Remember that if you're using data frames you can just pass a list of column names and you've already got exactly those columns in that order, and I think that's probably more understandable to the end-user.

[musslick]

Hi all, esp @musslick @younesStrittmatter @benwandrew @blinodelka @chadcwilliams @benwandrew I've updated this PR to include the core changes from the mini-hackathon. I think this might be ready to merge, so please let me know if there's something broken.

If anything is missing, I suggest we make that a new PR, because this one is pretty large and already does a lot of stuff.

• I didn't include the align_dataframe_to_ivs function from https://github.com/AutoResearch/autora-core/blob/mini-hackaton-before-refactor-state/src/autora/utils/conversion.py because I don't think this should be part of the on_state function.

  • The functions in autora.state.delta are by design completely agnostic about the contents of the state objects, other than assuming that they can be treated as State objects and are fundamentally dataclasses. That means that they don't do any transformations on the data beyond extending DataFrames and renaming things according to the alias rules.
  • I think it's best to include the align_dataframe_to_ivs in the places where you need it in your experimentalist, for instance, or to define a new function wrapper which you can use if you need it.
  • Remember that if you're using data frames you can just pass a list of column names and you've already got exactly those columns in that order, and I think that's probably more understandable to the end-user.

So far, we have been using it in both experimentalists and synthetic models (autora). Any other place where we can put it in the core?

[hollandjg]

So far, we have been using it in both experimentalists and synthetic models (autora). Any other place where we can put it in the core?

I can't think of any.

[hollandjg]

Hi all, esp. @musslick @benwandrew and @younesStrittmatter – I'd appreciate your re-review of this PR! Please let me know if there's anything else needing changing.

[musslick]

Looks great but docstrings need to be fixed

[musslick]

Looks great!

Covered in today's meeting. Thanks, Ben!