feat: add StandardState

[hollandjg]

Description

Add a new StandardState object which can be used as the default for future experimentalists, experiment runners and theorists.

• resolves feat: provide standard State with variable collection, params, conditions, experimental_data and models #24
• resolves feat: add support for np.arrays and np.recarrays in functional approach #22

Type of change

• fix: A bug fix
• feat: A new feature

Features

Add a new StandardState object with

• variables
• experiment_data
• conditions
• models
• alias model which returns the last model.

Make some fixes and refactors in the autora.state.delta module.

• Don't throw an error if a delta field is missing on the state, supersedes fix: remove error when a delta-field is missing on the state #30
• Allow State to coerce datatypes to the correct type with the metadata["converter"] parameter, supersedes feat: add ability to coerce datatypes in the state object on a Delta update #27
• Allow State to declare aliases which convert a particular Delta parameter into a format understood by the State, supersedes feat: add "aliases" to State fields #31

Questions (Optional)

• Are the field names understandable?
• Do they match the agreed names?
• Are there better names we could use?

Details

Aliases

Aliases work like this:

s = SomeState(models=[FittedModel0(), FittedModel1()]) # `models` is the list of models
t = s + Delta(model=FittedModel2())  # !! `model` is singular and not in a list!

...and get the following back as t:

SomeState(models=[FittedModel0(), FittedModel1(), FittedModel2()])

This is required for our Theorist interface which may by default return a single model:

def a_theorist(experiment_data):
    X, y = experiment_data[x_names], experiment_data[y_names]
    model = ModelFitter().fit(X, y)
    return Result(model=model)

Without this feature, this couldn't be handled without requiring each theorist to always return a list of models, which is a pain and feels wrong: return Result(models=[model])

[younesStrittmatter]

Looks great!

[benwandrew]

this is good, cool stuff! my only hesitation so far regarding names is models vs model (and more generally, things (full list) vs thing (last element); can we somehow more clearly differentiate the last model and the List?

[benwandrew]

i wonder if want a clearer distinction between the models and model properties? could get a little confusing for users since they're so close in name... although, admittedly hard to think of an alternative that keeps single words

[musslick]

Hmm... how about model vs. model_list or model_set? Are those better than models?

[benwandrew]

@musslick i'm fine with either of those! i'm also fine with models and model if others think it's not actually going to be an issue; just trying to maximally clear while still being Pythonic :)

[hollandjg]

On reflection, I feel like I still prefer models rather than anything longer – it feels a bit more natural to me. Combined with the type annotations (like BaseEstimator vs List[BaseEstimator]) and type checking, I feel like the risk of accidental confusion is sufficiently small to be unproblematic. I'm open to persuasion there though.

[benwandrew]

sounds good, i'm happy with models! it was only a slight hesitation on my part, and i think it is indeed quite natural.

[benwandrew]

a few more comments:

1. initially i was confused by some of the output displayed in Creating Generators With State Based Functions section of the notebook. specifically,

seemed like the outputs were reversed with respect to what was being described. i think the issue is that the output is correct the first pass through running the cells (as i verified), but then if you got back up and run a cell above again it will keep cycling but the descriptive text remains the same. no substantive error, just threw me off given what was displayed when you first open and read through the notebook.

2. in the subsequent section — Adding The Experimentalist — i encounter the following when running the first cell:

note, everything up to that point executed successfully.

[musslick]

Looks great, just had a question regarding numpy<>pandas conversion.

[musslick]

Quick question: Do we also allow for casting from pandas to numpy? I think we should allow that since most people will be feeding the theorists with numpy array (it's just simpler).

[hollandjg]

Yes! Casting from pandas to numpy is totally possible – using the np.asarray converter like on line 170, if the Delta includes a DataFrame, it will be converted to a numpy array.

You can always put a pd.DataFrame through np.asarray to get a numpy array.

My recommendation would be to have the functions and classes which want to represent the data internally as numpy arrays accept np.typing.ArrayLike as the input type, which allows for lots of input types – lists of values, or DataFrames or np.ndarrays – and then do a np.asarray() call at the top of the function to turn the input into the array you want.

It's really hard to do the casting properly from outside these functions, as we'd have to inspect the function signatures and work out whether the type we have is compatible. It's much simpler and more efficient to do this within the function itself, especially if we have a general interchange format like pd.DataFrame as our standard.

[musslick]

Hmm... how about model vs. model_list or model_set? Are those better than models?

[hollandjg]

seemed like the outputs were reversed with respect to what was being described

They were. Thanks for catching this! It should be in order now. Using generators is tricky, perhaps it's not something we should really recommend.

[hollandjg]

2. in the subsequent section — Adding The Experimentalist — i encounter the following when running the first cell:

This is because the new experimentalist isn't yet defined – that comes in #33 . I've taken out that block from this PR but reintroduce it (in a fixed form) in #32.

[benwandrew]

2. in the subsequent section — Adding The Experimentalist — i encounter the following when running the first cell:

This is because the new experimentalist isn't yet defined – that comes in #33 . I've taken out that block from this PR but reintroduce it (in a fixed form) in #32.

ok, that makes sense.

[benwandrew]

per the end of our conversation in group today, and with all previous requested changes/questions addressed, i'm happy to get this merged!

[benwandrew]

sounds good, i'm happy with models! it was only a slight hesitation on my part, and i think it is indeed quite natural.