Experiments Running and Management

[melo-gonzo]

This PR adds functionality to run and manage experiments in matsciml. Currently, the options for experimentation are to create a script for each experiment run and manually update parameters, or to use the pytorch lightning cli + yaml files. The first approach is time-consuming and clumsy, and the second approach does not handle multi task and multi data experiments in matsciml. This PR aims to bridge the gap between functional experimental pipelines without the headache of managing a ton of scripts. It takes inspiration from the lightning cli by managing the different aspects of an experiment with yaml files {trainer, model, dataset, experiment}.yaml and allows for cli updates by specifying a chain of parameters to traverse and update. Experimentation effectively comes down to managing these yaml files and greatly reduces the barrier to entry towards running complex experiments with multi task and multi data origins. See the README for more details.

[melo-gonzo]

Note that I updated matsciml/models/dgl/gaanet/tests/test_gala.py and matsciml/models/base.py to match changes from #228.

[laserkelvin]

I've put in some minor comments, but the two high level things that I'll bring up for discussion are:

1. Maybe we can finally use the registry, so instead of maintaining individual maps in task, dataset, and models, just rely on matsciml.common.registry to do the mapping?
2. I'm not sure if I missed it, but it might be good to refactor your object creation functions (i.e. mapping args into something you pull out of a mapping) into matsciml.common.registry. The "improvement" so to speak would be to also include arg/kwarg validation, similar to this and I think also in MACEWrapper. I think you want to raise an exception if someone passes something that isn't recognized, and this would be a good place to put it (even as a staticmethod of Registry).

[laserkelvin]

Do you want to move this things into the main datasets module?

You could repackage these functions as a from_config class method, then you don't need to include __init__.py in this folder. It could mess up pip installs

[melo-gonzo]

I think I'd prefer to keep this separate from the datasets module since it is also relying on other experiment related utils such as the command line parsing. If there ends up being other use cases where this functionality would be helpful from the dataset module we can do a refactor then.

[melo-gonzo]

bb140fb

I added the class argument verification check in this commit. I'm not sure it makes sense to refactor the object creation function instantiate_arg_dict and the argument checker verify_class_args into the Registry class, because these functions also are used to spin up lightning classes such as callbacks and loggers, and other random functions that may not be a part of the registered objects, such as matsciml.datasets.utils.element_types`, and lots of the MACE moules.

[laserkelvin]

I'll make this optional for you, but just looks cleaner if you use pathlib instead:

from pathlib import Path
yaml_dir = Path(__file__)

for filename in yaml_dir.rglob("*.yaml"):
    ...

[melo-gonzo]

fixed 759a4fc

[laserkelvin]

Same comment as for datasets, use pathlib instead of os.path

[melo-gonzo]

fixed 759a4fc

[laserkelvin]

Same comment for pathlib

[melo-gonzo]

fixed 759a4fc

[laserkelvin]

Thanks for making all the changes, LGTM!