Dataloader

[kks32]

Describe the PR
Supports both npz and hdf5 data format

Related Issues/PRs
#82

Additional Context
Will remove data_loader.py once we have merged multinode training to train.py

[yjchoi1]

Running it on single node shows the following error:

Error executing job with overrides: []
Traceback (most recent call last):
  File "/work2/08264/baagee/frontera/gns-main/gns/train.py", line 817, in main
    verbose, world_size = distribute.setup(local_rank)
UnboundLocalError: local variable 'local_rank' referenced before assignment

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

[yjchoi1]

Now, it works with my custom data on a single node and multi-nodes. I haven't checked with .h5 though.

[yjchoi1]

In the resume mode, simulator.module.load() seems to should be changed to simulator.load() although the previous version of parallel GNS used to work with simulator.module.load() in the distributed setting.

Error executing job with overrides: []
Traceback (most recent call last):
  File "/work2/08264/baagee/frontera/gns-main/gns/train.py", line 843, in main
    train(local_rank, cfg, world_size, device, verbose, use_dist)
  File "/work2/08264/baagee/frontera/gns-main/gns/train.py", line 475, in train
    simulator.module.load(cfg.model.path + cfg.model.file)
  File "/work2/08264/baagee/frontera/venvs/venv-frontera-gpu/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1207, in __getattr__
    raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'LearnedSimulator' object has no attribute 'module'

[kks32]

@skye-glitch : Could you address this please.

[skye-glitch]

Please check the updated code.

[yjchoi1]

Also, there is a minor error about tqdm count.

When resume, tqdm does not show the correct nstep. For example, given that the model is saved at 100/500000 step at epoch=0, when we resume from model-100.pt, tqdm count starts from 0/500000 rather than starting from 100/500000, although the code correctly loaded the model at 100 step.

[skye-glitch]

Also, there is a minor error about tqdm count.

When resume, tqdm does not show the correct nstep. For example, given that the model is saved at 100/500000 step at epoch=0, when we resume from model-100.pt, tqdm count starts from 0/500000 rather than starting from 100/500000, although the code correctly loaded the model at 100 step.

Should have been fixed

[yjchoi1]

Another minor issue is that gns/args.py still has n_gpus arg although we don't use them.

[skye-glitch]

Another minor issue is that gns/args.py still has n_gpus arg although we don't use them.

Fixed. Thanks！

[kks32]

@yjchoi1 Could you check to see if everything is good to merge?

[yjchoi1]

After addressing above comment about single node issue, everything seems good to merge.