Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Variable names and weights as part of static dataset files #3

Closed
joeloskarsson opened this issue Oct 23, 2023 · 1 comment
Closed
Labels
enhancement New feature or request

Comments

@joeloskarsson
Copy link
Collaborator

Currently the variables in the dataset are listed in constants.py. This is bad if the code is to be used with other datasets.

Proposition

Create a file variables.json in data/my_dataset/static that describe all variables. This includes:

  • Weather state variables (e.g. u_65)
  • Forcing variables for the full grid
  • Batch-static forcing variables (static during one forecast, but changing throughout the dataset. i.e. open water currently)

All of these should be listed in order with names. For the weather state variables, their weighting (as in parameter_weights.npy currently) should also be listed with them. We can then remove the lines https://github.com/joeloskarsson/neural-lam/blob/89a4c63370201c9ea1a5f04d4cf1e5e75b7cc83e/create_parameter_weights.py#L26-L31 that generate this weighting file. It is better to let this be something that is set manually when preparing a dataset.

Such a variables.json file could then be loaded into a VariableDescription object and used in the models. The variable dimensions https://github.com/joeloskarsson/neural-lam/blob/89a4c63370201c9ea1a5f04d4cf1e5e75b7cc83e/neural_lam/models/ar_model.py#L22-L24 should then be read from this object rather than hard-coded in a model definition.

@joeloskarsson joeloskarsson added the enhancement New feature or request label Oct 23, 2023
@joeloskarsson
Copy link
Collaborator Author

Superseded by #23

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant