Hydra configuration

pl_examples/hydra_examples/README.md

@@ -0,0 +1,21 @@
+## Hydra Pytorch Lightning Example
+
+This directory consists of an example of configuring Pytorch Lightning with [Hydra](https://hydra.cc/). Hydra is a tool that allows for the easy configuration of complex applications.  
+The core of this directory consists of a set of structured configs used for pytorch lightining, which are stored under the `from pytorch_lightning.trainer.trainer_conf import PLConfig`. Within the PL config there are 5 cofigurations: 1) Trainer Configuration, 2) Profiler Configuration, 3) Early Stopping Configuration, 4) Logger Configuration and 5) Checkpoint Configuration. All of these are basically mirrors of the arguments that make up these objects. These configuration are used to instantiate the objects using Hydras instantiation utility.
+
+Aside from the PyTorch Lightning configuration we have included a few other important configurations. Optimizer and Scheduler are easy off-the-shelf configurations for configuring your optimizer and learning rate scheduler. You can add them to your config defaults list as needed and use them to configure these objects. Additionally, we provide the arch and data configurations for changing model and data hyperparameters.
+
+All of the above hyperparameters are configured in the config.yaml file which contains the top level configuration for all these configurations. Under this file is a defaults list which highlights for each of these Hydra groups what is the default configuration. Beyond this configuration file, all of the parameters defined can be overriden via the command line.
+
+Additionally, for type safety we highlight in our file `user_config.py` an example of extending the `PLConfig` data class with a user configuration. Hence, we can get the benefits of type safety for our entire config.yaml. Please read through the [basic tutorial](https://hydra.cc/docs/next/tutorials/basic/your_first_app/simple_cli) and [structured configuration tutorial](https://hydra.cc/docs/next/tutorials/structured_config/intro) for more information on using Hydra.
+
+### Tensorboard Visualization
+
+Hydra by default changes the running directory of your program when running into outputs/[DATE]/[TIME]. Hence, all data with a relative path is submitted into this directory. Therefore to visualize all your tensorboard runs one should run the command: `tensorboard --logdir outputs`. This will then allow you to compare your results across runs.  
+You can also [customize](https://hydra.cc/docs/configure_hydra/workdir) your Hydra working directory.
+
+### Multi Run
+
+One nice feature about Hydra in [multi-run](https://hydra.cc/docs/next/tutorials/basic/running_your_app/multi-run/). This can enable you to run your application multiple times with different configurations. A new directory will be created called multirun with the results of these various parameters. You can visualize from tensorboard these results by running: `tensorboard --logdir multirun`.
+
+Other interesting information about Hydra can be found in the [docs](https://hydra.cc/docs/intro/).

pl_examples/hydra_examples/conf/callbacks/basic.yaml

@@ -0,0 +1,6 @@
+# @package _group_
+callbacks:
+  - target: pl_examples.hydra_examples.user_config.MyPrintingCallback
+  - target: pl_examples.hydra_examples.user_config.MessageCallback
+    params:
+      iter_num: 12

pl_examples/hydra_examples/conf/config.yaml

@@ -0,0 +1,11 @@
+defaults:
+  - data: mnist
+  - model: basic
+  - callbacks: basic
+  - profiler: null
+  - logger: null
+  - checkpoint: null
+  - early_stopping: null
+  - trainer: trainer
+  - scheduler: step
+  - opt: adam

pl_examples/hydra_examples/conf/data/fashionmnist.yaml

@@ -0,0 +1,27 @@
+# @package _group_
+
+ds:
+  target: torchvision.datasets.FashionMNIST
+  params:
+    root: ${hydra:runtime.cwd}/datasets
+    download: true
+    train: null
+
+tf:
+  - tensor:
+    target: torchvision.transforms.ToTensor
+  - normal:
+    target: torchvision.transforms.Normalize
+    params:
+      mean: .5
+      std: .2
+
+dl:
+  target: torch.utils.data.DataLoader
+  params:
+    batch_size: 5
+    shuffle: true
+    num_workers: 4
+    pin_memory: False
+    drop_last: False
+    timeout: 0

pl_examples/hydra_examples/conf/data/kmnist.yaml

@@ -0,0 +1,27 @@
+# @package _group_
+
+ds:
+  target: torchvision.datasets.KMNIST
+  params:
+    root: ${hydra:runtime.cwd}/datasets
+    download: true
+    train: null
+
+tf:
+  - tensor:
+    target: torchvision.transforms.ToTensor
+  - normal:
+    target: torchvision.transforms.Normalize
+    params:
+      mean: .5
+      std: .2
+
+dl:
+  target: torch.utils.data.DataLoader
+  params:
+    batch_size: 5
+    shuffle: true
+    num_workers: 4
+    pin_memory: False
+    drop_last: False
+    timeout: 0

pl_examples/hydra_examples/conf/data/mnist.yaml

@@ -0,0 +1,27 @@
+# @package _group_
+
+ds:
+  target: torchvision.datasets.MNIST
+  params:
+    root: ${hydra:runtime.cwd}/datasets
+    download: true
+    train: null
+
+tf:
+  - tensor:
+    target: torchvision.transforms.ToTensor
+  - normal:
+    target: torchvision.transforms.Normalize
+    params:
+      mean: .5
+      std: .2
+
+dl:
+  target: torch.utils.data.DataLoader
+  params:
+    batch_size: 5
+    shuffle: true
+    num_workers: 4
+    pin_memory: False
+    drop_last: False
+    timeout: 0

pl_examples/hydra_examples/conf/model/basic.yaml

@@ -0,0 +1,7 @@
+# @package _group_
+
+drop_prob: 0.2
+in_features: 784
+out_features: 10
+hidden_dim: 1000
+seed: 123

pl_examples/hydra_examples/conf/optimizer.py

@@ -0,0 +1,108 @@
+from dataclasses import dataclass
+from typing import Optional
+
+from hydra.core.config_store import ConfigStore
+from hydra.types import ObjectConf
+
+cs = ConfigStore.instance()
+
+
+@dataclass
+class AdamConf:
+    betas: tuple = (0.9, 0.999)
+    lr: float = 1e-3
+    eps: float = 1e-8
+    weight_decay: float = 0
+    amsgrad: bool = False
+
+
+cs.store(
+    group="opt", name="adam", node=ObjectConf(target="torch.optim.Adam", params=AdamConf()),
+)
+cs.store(
+    group="opt", name="adamw", node=ObjectConf(target="torch.optim.AdamW", params=AdamConf()),
+)
+
+
+@dataclass
+class AdamaxConf:
+    betas: tuple = (0.9, 0.999)
+    lr: float = 1e-3
+    eps: float = 1e-8
+    weight_decay: float = 0
+
+
+cs.store(
+    group="opt", name="adamax", node=ObjectConf(target="torch.optim.Adamax", params=AdamaxConf()),
+)
+
+
+@dataclass
+class ASGDConf:
+    alpha: float = 0.75
+    lr: float = 1e-3
+    lambd: float = 1e-4
+    t0: float = 1e6
+    weight_decay: float = 0
+
+
+cs.store(
+    group="opt", name="asgd", node=ObjectConf(target="torch.optim.ASGD", params=ASGDConf()),
+)
+
+
+@dataclass
+class LBFGSConf:
+    lr: float = 1
+    max_iter: int = 20
+    max_eval: int = 25
+    tolerance_grad: float = 1e-5
+    tolerance_change: float = 1e-9
+    history_size: int = 100
+    line_search_fn: Optional[str] = None
+
+
+cs.store(
+    group="opt", name="lbfgs", node=ObjectConf(target="torch.optim.LBFGS", params=LBFGSConf()),
+)
+
+
+@dataclass
+class RMSpropConf:
+    lr: float = 1e-2
+    momentum: float = 0
+    alpha: float = 0.99
+    eps: float = 1e-8
+    centered: bool = True
+    weight_decay: float = 0
+
+
+cs.store(
+    group="opt", name="rmsprop", node=ObjectConf(target="torch.optim.RMSprop", params=RMSpropConf()),
+)
+
+
+@dataclass
+class RpropConf:
+    lr: float = 1e-2
+    etas: tuple = (0.5, 1.2)
+    step_sizes: tuple = (1e-6, 50)
+
+
+cs.store(
+    group="opt", name="rprop", node=ObjectConf(target="torch.optim.Rprop", params=RpropConf()),
+)
+
+
+@dataclass
+class SGDConf:
+    lr: float = 1e-2
+    momentum: float = 0
+    weight_decay: float = 0
+    dampening: float = 0
+    nesterov: bool = False
+
+
+cs.store(
+    group="opt", name="sgd", node=ObjectConf(target="torch.optim.SGD", params=SGDConf()),
+)

pl_examples/hydra_examples/conf/scheduler.py

@@ -0,0 +1,139 @@
+from dataclasses import dataclass, field
+from typing import Any, List, Optional
+
+from hydra.core.config_store import ConfigStore
+from hydra.types import ObjectConf
+
+cs = ConfigStore.instance()
+
+
+@dataclass
+class CosineConf:
+    T_max: int = 100
+    eta_min: float = 0
+    last_epoch: int = -1
+
+
+cs.store(
+    group="scheduler",
+    name="cosine",
+    node=ObjectConf(target="torch.optim.lr_scheduler.CosineAnnealingLR", params=CosineConf()),
+)
+
+
+@dataclass
+class CosineWarmConf:
+    T_0: int = 10
+    T_mult: int = 1
+    eta_min: float = 0
+    last_epoch: int = -1
+
+
+cs.store(
+    group="scheduler",
+    name="cosinewarm",
+    node=ObjectConf(target="torch.optim.lr_scheduler.CosineAnnealingLR", params=CosineWarmConf()),
+)
+
+
+@dataclass
+class CyclicConf:
+    base_lr: Any = 1e-3
+    max_lr: Any = 1e-2
+    step_size_up: int = 2000
+    step_size_down: int = 2000
+    mode: str = "triangular"
+    gamma: float = 1
+    scale_fn: Optional[Any] = None
+    scal_mode: str = "cycle"
+    cycle_momentum: bool = True
+    base_momentum: Any = 0.8
+    max_momentum: Any = 0.9
+    last_epoch: int = -1
+
+
+cs.store(
+    group="scheduler", name="cyclic", node=ObjectConf(target="torch.optim.lr_scheduler.CyclicLR", params=CyclicConf()),
+)
+
+
+@dataclass
+class ExponentialConf:
+    gamma: float = 1
+    last_epoch: int = -1
+
+
+cs.store(
+    group="scheduler",
+    name="exponential",
+    node=ObjectConf(target="torch.optim.lr_scheduler.ExponentialLR", params=ExponentialConf()),
+)
+
+
+@dataclass
+class RedPlatConf:
+    mode: str = "min"
+    factor: float = 0.1
+    patience: int = 10
+    verbose: bool = False
+    threshold: float = 1e-4
+    threshold_mode: str = "rel"
+    cooldown: int = 0
+    min_lr: Any = 0
+    eps: float = 1e-8
+
+
+cs.store(
+    group="scheduler",
+    name="redplat",
+    node=ObjectConf(target="torch.optim.lr_scheduler.ReduceLROnPlateau", params=RedPlatConf()),
+)
+
+
+@dataclass
+class MultiStepConf:
+    milestones: List = field(default_factory=lambda: [10, 20, 30, 40])
+    gamma: float = 0.1
+    last_epoch: int = -1
+
+
+cs.store(
+    group="scheduler",
+    name="multistep",
+    node=ObjectConf(target="torch.optim.lr_scheduler.MultiStepLR", params=MultiStepConf()),
+)
+
+
+@dataclass
+class OneCycleConf:
+    max_lr: Any = 1e-2
+    total_steps: int = 2000
+    epochs: int = 200
+    steps_per_epoch: int = 100
+    pct_start: float = 0.3
+    anneal_strategy: str = "cos"
+    cycle_momentum: bool = True
+    base_momentum: Any = 0.8
+    max_momentum: Any = 0.9
+    div_factor: float = 25
+    final_div_factor: float = 1e4
+    last_epoch: int = -1
+
+
+cs.store(
+    group="scheduler",
+    name="onecycle",
+    node=ObjectConf(target="torch.optim.lr_scheduler.OneCycleLR", params=OneCycleConf()),
+)
+
+
+@dataclass
+class StepConf:
+    step_size: int = 20
+    gamma: float = 0.1
+    last_epoch: int = -1
+
+
+cs.store(
+    group="scheduler", name="step", node=ObjectConf(target="torch.optim.lr_scheduler.StepLR", params=StepConf()),
+)

pl_examples/hydra_examples/conf_simple/config_simple.yaml

@@ -0,0 +1,9 @@
+defaults:
+  - model: basic
+  - trainer: trainer
+
+misc:
+  data_root: ${hydra:runtime.cwd}/data
+  batch_size: 128
+  learning_rate: 0.001 
+  num_workers: 4