Learning Rate finder

CHANGELOG.md

@@ -28,6 +28,8 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 - Added option to run without an optimizer by returning `None` from `configure_optimizers`. ([#1279](https://github.com/PyTorchLightning/pytorch-lightning/pull/1279))
 - Added a warning when the number of data loader workers is small. ([#1378](https://github.com/PyTorchLightning/pytorch-lightning/pull/1378))
 
+- Added learining rate finder ([#1347](https://github.com/PyTorchLightning/pytorch-lightning/pull/1347))
+
 ### Changed
 
 - Changed `progress_bar_refresh_rate` trainer flag to disable progress bar when set to 0. ([#1108](https://github.com/PyTorchLightning/pytorch-lightning/pull/1108))

docs/source/index.rst

@@ -66,6 +66,7 @@ PyTorch Lightning Documentation
    fast_training
    hooks
    hyperparameters
+   lr_finder
    multi_gpu
    weights_loading
    optimizers

docs/source/lr_finder.rst

@@ -0,0 +1,72 @@
+Learning Rate Finder
+--------------------
+
+For training deep neural networks, selecting a good learning rate is essential
+for both better performance and faster convergence. Even optimizers such as
+`Adam` that are self-adjusting the learning rate can benefit from more optimal
+choices.
+
+To reduce the amount of guesswork concerning choosing a good initial learning
+rate, a `learning rate finder` can be used. As described in this `paper <https://arxiv.org/abs/1506.01186>`_ 
+a learning rate finder does a small run where the learning rate is increased 
+after each processed batch and the corresponding loss is logged. The result of 
+this is a `lr` vs. `loss` plot that can be used as guidence for choosing a optimal
+initial lr. 
+
+Using Lightnings build-in LR finder
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In the most basic use case, this feature can be enabled during trainer construction
+with ``Trainer(auto_lr_find=True)``. When ``.fit(model)`` is called, the lr finder
+will automatically be run before any training is done. The ``lr`` that is found
+and used will be written to the console and logged together with all other
+hyperparameters of the model.
+
+.. note:: If ``auto_lr_find=True``, it is expected that the ``hparams`` of the 
+    model either has a ``lr`` or ``learning_rate`` field that can be overridden. 
+    Additionally ``auto_lr_find`` can be set to a string ``s``, which will then
+    try to override ``model.hparams.s``. In both cases, if the respective fields
+    are not found, an error will be thrown.
+
+If you want to inspect the results of the learning rate finder before doing any
+actual training or just play around with the parameters of the algorithm, this
+can be done by invoking the ``lr_find`` method of the trainer. A typical example
+of this would look like
+
+.. code-block:: python
+
+    model = MyModelClass(hparams)
+    trainer = pl.Trainer()
+
+    # Run learning rate finder
+    lr_finder = trainer.lr_find(model)
+
+    # Results can be found in
+    lr_finder.results
+
+    # Plot with
+    fig = lr_finder.plot(suggest=True)
+    fig.show()
+
+    # Pick point based on plot, or get suggestion
+    new_lr = lr_finder.suggestion()
+
+    # update hparams of the model
+    model.hparams.lr = new_lr
+
+    # Fit model
+    trainer.fit(model)
+
+The figure produced by ``lr_finder.plot()`` should look something like the figure
+below. It is recommended to not pick the learning rate that achives the lowest
+loss, but instead something in the middle of the sharpest downward slope (red point).
+This is the point returned py ``lr_finder.suggestion()``.
+
+.. figure:: /_images/trainer/lr_finder.png
+
+The parameters of the algorithm can be seen below.
+
+.. autoclass:: pytorch_lightning.trainer.lr_finder.TrainerLRFinderMixin
+   :members: lr_find
+   :noindex:
+   :exclude-members: _run_lr_finder_internally, save_checkpoint, restore

pytorch_lightning/trainer/__init__.py

@@ -135,6 +135,27 @@ def forward(self, x):
     # default used by the Trainer
     trainer = Trainer(amp_level='O1')
 
+auto_lr_find
+^^^^^^^^^^^^
+Runs a learning rate finder algorithm (see this `paper <https://arxiv.org/abs/1506.01186>`_)
+before any training, to find optimal initial learning rate.
+
+.. code-block:: python
+
+    # default used by the Trainer (no learning rate finder)
+    trainer = Trainer(auto_lr_find=False)
+
+Example::
+
+    # run learning rate finder, results override hparams.learning_rate
+    trainer = Trainer(auto_lr_find=True)
+
+    # run learning rate finder, results override hparams.my_lr_arg
+    trainer = Trainer(auto_lr_find='my_lr_arg')
+
+.. note::
+    See the `learning rate finder guide <lr_finder.rst>`_
+
 benchmark
 ^^^^^^^^^