Adapt NeptuneLogger to new neptune-client api (#6867)

* Initial split to NeptuneLegacyLogger and NeptuneLogger

* Adapt NeptuneLogger to neptune-pytorch-lightning repo version

* Fix stylecheck and tests

* Fix style and PR suggestions

* Expect Run object in NeptuneLogger.init

* Model checkpoint support and restructured tests

* Reformat code - use " instead of '

* Fix logging INTEGRATION_VERSION_KEY

* Update CHANGELOG.md

* Fix stylecheck

* Remove NeptuneLegacyLogger

* updated neptune-related docstrings

* PR suggestions

* update CODEOWERS file
* move import logic to imports.py
* minor neptune.py improvements

* formatting fixes and minor updates

* Fix generation of docs

* formatting fixes and minor updates

* fix

* PR fixes vol. 2

* define return type of _dict_paths method
* bump required version of `neptune-client`

* Enable log_* functions

* Update pytorch_lightning/loggers/neptune.py

Co-authored-by: Adrian Wälchli <[email protected]>

* Revert "Enable log_* functions"

This reverts commit 050a436.

* Make global helper lists internal

* Logger's `name` and `version` methods return proper results

* Initialize Run and its name and id at logger init level

* Make _init_run_instance static

* Add pre-commit hook code changes.

* Fix blacken-docs check

* Fix neptune doctests and test_all

* added docs comment about neptune-specific syntax

* added docs comment about neptune-specific syntax in the loggers.rst

* fix

* Add pickling test

* added myself to neptune codeowners

* Enable some of deprecated log_* functions

* Restore _run_instance for unpickled logger

* Add `step` parameter to log_* functions

* Fix stylecheck

* Fix checkstyle

* Fix checkstyle

* Update pytorch_lightning/loggers/neptune.py

Co-authored-by: thomas chaton <[email protected]>

* Fix tests

* Fix stylecheck

* fixed project name

* Fix windows tests

* Fix stylechecks

* Fix neptune docs tests

* docformatter fixes

* De-duplicate legacy_kwargs_msg

* Update .github/CODEOWNERS

Co-authored-by: Adrian Wälchli <[email protected]>

Co-authored-by: kamil-kaczmarek <[email protected]>
Co-authored-by: Adrian Wälchli <[email protected]>
Co-authored-by: thomas chaton <[email protected]>

.gitignore

@@ -147,6 +147,7 @@ wandb
 .forked/
 *.prof
 *.tar.gz
+.neptune/
 
 # dataset generated from bolts in examples.
 cifar-10-batches-py

CHANGELOG.md

@@ -124,6 +124,11 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 
 ### Changed
 
+- `pytorch_lightning.loggers.neptune.NeptuneLogger` is now consistent with new [neptune-client](https://github.com/neptune-ai/neptune-client) API ([#6867](https://github.com/PyTorchLightning/pytorch-lightning/pull/6867)).
+
+  Old [neptune-client](https://github.com/neptune-ai/neptune-client) API is supported by `NeptuneClient` from [neptune-contrib](https://github.com/neptune-ai/neptune-contrib) repo.
+
+
 - Parsing of the `gpus` Trainer argument has changed: `gpus="n"` (str) no longer selects the GPU index n and instead selects the first n devices. ([#8770](https://github.com/PyTorchLightning/pytorch-lightning/pull/8770))
 
 

docs/source/common/loggers.rst

@@ -9,7 +9,7 @@
 Loggers
 *******
 
-Lightning supports the most popular logging frameworks (TensorBoard, Comet, etc...). TensorBoard is used by default,
+Lightning supports the most popular logging frameworks (TensorBoard, Comet, Neptune, etc...). TensorBoard is used by default,
 but you can pass to the :class:`~pytorch_lightning.trainer.trainer.Trainer` any combination of the following loggers.
 
 .. note::
@@ -107,34 +107,50 @@ First, install the package:
 
     pip install neptune-client
 
+or with conda:
+
+.. code-block:: bash
+
+    conda install -c conda-forge neptune-client
+
 Then configure the logger and pass it to the :class:`~pytorch_lightning.trainer.trainer.Trainer`:
 
-.. testcode::
+.. code-block:: python
 
     from pytorch_lightning.loggers import NeptuneLogger
 
     neptune_logger = NeptuneLogger(
         api_key="ANONYMOUS",  # replace with your own
-        project_name="shared/pytorch-lightning-integration",
-        experiment_name="default",  # Optional,
-        params={"max_epochs": 10},  # Optional,
-        tags=["pytorch-lightning", "mlp"],  # Optional,
+        project="common/pytorch-lightning-integration",  # format "<WORKSPACE/PROJECT>"
+        tags=["training", "resnet"],  # optional
     )
     trainer = Trainer(logger=neptune_logger)
 
 The :class:`~pytorch_lightning.loggers.NeptuneLogger` is available anywhere except ``__init__`` in your
 :class:`~pytorch_lightning.core.lightning.LightningModule`.
 
-.. testcode::
+.. code-block:: python
 
     class MyModule(LightningModule):
         def any_lightning_module_function_or_hook(self):
-            some_img = fake_image()
-            self.logger.experiment.add_image("generated_images", some_img, 0)
+            # generic recipe for logging custom metadata (neptune specific)
+            metadata = ...
+            self.logger.experiment["your/metadata/structure"].log(metadata)
+
+Note that syntax: ``self.logger.experiment["your/metadata/structure"].log(metadata)``
+is specific to Neptune and it extends logger capabilities.
+Specifically, it allows you to log various types of metadata like scores, files,
+images, interactive visuals, CSVs, etc. Refer to the
+`Neptune docs <https://docs.neptune.ai/you-should-know/logging-metadata#essential-logging-methods>`_
+for more detailed explanations.
+
+You can always use regular logger methods: ``log_metrics()`` and ``log_hyperparams()`` as these are also supported.
 
 .. seealso::
     :class:`~pytorch_lightning.loggers.NeptuneLogger` docs.
 
+    Logger `user guide <https://docs.neptune.ai/integrations-and-supported-tools/model-training/pytorch-lightning>`_.
+
 ----------------
 
 Tensorboard
@@ -227,7 +243,7 @@ Then configure the logger and pass it to the :class:`~pytorch_lightning.trainer.
 The :class:`~pytorch_lightning.loggers.WandbLogger` is available anywhere except ``__init__`` in your
 :class:`~pytorch_lightning.core.lightning.LightningModule`.
 
-.. testcode::
+.. code-block:: python
 
     class MyModule(LightningModule):
         def any_lightning_module_function_or_hook(self):