Adapt `NeptuneLogger` to new `neptune-client` api #6867

shnela · 2021-04-07T12:15:14Z

Adapt NeptuneLogger to changes in neptune-client api changes

Deprecated NeptuneLogger was moved to separate repo
New NeptuneLogger which supports new neptune-client is created
Tests for new NeptuneLogger are implemented
Documentation has been updated
piggyback:
Broken proxy support in old NeptuneLogger PyTorch Lightning Integration neptune-ai/neptune-client#413
- This issue was closely related to other changes in this PR

Before submitting

Was this discussed/approved via a GitHub issue? (not for typos and docs)
Did you read the contributor guideline, Pull Request section?
Did you make sure your PR does only one thing, instead of bundling different changes together?
Did you make sure to update the documentation with your changes? (if necessary)
Did you write any new necessary tests? (not for typos and docs)
Did you verify new and existing tests pass locally with your changes?
Did you update the CHANGELOG? (not for typos, docs, test updates, or internal minor changes/refactorings)

PR review

Anyone in the community is free to review the PR once the tests have passed.
Before you start reviewing make sure you have read Review guidelines. In short, see the following bullet-list:

Is this pull request ready for review? (if not, please submit in draft mode)
Check that all items from Before submitting are resolved
Make sure the title is self-explanatory and the description concisely explains the PR
Add labels and milestones (and optionally projects) to the PR so it can be classified

Did you have fun?

I made sure I had fun coding 🙃

codecov · 2021-04-07T12:34:18Z

Codecov Report

Merging #6867 (9338bf3) into master (439db22) will decrease coverage by 3%.
The diff coverage is 91%.

@@           Coverage Diff           @@
##           master   #6867    +/-   ##
=======================================
- Coverage      92%     89%    -3%     
=======================================
  Files         179     180     +1     
  Lines       14904   15046   +142     
=======================================
- Hits        13750   13358   -392     
- Misses       1154    1688   +534

pep8speaks · 2021-04-14T08:20:09Z

Hello @shnela! Thanks for updating this PR.

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2021-05-13 09:09:34 UTC

shnela · 2021-04-19T11:51:16Z

@williamFalcon @awaelchli @Borda @SeanNaren @carmocca @tchaton @justusschock @kaushikb11 @edenlightning

Hi guys.
Is there anything I have to do before you can start review of this branch?
From my perspective it's done, but if I need to fix something before you can start review, or if I can help you somehow with that, don't hesitate to ask ;)

.gitignore

pytorch_lightning/loggers/neptune.py

README.md

awaelchli

LGTM on a high level.

README.md

pytorch_lightning/loggers/neptune.py

Borda

Pls, consider rather use a new name for this new logger than rename the old Legacy...

CHANGELOG.md

pytorch_lightning/loggers/neptune.py

shnela · 2021-04-22T22:41:15Z

@awaelchli @Borda

Ok, everything updated and ready to merge.

awaelchli · 2021-04-23T12:42:46Z

Thanks a lot! We will follow up here as soon as 1.3 is released (May 5ish). We currently have feature freeze for stability testing and will not be able to merge new features and refactors.

pytorch_lightning/loggers/neptune.py

shnela · 2021-09-10T09:10:02Z

@awaelchli @tchaton so can we merge now? :p

.github/CODEOWNERS

Co-authored-by: Adrian Wälchli <[email protected]>

tchaton · 2021-09-10T17:40:07Z

Hey @shnela,

Great work there ! Really excited about this :)

Best,
T.C

shnela · 2021-09-11T00:14:16Z

@tchaton

Great work there ! Really excited about this :)

We to! Thank you for review and your help!

Best,
Jakub

aarzchan · 2021-09-27T22:06:19Z

Hi! For the new NeptuneLogger, how do you set offline_mode? The only thing I could find in the docs is this.

kamil-kaczmarek · 2021-09-27T22:19:53Z

Hi @aarzchan ,

In the new logger you can do it by parameterizing your Logger, like this (mode="offline"):

neptune_logger = NeptuneLogger(
    api_key="ANONYMOUS",  # replace with your own
    project="WORKSPACE/PROJECT",
    mode="offline",
)

There are more modes available.
Overall you pass any neptune.init() argument to the NeptuneLogger and in this way further define its' behavior.

aarzchan · 2021-09-27T22:30:36Z

Thanks, @kamil-kaczmarek!

raman-r-4978 · 2021-09-30T16:32:09Z

Hi @shnela @kamil-kaczmarek ,
Is it possible to use old neptune API when using master? If yes, are there any examples?

kamil-kaczmarek · 2021-10-02T14:28:18Z

Hi @raman-rajarathinam,

We are preparing an update to the neptune legacy-docs, where this use case will be supported and described.

We are planning to release an update next week.

i008 · 2021-11-05T10:08:27Z

Hi @kamil-kaczmarek after upgrading to pl==1.5.0 Neptune stopped working for me:

When i start it it print the experiment multiple times and keeps printing the messege as below.

https://app.neptune.ai/i008/bovine/e/BOV-575
https://app.neptune.ai/i008/bovine/e/BOV-575
https://app.neptune.ai/i008/bovine/e/BOV-575
https://app.neptune.ai/i008/bovine/e/BOV-575
https://app.neptune.ai/i008/bovine/e/BOV-575
https://app.neptune.ai/i008/bovine/e/BOV-575
https://app.neptune.ai/i008/bovine/e/BOV-575
https://app.neptune.ai/i008/bovine/e/BOV-575
https://app.neptune.ai/i008/bovine/e/BOV-575
https://app.neptune.ai/i008/bovine/e/BOV-575
https://app.neptune.ai/i008/bovine/e/BOV-575
https://app.neptune.ai/i008/bovine/e/BOV-575
https://app.neptune.ai/i008/bovine/e/BOV-575
https://app.neptune.ai/i008/bovine/e/BOV-575
https://app.neptune.ai/i008/bovine/e/BOV-575
Global seed set to 1245
Error occurred during asynchronous operation processing: Timestamp must be non-decreasing for series attribute: monitoring/stdout. Invalid point: 2021-11-05T10:06:48.990Z
Error occurred during asynchronous operation processing: Timestamp must be non-decreasing for series attribute: monitoring/stdout. Invalid point: 2021-11-05T10:06:48.992Z
Error occurred during asynchronous operation processing: Timestamp must be non-decreasing for series attribute: monitoring/stdout. Invalid point: 2021-11-05T10:06:49.359Z
Error occurred during asynchronous operation processing: Timestamp must be non-decreasing for series attribute: monitoring/stdout. Invalid point: 2021-11-05T10:06:49.361Z
Error occurred during asynchronous operation processing: Timestamp must be non-decreasing for series attribute: monitoring/stdout. Invalid point: 2021-11-05T10:06:49.722Z
Error occurred during asynchronous operation processing: Timestamp must be non-decreasing for series attribute: monitoring/stdout. Invalid point: 2021-11-05T10:06:49.726Z
Error occurred during asynchronous operation processing: Timestamp must be non-decreasing for series attribute: monitoring/stdout. Invalid point: 2021-11-05T10:06:50.099Z
Error occurred during asynchronous operation processing: Timestamp must be non-decreasing for series attribute: monitoring/stdout. Invalid point: 2021-11-05T10:06:50.105Z
Error occurred during asynchronous operation processing: Timestamp must be non-decreasing for series attribute: monitoring/stdout. Invalid point: 2021-11-05T10:06:50.477Z
Error occurred during asynchronous operation processing: Timestamp must be non-decreasing for series attribute: monitoring/stdout. Invalid point: 2021-11-05T10:06:50.481Z
Error occurred during asynchronous operation processing: Timestamp must be non-decreasing for series attribute: monitoring/stdout. Invalid point: 2021-11-05T10:06:50.840Z

kamil-kaczmarek · 2021-11-05T10:29:40Z

Hey @i008,

Did the same code worked well before upgrade to 1.5 ?

To confirm: is it training on many machines or single-machine with multiple processes?

i008 · 2021-11-05T10:37:10Z

One local gpu. The (almost) same code worked. I had to adjust the init of the logger.

kamil-kaczmarek · 2021-11-05T12:16:12Z

@i008,

Do you use CUSTOM_RUN_ID or pass the run object around in your code base?
Can you share what you modified in the init?

Problem is that async processes write to the same place (stdout in your case) in Neptune and this causes these errors msgs. In practice it does not break your run, these points are simply skipped.

There are few strategies to mitigate that. Happy to chat online if it's ok for you.

i008 · 2021-11-05T12:56:28Z

Hey @kamil-kaczmarek
Whats the best way to get in touch online? Maybe you can send me a hangout invite @ [email protected] Im avaialble next ~3 hours if that works for you.

i008 · 2021-11-05T13:19:02Z

I did some further debugging and it seems that this i related to the CLI interface. The problem only occurs while using neptune like that:

    neptune_logger = NeptuneLogger(
        api_key=NEPTUNE_KEY,
        project='projec',
        source_files=["**/*.py", "**/*.ipynb"],

    )
    
    train_defaults = {
        'logger': neptune_logger,
        'gpus': 1,
        'precision': 16

    }
    cli = CLI(
        LitAutoEncoder,
        Data,
        trainer_defaults=train_defaults,
    )

If i do it the regular way ( trainer = , trainer.fit()) things work as expected.

kamil-kaczmarek · 2021-11-05T13:29:37Z

Great, thanks for the info. Let's discuss more online

carmocca · 2022-08-31T14:38:26Z

@shnela @kamil-kaczmarek The deprecated code can now be removed (_signal_deprecated_api_usage)

shnela marked this pull request as ready for review April 14, 2021 08:08

shnela requested review from awaelchli, Borda, carmocca, edenlightning, justusschock, kaushikb11, SeanNaren, tchaton and williamFalcon as code owners April 14, 2021 08:08

mergify bot added has conflicts and removed has conflicts labels Apr 14, 2021

awaelchli added 3rd party Related to a 3rd-party logger Related to the Loggers feature Is an improvement or enhancement labels Apr 14, 2021

shnela force-pushed the dev/update_neptune_logger branch 2 times, most recently from 9c0bc40 to 4cfa2d0 Compare April 16, 2021 10:10

mergify bot added the has conflicts label Apr 16, 2021

carmocca reviewed Apr 19, 2021

View reviewed changes

.gitignore Outdated Show resolved Hide resolved

pytorch_lightning/loggers/neptune.py Outdated Show resolved Hide resolved

README.md Outdated Show resolved Hide resolved

awaelchli reviewed Apr 20, 2021

View reviewed changes

README.md Outdated Show resolved Hide resolved

pytorch_lightning/loggers/neptune.py Outdated Show resolved Hide resolved

awaelchli added this to the v1.4 milestone Apr 20, 2021

Borda added the design Includes a design discussion label Apr 20, 2021

Borda previously requested changes Apr 20, 2021

View reviewed changes

mergify bot removed the has conflicts label Apr 22, 2021

PiotrJander reviewed Apr 23, 2021

View reviewed changes

pytorch_lightning/loggers/neptune.py Outdated Show resolved Hide resolved

shnela added 4 commits September 9, 2021 20:15

Fix stylechecks

b7d56c3

Fix neptune docs tests

2789eb5

docformatter fixes

b8758ac

De-duplicate legacy_kwargs_msg

5b11de8

shnela force-pushed the dev/update_neptune_logger branch from d9e6ca6 to 5b11de8 Compare September 9, 2021 18:15

mergify bot removed the has conflicts label Sep 9, 2021

ethanwharris approved these changes Sep 10, 2021

View reviewed changes

awaelchli reviewed Sep 10, 2021

View reviewed changes

.github/CODEOWNERS Outdated Show resolved Hide resolved

awaelchli enabled auto-merge (squash) September 10, 2021 13:10

awaelchli mentioned this pull request Sep 10, 2021

add neptune staff as code owners for neptune.py #9438

Merged

11 tasks

Update .github/CODEOWNERS

9338bf3

Co-authored-by: Adrian Wälchli <[email protected]>

auto-merge was automatically disabled September 10, 2021 13:24
Head branch was pushed to by a user without write access

awaelchli enabled auto-merge (squash) September 10, 2021 13:32

carmocca merged commit ee37872 into Lightning-AI:master Sep 10, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adapt `NeptuneLogger` to new `neptune-client` api #6867

Adapt `NeptuneLogger` to new `neptune-client` api #6867

shnela commented Apr 7, 2021 •

edited by awaelchli

Loading

codecov bot commented Apr 7, 2021 •

edited

Loading

pep8speaks commented Apr 14, 2021 •

edited

Loading

shnela commented Apr 19, 2021

awaelchli left a comment

Borda left a comment

shnela commented Apr 22, 2021

awaelchli commented Apr 23, 2021

shnela commented Sep 10, 2021

tchaton commented Sep 10, 2021

shnela commented Sep 11, 2021

aarzchan commented Sep 27, 2021

kamil-kaczmarek commented Sep 27, 2021

aarzchan commented Sep 27, 2021

raman-r-4978 commented Sep 30, 2021

kamil-kaczmarek commented Oct 2, 2021

i008 commented Nov 5, 2021

kamil-kaczmarek commented Nov 5, 2021 •

edited

Loading

i008 commented Nov 5, 2021

kamil-kaczmarek commented Nov 5, 2021

i008 commented Nov 5, 2021

i008 commented Nov 5, 2021

kamil-kaczmarek commented Nov 5, 2021 •

edited

Loading

carmocca commented Aug 31, 2022

Adapt NeptuneLogger to new neptune-client api #6867

Adapt NeptuneLogger to new neptune-client api #6867

Conversation

shnela commented Apr 7, 2021 • edited by awaelchli Loading

Adapt NeptuneLogger to changes in neptune-client api changes

Before submitting

PR review

Did you have fun?

codecov bot commented Apr 7, 2021 • edited Loading

Codecov Report

pep8speaks commented Apr 14, 2021 • edited Loading

Comment last updated at 2021-05-13 09:09:34 UTC

shnela commented Apr 19, 2021

awaelchli left a comment

Choose a reason for hiding this comment

Borda left a comment

Choose a reason for hiding this comment

shnela commented Apr 22, 2021

awaelchli commented Apr 23, 2021

shnela commented Sep 10, 2021

tchaton commented Sep 10, 2021

shnela commented Sep 11, 2021

aarzchan commented Sep 27, 2021

kamil-kaczmarek commented Sep 27, 2021

aarzchan commented Sep 27, 2021

raman-r-4978 commented Sep 30, 2021

kamil-kaczmarek commented Oct 2, 2021

i008 commented Nov 5, 2021

kamil-kaczmarek commented Nov 5, 2021 • edited Loading

i008 commented Nov 5, 2021

kamil-kaczmarek commented Nov 5, 2021

i008 commented Nov 5, 2021

i008 commented Nov 5, 2021

kamil-kaczmarek commented Nov 5, 2021 • edited Loading

carmocca commented Aug 31, 2022

Adapt `NeptuneLogger` to new `neptune-client` api #6867

Adapt `NeptuneLogger` to new `neptune-client` api #6867

shnela commented Apr 7, 2021 •

edited by awaelchli

Loading

codecov bot commented Apr 7, 2021 •

edited

Loading

pep8speaks commented Apr 14, 2021 •

edited

Loading

kamil-kaczmarek commented Nov 5, 2021 •

edited

Loading

kamil-kaczmarek commented Nov 5, 2021 •

edited

Loading