[1/4] Add get_device_stats to accelerator interface

[daniellepintz]

What does this PR do?

Adds get_device_stats function to accelerator class, and implements it for GPU and TPU. To be called in later PRs
Tested with daniellepintz#2

Part of #9032

Please also see subsequent PRs:
daniellepintz#2
daniellepintz#3
daniellepintz#4

Does your PR introduce any breaking changes? If yes, please list them.

Before submitting

• Was this discussed/approved via a GitHub issue? (not for typos and docs)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure your PR does only one thing, instead of bundling different changes together?
• Did you make sure to update the documentation with your changes? (if necessary)
• Did you write any new necessary tests? (not for typos and docs)
• Did you verify new and existing tests pass locally with your changes?
• Did you list all the breaking changes introduced by this pull request?
• Did you update the CHANGELOG? (not for typos, docs, test updates, or internal minor changes/refactorings)

PR review

Anyone in the community is welcome to review the PR.
Before you start reviewing make sure you have read Review guidelines. In short, see the following bullet-list:

• Is this pull request ready for review? (if not, please submit in draft mode)
• Check that all items from Before submitting are resolved
• Make sure the title is self-explanatory and the description concisely explains the PR
• Add labels and milestones (and optionally projects) to the PR so it can be classified

Did you have fun?

Make sure you had fun coding 🙃

[codecov]

Codecov Report

Merging #9586 (ccadca5) into master (c7451b3) will decrease coverage by 4%.
The diff coverage is 28%.

❗ Current head ccadca5 differs from pull request most recent head 07bc597. Consider uploading reports for the commit 07bc597 to get more accurate results

@@           Coverage Diff           @@
##           master   #9586    +/-   ##
=======================================
- Coverage      93%     89%    -4%     
=======================================
  Files         181     179     -2     
  Lines       15349   15350     +1     
=======================================
- Hits        14246   13591   -655     
- Misses       1103    1759   +656     

[tchaton]

Hey @daniellepintz, thanks for linking all the following PRs :)

noob question for PR 2/4. Should we differentiate on_train_batch_start and on_train_batch_end ?

Best,
T.C

[ananthsub]

@daniellepintz it'd be good to also add tests for the GPU and TPU accelerator to confirm that get_device_stats returns something

[daniellepintz]

noob question for PR 2/4. Should we differentiate on_train_batch_start and on_train_batch_end ?

@tchaton this was modeled after GPUStatsMonitor where we log in both places. I myself am not sure what is best so please lmk if you think I should change this

[daniellepintz]

@ananthsub I believe any tests I would write would be similar to the ones in https://github.com/daniellepintz/pytorch-lightning/pull/2/files, or did you have something else in mind?

[ananthsub]

@ananthsub I believe any tests I would write would be similar to the ones in https://github.com/daniellepintz/pytorch-lightning/pull/2/files, or did you have something else in mind?

the tests there are end to end tests relying on multiple components: the accelerator, callback, and trainer.

the accelerator tests would test similar functionality but should be more self-contained as a unit test specifically for get_device_stats. for example, we could instantiate some large tensors on gpu, and ensure that we get corresponding keys back from the GPU Accelerator as a sanity check.

having the tests here & in the callback will be helpful to detect if something breaks in one place vs another

[daniellepintz]

Looking at the output of the CI TPU tests I see two odd things:

1. There are a bunch of RuntimeErrors but it says at the bottom that all the tests passed (this is the case on other PRs as well)
2. I can't find test_device_stats_tpu in the output (added in this PR), and it doesn't seem to be running it.

Does anyone have any insight into these issues?

[tchaton]

Hey @kaushikb11 Mind looking into @daniellepintz TPU testing ?