[codecarbon] integration #15
Conversation
|
Sorry, is this ready to be merged? I am unsure from your initial comment :) |
|
Not yet, Waiting for the codecarbon group to address the issue above. |
|
blocking issue: mlco2/codecarbon#217 |
|
Current solution: bigscience-workshop/codecarbon#1 update: the maintainers created a different solution, see the next comment. |
|
@jaketae, please have a look before we merge this. If you want to test, first install mlco2/codecarbon#236 Thanks |
|
Thanks for keeping me in the loop. I've checked the upstream PR. I also recall there was another PR before this one, but I think the more recent solution suits our purpose better. I think we can merge this one! |
|
once they merge their side, yes. Thank you for reviewing, @jaketae. |
|
mlco2/codecarbon#236 is in so merging this. |
* add codecarbon * switch to offline * rework to also restart the tracker at each checkpoint save to ensure as little as possible data is lost * adjust API to match bigscience-workshop/codecarbon#1 * fix logging * new implementation based on mlco2/codecarbon#236 * add test * update requirements
| if not hasattr(args, 'codecarbon_dir'): | ||
| return |
There was a problem hiding this comment.
I think this should check if it's none. @TevenLeScao
Basically this line adds the attribute, but assigns None when not set: https://github.com/bigscience-workshop/Megatron-DeepSpeed/pull/15/files#diff-5f7d1ddfb0666cb6bb4ec0f07fd2fd7b1cd0354f421df5560489091db2ff5a55R455
So I believe hasattr(args, "codecarbon_dir") will return True. Despite having no paths.
* add direct meg-ds to hf format script (bigscience-workshop#110) * add direct meg-ds to hf format script (part2) (bigscience-workshop#111) * add direct meg-ds to hf format script * split into 2 function * update the usage doc * make scripts executable * add shebang Co-authored-by: Stas Bekman <stas00@users.noreply.github.com> Co-authored-by: Stas Bekman <stas@stason.org>
4 participants
This PR plugs in https://github.com/mlco2/codecarbon/ via:
Currently requires mlco2/codecarbon#236 commit
@jaketae