Call `ckpt_to_weights_subdir` from `MegatronCheckpointIO` #10897

ashors1 · 2024-10-16T04:20:24Z

What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

Collection: [Note which collection this PR will affect]

Changelog

Add specific line by line info of high level changes in this PR.

Usage

You can potentially add a usage example below

# Add a code snippet demonstrating how to use this

GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

Before your PR is "Ready for review"

Pre checks:

Make sure you read and followed Contributor guidelines
Did you write any new necessary tests?
Did you add or update any necessary documentation?
Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
- Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

New Feature
Bugfix
Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

Related to # (issue)

Signed-off-by: ashors1 <[email protected]>

Signed-off-by: artbataev <[email protected]>

Signed-off-by: ashors1 <[email protected]>

nemo/lightning/ckpt_utils.py

nemo/lightning/io/connector.py

@@ -8,7 +8,7 @@
 from filelock import FileLock, Timeout
 from pytorch_lightning.trainer.states import TrainerFn

-from nemo.lightning.ckpt_utils import ckpt_to_context_subdir, ckpt_to_weights_subdir
+from nemo.lightning.ckpt_utils import ckpt_to_context_subdir


nemo/lightning/io/pl.py

Signed-off-by: ashors1 <[email protected]>

nemo/lightning/io/pl.py

Signed-off-by: ashors1 <[email protected]>

github-actions · 2024-10-30T00:09:48Z

[🤖]: Hi @ashors1 👋,

We wanted to let you know that a CICD pipeline for this PR just finished successfully

So it might be time to merge this PR or get some approvals

I'm just a bot so I'll leave it you what to do next.

//cc @pablo-garay @ko3n1g

Signed-off-by: ashors1 <[email protected]>

nemo/lightning/io/pl.py

@@ -37,7 +37,7 @@
 from torch import nn
 from typing_extensions import Self, override

-from nemo.lightning.ckpt_utils import ckpt_to_dir
+from nemo.lightning.ckpt_utils import WEIGHTS_PATH, ckpt_to_dir


Signed-off-by: Chen Cui <[email protected]>

cuichenx · 2024-11-05T15:31:02Z

peft test passed https://github.com/NVIDIA/NeMo/actions/runs/11672168085/job/32500417335 launching full ci now

github-actions · 2024-11-05T18:01:43Z

[🤖]: Hi @ashors1 👋,

We wanted to let you know that a CICD pipeline for this PR just finished successfully

So it might be time to merge this PR or get some approvals

I'm just a bot so I'll leave it you what to do next.

//cc @pablo-garay @ko3n1g

* locate weights path within MegatronCheckpointIO Signed-off-by: ashors1 <[email protected]> * small refactor Signed-off-by: ashors1 <[email protected]> * remove another instance of ckpt_to_weights_subdir Signed-off-by: ashors1 <[email protected]> * move ckpt_to_weights_subdir Signed-off-by: ashors1 <[email protected]> * Apply isort and black reformatting Signed-off-by: ashors1 <[email protected]> * Apply isort and black reformatting Signed-off-by: artbataev <[email protected]> * add weights path in save_checkpoint Signed-off-by: ashors1 <[email protected]> * fix circular import Signed-off-by: ashors1 <[email protected]> * Apply isort and black reformatting Signed-off-by: ashors1 <[email protected]> * handle saving in ckpt_to_weights_subdir Signed-off-by: ashors1 <[email protected]> * fix minor typo Signed-off-by: ashors1 <[email protected]> * bug fixes Signed-off-by: ashors1 <[email protected]> * fix undefined variable Signed-off-by: ashors1 <[email protected]> * move function Signed-off-by: ashors1 <[email protected]> * Apply isort and black reformatting Signed-off-by: ashors1 <[email protected]> * fix adapter meta file path Signed-off-by: Chen Cui <[email protected]> * Apply isort and black reformatting Signed-off-by: cuichenx <[email protected]> * fix mixtral test Signed-off-by: ashors1 <[email protected]> * fix mixtral test Signed-off-by: ashors1 <[email protected]> * use function for weights subdir Signed-off-by: Chen Cui <[email protected]> * address comments Signed-off-by: ashors1 <[email protected]> * move asserts Signed-off-by: ashors1 <[email protected]> * fix undefined vars Signed-off-by: ashors1 <[email protected]> * bug fix Signed-off-by: ashors1 <[email protected]> --------- Signed-off-by: ashors1 <[email protected]> Signed-off-by: ashors1 <[email protected]> Signed-off-by: artbataev <[email protected]> Signed-off-by: Chen Cui <[email protected]> Signed-off-by: cuichenx <[email protected]> Co-authored-by: ashors1 <[email protected]> Co-authored-by: artbataev <[email protected]> Co-authored-by: Chen Cui <[email protected]> Co-authored-by: cuichenx <[email protected]>

ashors1 added 4 commits October 15, 2024 12:57

locate weights path within MegatronCheckpointIO

dcb38f3

Signed-off-by: ashors1 <[email protected]>

small refactor

6010377

Signed-off-by: ashors1 <[email protected]>

remove another instance of ckpt_to_weights_subdir

2417023

Signed-off-by: ashors1 <[email protected]>

move ckpt_to_weights_subdir

eed4bad

Signed-off-by: ashors1 <[email protected]>

ashors1 added Run CICD r2.0.0 labels Oct 16, 2024

ashors1 requested review from cuichenx and akoumpa October 16, 2024 04:20

Apply isort and black reformatting

52c0ad3

Signed-off-by: ashors1 <[email protected]>

github-actions bot added the common label Oct 16, 2024

artbataev and others added 2 commits October 16, 2024 04:21

Apply isort and black reformatting

e5dbd61

Signed-off-by: artbataev <[email protected]>

add weights path in save_checkpoint

45df47d

Signed-off-by: ashors1 <[email protected]>

ashors1 added Run CICD and removed Run CICD labels Oct 16, 2024

akoumpa reviewed Oct 16, 2024

View reviewed changes

nemo/lightning/ckpt_utils.py Outdated Show resolved Hide resolved

github-advanced-security bot found potential problems Oct 16, 2024

View reviewed changes

fix circular import

c49e2a6

Signed-off-by: ashors1 <[email protected]>

ashors1 added Run CICD and removed Run CICD labels Oct 16, 2024

Apply isort and black reformatting

d3ffd5d

Signed-off-by: ashors1 <[email protected]>

ashors1 added Run CICD and removed Run CICD labels Oct 16, 2024

github-advanced-security bot found potential problems Oct 16, 2024

View reviewed changes

nemo/lightning/io/pl.py Fixed Show fixed Hide fixed

ashors1 added 2 commits October 16, 2024 10:49

handle saving in ckpt_to_weights_subdir

ea49e20

Signed-off-by: ashors1 <[email protected]>

fix minor typo

c4c3fd5

Signed-off-by: ashors1 <[email protected]>

ashors1 added Run CICD and removed Run CICD labels Oct 16, 2024

ericharper requested a review from stevehuang52 October 16, 2024 18:47

ericharper removed the r2.0.0 label Oct 16, 2024

bug fixes

3ae933e

Signed-off-by: ashors1 <[email protected]>

ashors1 added the Run CICD label Oct 29, 2024

Merge branch 'main' of github.com:NVIDIA/NeMo into ashors/ckpt-subdirs

2cdd672

ashors1 added Run CICD and removed Run CICD labels Oct 29, 2024

ashors1 added 2 commits October 29, 2024 21:58

Merge branch 'main' of github.com:NVIDIA/NeMo into ashors/ckpt-subdirs

0954f6b

revert some unnecessary changes

e997e24

Signed-off-by: ashors1 <[email protected]>

ashors1 added Run CICD and removed Run CICD labels Oct 30, 2024

Apply isort and black reformatting

7ef6ab7

Signed-off-by: ashors1 <[email protected]>

github-advanced-security bot found potential problems Oct 30, 2024

View reviewed changes

ashors1 added Run CICD and removed Run CICD labels Oct 30, 2024

Merge branch 'main' into ashors/ckpt-subdirs

d20d4a8

cuichenx added Run CICD and removed Run CICD labels Oct 31, 2024

cuichenx self-assigned this Nov 4, 2024

ashors1 added Run CICD and removed Run CICD labels Nov 4, 2024

fix for peft

9c3fa3c

Signed-off-by: Chen Cui <[email protected]>

cuichenx added the Run CICD label Nov 5, 2024

cuichenx approved these changes Nov 5, 2024

View reviewed changes

akoumpa merged commit fb00406 into main Nov 5, 2024
303 of 305 checks passed

akoumpa deleted the ashors/ckpt-subdirs branch November 5, 2024 18:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Call `ckpt_to_weights_subdir` from `MegatronCheckpointIO` #10897

Call `ckpt_to_weights_subdir` from `MegatronCheckpointIO` #10897

ashors1 commented Oct 16, 2024

github-actions bot commented Oct 30, 2024

cuichenx commented Nov 5, 2024

github-actions bot commented Nov 5, 2024

Call ckpt_to_weights_subdir from MegatronCheckpointIO #10897

Call ckpt_to_weights_subdir from MegatronCheckpointIO #10897

Conversation

ashors1 commented Oct 16, 2024

What does this PR do ?

Changelog

Usage

GitHub Actions CI

Before your PR is "Ready for review"

Who can review?

Additional Information

github-actions bot commented Oct 30, 2024

cuichenx commented Nov 5, 2024

github-actions bot commented Nov 5, 2024

Call `ckpt_to_weights_subdir` from `MegatronCheckpointIO` #10897

Call `ckpt_to_weights_subdir` from `MegatronCheckpointIO` #10897