switch to mcore dataset [with FIM support] #8149

dimapihtar · 2024-01-10T14:24:38Z

What does this PR do ?

Switches to megatron core dataset.
Adds FIM support.
Adds starcoder config.

Collection: NLP

Changelog

Added additional functions in oreder to use mcore ds.
Changed NeMo ds classes to mcore ds classes (megatron_gpt_model.py).
Created gpt_fim_dataset.py with GPTFIMDatasetConfig and GPTFIMDataset classes in order to use FIM techinque.
Added new config for starcoder models (megatron_starcoder_config.yaml).

Usage

Example of FIM usage. This part should be added to the data section in the model config:

add_fim: True # Set to True to use FIM
    fim:
      # fill in the middle
      rate: 0.5 # Probability to convert a training sample into a "Fill-in-the-Middle" format. Must be between 0 and 1
      spm_rate: 0.5 # Probability that the a FIM sample uses the SPM format over the PSM format
      split_sample: null # String around which to split the sample for FIM. If None (default), FIM is applied on the sample-level
      fragment_rate: 0.5 # Rate of FIM on each fragment when fim_split_sample is not None
      no_prefix: null # Do not apply FIM to fragments that start with this prefix
      extra_tokens:
        prefix: "<fim_prefix>"
        middle: "<fim_middle>"
        suffix: "<fim_suffix>"
        pad: "<fim_pad>"
        eod: "<|endoftext|>"

Jenkins CI

To run Jenkins, a NeMo User with write access must comment jenkins on the PR.

Before your PR is "Ready for review"

Pre checks:

Make sure you read and followed Contributor guidelines
Did you write any new necessary tests?
Did you add or update any necessary documentation?
Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
- Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

New Feature
Bugfix
Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

Related to # (issue)

Signed-off-by: dimapihtar <[email protected]>

Signed-off-by: Dmytro Pykhtar <[email protected]>

for more information, see https://pre-commit.ci

nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py

Signed-off-by: dimapihtar <[email protected]>

for more information, see https://pre-commit.ci

Signed-off-by: dimapihtar <[email protected]>

…to dpykhtar/mcore_ds Signed-off-by: dimapihtar <[email protected]>

nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py

Signed-off-by: dimapihtar <[email protected]>

for more information, see https://pre-commit.ci

Signed-off-by: dimapihtar <[email protected]>

for more information, see https://pre-commit.ci

Signed-off-by: dimapihtar <[email protected]>

dimapihtar · 2024-01-16T16:44:29Z

jenkins

Signed-off-by: dimapihtar <[email protected]>

dimapihtar · 2024-01-22T22:16:39Z

jenkins

Signed-off-by: dimapihtar <[email protected]>

dimapihtar · 2024-01-23T14:38:06Z

jenkins

Signed-off-by: dimapihtar <[email protected]>

dimapihtar · 2024-01-23T17:37:58Z

jenkins

for more information, see https://pre-commit.ci

dimapihtar · 2024-01-23T17:46:54Z

jenkins

Signed-off-by: dimapihtar <[email protected]>

…to dpykhtar/mcore_ds

dimapihtar · 2024-01-24T17:22:40Z

jenkins

Signed-off-by: dimapihtar <[email protected]>

…to dpykhtar/mcore_ds

dimapihtar · 2024-01-25T13:37:18Z

jenkins

ericharper

LGTM. Thanks!

* switch to mcore dataset Signed-off-by: dimapihtar <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * remove commented lines Signed-off-by: dimapihtar <[email protected]> * fix rank issue Signed-off-by: dimapihtar <[email protected]> * fix rank issue Signed-off-by: dimapihtar <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * remove unnecessary prints Signed-off-by: dimapihtar <[email protected]> * fix typo Signed-off-by: dimapihtar <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add FIM support Signed-off-by: dimapihtar <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * revert gpt config Signed-off-by: dimapihtar <[email protected]> * change if statement Signed-off-by: dimapihtar <[email protected]> * add starcoder config Signed-off-by: dimapihtar <[email protected]> * add starcoder config Signed-off-by: dimapihtar <[email protected]> * remove commented lines Signed-off-by: dimapihtar <[email protected]> * code changes Signed-off-by: dimapihtar <[email protected]> * change mcore commit Signed-off-by: dimapihtar <[email protected]> * add copyright header Signed-off-by: dimapihtar <[email protected]> * code changes Signed-off-by: dimapihtar <[email protected]> * remove if statement Signed-off-by: dimapihtar <[email protected]> * fix batch for fine-tuning Signed-off-by: dimapihtar <[email protected]> * move is_dataset_built_on_rank function Signed-off-by: dimapihtar <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * remove commented lines Signed-off-by: dimapihtar <[email protected]> * config changes Signed-off-by: dimapihtar <[email protected]> * revert gpt config Signed-off-by: dimapihtar <[email protected]> * revert falcon model Signed-off-by: dimapihtar <[email protected]> * fix falcon tests Signed-off-by: dimapihtar <[email protected]> * fix tests Signed-off-by: dimapihtar <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * revert gpt config Signed-off-by: dimapihtar <[email protected]> * comment out MockGPTDataset test Signed-off-by: dimapihtar <[email protected]> --------- Signed-off-by: dimapihtar <[email protected]> Signed-off-by: Dmytro Pykhtar <[email protected]> Signed-off-by: dimapihtar <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* switch to mcore dataset Signed-off-by: dimapihtar <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * remove commented lines Signed-off-by: dimapihtar <[email protected]> * fix rank issue Signed-off-by: dimapihtar <[email protected]> * fix rank issue Signed-off-by: dimapihtar <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * remove unnecessary prints Signed-off-by: dimapihtar <[email protected]> * fix typo Signed-off-by: dimapihtar <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add FIM support Signed-off-by: dimapihtar <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * revert gpt config Signed-off-by: dimapihtar <[email protected]> * change if statement Signed-off-by: dimapihtar <[email protected]> * add starcoder config Signed-off-by: dimapihtar <[email protected]> * add starcoder config Signed-off-by: dimapihtar <[email protected]> * remove commented lines Signed-off-by: dimapihtar <[email protected]> * code changes Signed-off-by: dimapihtar <[email protected]> * change mcore commit Signed-off-by: dimapihtar <[email protected]> * add copyright header Signed-off-by: dimapihtar <[email protected]> * code changes Signed-off-by: dimapihtar <[email protected]> * remove if statement Signed-off-by: dimapihtar <[email protected]> * fix batch for fine-tuning Signed-off-by: dimapihtar <[email protected]> * move is_dataset_built_on_rank function Signed-off-by: dimapihtar <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * remove commented lines Signed-off-by: dimapihtar <[email protected]> * config changes Signed-off-by: dimapihtar <[email protected]> * revert gpt config Signed-off-by: dimapihtar <[email protected]> * revert falcon model Signed-off-by: dimapihtar <[email protected]> * fix falcon tests Signed-off-by: dimapihtar <[email protected]> * fix tests Signed-off-by: dimapihtar <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * revert gpt config Signed-off-by: dimapihtar <[email protected]> * comment out MockGPTDataset test Signed-off-by: dimapihtar <[email protected]> --------- Signed-off-by: dimapihtar <[email protected]> Signed-off-by: Dmytro Pykhtar <[email protected]> Signed-off-by: dimapihtar <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: stevehuang52 <[email protected]>

* switch to mcore dataset Signed-off-by: dimapihtar <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * remove commented lines Signed-off-by: dimapihtar <[email protected]> * fix rank issue Signed-off-by: dimapihtar <[email protected]> * fix rank issue Signed-off-by: dimapihtar <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * remove unnecessary prints Signed-off-by: dimapihtar <[email protected]> * fix typo Signed-off-by: dimapihtar <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add FIM support Signed-off-by: dimapihtar <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * revert gpt config Signed-off-by: dimapihtar <[email protected]> * change if statement Signed-off-by: dimapihtar <[email protected]> * add starcoder config Signed-off-by: dimapihtar <[email protected]> * add starcoder config Signed-off-by: dimapihtar <[email protected]> * remove commented lines Signed-off-by: dimapihtar <[email protected]> * code changes Signed-off-by: dimapihtar <[email protected]> * change mcore commit Signed-off-by: dimapihtar <[email protected]> * add copyright header Signed-off-by: dimapihtar <[email protected]> * code changes Signed-off-by: dimapihtar <[email protected]> * remove if statement Signed-off-by: dimapihtar <[email protected]> * fix batch for fine-tuning Signed-off-by: dimapihtar <[email protected]> * move is_dataset_built_on_rank function Signed-off-by: dimapihtar <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * remove commented lines Signed-off-by: dimapihtar <[email protected]> * config changes Signed-off-by: dimapihtar <[email protected]> * revert gpt config Signed-off-by: dimapihtar <[email protected]> * revert falcon model Signed-off-by: dimapihtar <[email protected]> * fix falcon tests Signed-off-by: dimapihtar <[email protected]> * fix tests Signed-off-by: dimapihtar <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * revert gpt config Signed-off-by: dimapihtar <[email protected]> * comment out MockGPTDataset test Signed-off-by: dimapihtar <[email protected]> --------- Signed-off-by: dimapihtar <[email protected]> Signed-off-by: Dmytro Pykhtar <[email protected]> Signed-off-by: dimapihtar <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: Sasha Meister <[email protected]>

* switch to mcore dataset Signed-off-by: dimapihtar <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * remove commented lines Signed-off-by: dimapihtar <[email protected]> * fix rank issue Signed-off-by: dimapihtar <[email protected]> * fix rank issue Signed-off-by: dimapihtar <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * remove unnecessary prints Signed-off-by: dimapihtar <[email protected]> * fix typo Signed-off-by: dimapihtar <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add FIM support Signed-off-by: dimapihtar <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * revert gpt config Signed-off-by: dimapihtar <[email protected]> * change if statement Signed-off-by: dimapihtar <[email protected]> * add starcoder config Signed-off-by: dimapihtar <[email protected]> * add starcoder config Signed-off-by: dimapihtar <[email protected]> * remove commented lines Signed-off-by: dimapihtar <[email protected]> * code changes Signed-off-by: dimapihtar <[email protected]> * change mcore commit Signed-off-by: dimapihtar <[email protected]> * add copyright header Signed-off-by: dimapihtar <[email protected]> * code changes Signed-off-by: dimapihtar <[email protected]> * remove if statement Signed-off-by: dimapihtar <[email protected]> * fix batch for fine-tuning Signed-off-by: dimapihtar <[email protected]> * move is_dataset_built_on_rank function Signed-off-by: dimapihtar <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * remove commented lines Signed-off-by: dimapihtar <[email protected]> * config changes Signed-off-by: dimapihtar <[email protected]> * revert gpt config Signed-off-by: dimapihtar <[email protected]> * revert falcon model Signed-off-by: dimapihtar <[email protected]> * fix falcon tests Signed-off-by: dimapihtar <[email protected]> * fix tests Signed-off-by: dimapihtar <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * revert gpt config Signed-off-by: dimapihtar <[email protected]> * comment out MockGPTDataset test Signed-off-by: dimapihtar <[email protected]> --------- Signed-off-by: dimapihtar <[email protected]> Signed-off-by: Dmytro Pykhtar <[email protected]> Signed-off-by: dimapihtar <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: Pablo Garay <[email protected]>

* switch to mcore dataset Signed-off-by: dimapihtar <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * remove commented lines Signed-off-by: dimapihtar <[email protected]> * fix rank issue Signed-off-by: dimapihtar <[email protected]> * fix rank issue Signed-off-by: dimapihtar <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * remove unnecessary prints Signed-off-by: dimapihtar <[email protected]> * fix typo Signed-off-by: dimapihtar <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add FIM support Signed-off-by: dimapihtar <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * revert gpt config Signed-off-by: dimapihtar <[email protected]> * change if statement Signed-off-by: dimapihtar <[email protected]> * add starcoder config Signed-off-by: dimapihtar <[email protected]> * add starcoder config Signed-off-by: dimapihtar <[email protected]> * remove commented lines Signed-off-by: dimapihtar <[email protected]> * code changes Signed-off-by: dimapihtar <[email protected]> * change mcore commit Signed-off-by: dimapihtar <[email protected]> * add copyright header Signed-off-by: dimapihtar <[email protected]> * code changes Signed-off-by: dimapihtar <[email protected]> * remove if statement Signed-off-by: dimapihtar <[email protected]> * fix batch for fine-tuning Signed-off-by: dimapihtar <[email protected]> * move is_dataset_built_on_rank function Signed-off-by: dimapihtar <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * remove commented lines Signed-off-by: dimapihtar <[email protected]> * config changes Signed-off-by: dimapihtar <[email protected]> * revert gpt config Signed-off-by: dimapihtar <[email protected]> * revert falcon model Signed-off-by: dimapihtar <[email protected]> * fix falcon tests Signed-off-by: dimapihtar <[email protected]> * fix tests Signed-off-by: dimapihtar <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * revert gpt config Signed-off-by: dimapihtar <[email protected]> * comment out MockGPTDataset test Signed-off-by: dimapihtar <[email protected]> --------- Signed-off-by: dimapihtar <[email protected]> Signed-off-by: Dmytro Pykhtar <[email protected]> Signed-off-by: dimapihtar <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

switch to mcore dataset

d6247fd

Signed-off-by: dimapihtar <[email protected]>

github-actions bot added the NLP label Jan 10, 2024

dimapihtar and others added 2 commits January 10, 2024 16:36

Merge branch 'main' into dpykhtar/mcore_ds

f18928e

Signed-off-by: Dmytro Pykhtar <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

12de8fd

for more information, see https://pre-commit.ci

github-advanced-security bot found potential problems Jan 10, 2024

View reviewed changes

nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py Fixed Show fixed Hide fixed

dimapihtar and others added 7 commits January 10, 2024 08:22

remove commented lines

a957966

Signed-off-by: dimapihtar <[email protected]>

fix rank issue

afebd65

Signed-off-by: dimapihtar <[email protected]>

fix rank issue

8a07519

Signed-off-by: dimapihtar <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

20241f5

for more information, see https://pre-commit.ci

remove unnecessary prints

6734fb3

Signed-off-by: dimapihtar <[email protected]>

Merge branch 'dpykhtar/mcore_ds' of https://github.com/NVIDIA/NeMo in…

31735e0

…to dpykhtar/mcore_ds Signed-off-by: dimapihtar <[email protected]>

Merge branch 'main' into dpykhtar/mcore_ds

48f4b8c

github-advanced-security bot found potential problems Jan 11, 2024

View reviewed changes

nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py Fixed Show fixed Hide fixed

nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py Fixed Show fixed Hide fixed

dimapihtar and others added 2 commits January 12, 2024 06:30

fix typo

e37dba3

Signed-off-by: dimapihtar <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

91789b5

for more information, see https://pre-commit.ci

dimapihtar changed the title ~~switch to mcore dataset~~ switch to mcore dataset [with FIM support] Jan 15, 2024

dimapihtar and others added 11 commits January 15, 2024 17:19

Merge branch 'main' into dpykhtar/mcore_ds

d4d515e

add FIM support

e43dd09

Signed-off-by: dimapihtar <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

4877011

for more information, see https://pre-commit.ci

revert gpt config

2d9dc42

Signed-off-by: dimapihtar <[email protected]>

change if statement

e198bdb

Signed-off-by: dimapihtar <[email protected]>

add starcoder config

a3e3c9c

Signed-off-by: dimapihtar <[email protected]>

add starcoder config

6114e2a

Signed-off-by: dimapihtar <[email protected]>

remove commented lines

7253779

Signed-off-by: dimapihtar <[email protected]>

Merge branch 'main' into dpykhtar/mcore_ds

9229474

code changes

1d00520

Signed-off-by: dimapihtar <[email protected]>

change mcore commit

2ab25b7

Signed-off-by: dimapihtar <[email protected]>

github-actions bot added the CI label Jan 16, 2024

add copyright header

9635e31

Signed-off-by: dimapihtar <[email protected]>

dimapihtar and others added 3 commits January 22, 2024 14:12

config changes

9d9bd3f

Signed-off-by: dimapihtar <[email protected]>

revert gpt config

9b02de9

Signed-off-by: dimapihtar <[email protected]>

Merge branch 'main' into dpykhtar/mcore_ds

8d7014d

dimapihtar requested a review from ericharper January 22, 2024 22:16

dimapihtar and others added 3 commits January 23, 2024 06:32

revert falcon model

c3b26ce

Signed-off-by: dimapihtar <[email protected]>

fix falcon tests

d41c6fc

Signed-off-by: dimapihtar <[email protected]>

Merge branch 'main' into dpykhtar/mcore_ds

54fa7a1

dimapihtar and others added 2 commits January 23, 2024 09:35

fix tests

11d7d4d

Signed-off-by: dimapihtar <[email protected]>

Merge branch 'main' into dpykhtar/mcore_ds

52b1960

[pre-commit.ci] auto fixes from pre-commit.com hooks

771be12

for more information, see https://pre-commit.ci

dimapihtar and others added 3 commits January 24, 2024 09:19

revert gpt config

417f1d4

Signed-off-by: dimapihtar <[email protected]>

Merge branch 'dpykhtar/mcore_ds' of https://github.com/NVIDIA/NeMo in…

6dfdf9b

…to dpykhtar/mcore_ds

Merge branch 'main' into dpykhtar/mcore_ds

cef6719

dimapihtar and others added 3 commits January 25, 2024 05:33

comment out MockGPTDataset test

dd4ac81

Signed-off-by: dimapihtar <[email protected]>

Merge branch 'dpykhtar/mcore_ds' of https://github.com/NVIDIA/NeMo in…

d1b09b7

…to dpykhtar/mcore_ds

Merge branch 'main' into dpykhtar/mcore_ds

f11c4fd

ericharper approved these changes Jan 25, 2024

View reviewed changes

ericharper merged commit a93589d into main Jan 25, 2024
15 checks passed

ericharper deleted the dpykhtar/mcore_ds branch January 25, 2024 22:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

switch to mcore dataset [with FIM support] #8149

switch to mcore dataset [with FIM support] #8149

dimapihtar commented Jan 10, 2024 •

edited

Loading

dimapihtar commented Jan 16, 2024

dimapihtar commented Jan 22, 2024

dimapihtar commented Jan 23, 2024

dimapihtar commented Jan 23, 2024

dimapihtar commented Jan 23, 2024

dimapihtar commented Jan 24, 2024

dimapihtar commented Jan 25, 2024

ericharper left a comment

switch to mcore dataset [with FIM support] #8149

switch to mcore dataset [with FIM support] #8149

Conversation

dimapihtar commented Jan 10, 2024 • edited Loading

What does this PR do ?

Changelog

Usage

Jenkins CI

Before your PR is "Ready for review"

Who can review?

Additional Information

dimapihtar commented Jan 16, 2024

dimapihtar commented Jan 22, 2024

dimapihtar commented Jan 23, 2024

dimapihtar commented Jan 23, 2024

dimapihtar commented Jan 23, 2024

dimapihtar commented Jan 24, 2024

dimapihtar commented Jan 25, 2024

ericharper left a comment

Choose a reason for hiding this comment

dimapihtar commented Jan 10, 2024 •

edited

Loading