Megatron BART

[michalivne]

What does this PR do ?

This PR adds support in BART Megatron model https://arxiv.org/abs/1910.13461.

Collection: nemo.collections.nlp.models.language_modeling.megatron_bart_model

Usage

python /code/examples/nlp/language_modeling/megatron_bart_pretraining.py \
	--config-path=/code/examples/nlp/language_modeling/conf \
	--config-name=megatron_bart_config \
	trainer.num_nodes=1 \
	trainer.devices=1 \
	trainer.max_epochs=null \
	trainer.max_steps=${MAX_STEPS} \
	trainer.val_check_interval=${VAL_CHECK_INTERVAL} \
	trainer.accumulate_grad_batches=${ACCUMULATE_GRAD_BATCHES} \
	trainer.precision=16 \
	model.micro_batch_size=${MICRO_BATCH_SIZE} \
	model.tensor_model_parallel_size=${TENSOR_MODEL_PARALLEL_SIZE} \
	model.max_position_embeddings=${SEQ_LENGTH} \
	model.seq_length=${SEQ_LENGTH} \
	model.hidden_size=${HIDDEN_SIZE} \
	model.num_layers=${NUM_LAYERS} \
	model.num_attention_heads=${NUM_ATTENTION_HEADS} \
	model.init_method_std=0.015 \
	model.data.data_prefix=[1.0,/preproc_data/train/my-t5_test_bert_tokenizer_text_document] \
	model.data.seq_length=${SEQ_LENGTH} \

[lgtm-com]

This pull request introduces 2 alerts when merging c84dbb4 into c45aabe - view on LGTM.com

new alerts:

• 1 for Wrong name for an argument in a call
• 1 for Mismatch in multiple assignment

[lgtm-com]

This pull request introduces 2 alerts when merging 8eab12d into 69385da - view on LGTM.com

new alerts:

• 1 for Wrong name for an argument in a call
• 1 for Mismatch in multiple assignment

[lgtm-com]

This pull request introduces 4 alerts when merging b8cbfad into 989e6ab - view on LGTM.com

new alerts:

• 3 for Unused import
• 1 for Mismatch in multiple assignment

[lgtm-com]

This pull request introduces 4 alerts when merging daf9891 into 989e6ab - view on LGTM.com

new alerts:

• 3 for Unused import
• 1 for Mismatch in multiple assignment

[lgtm-com]

This pull request introduces 4 alerts when merging db88329 into 989e6ab - view on LGTM.com

new alerts:

• 3 for Unused import
• 1 for Mismatch in multiple assignment

[lgtm-com]

This pull request introduces 4 alerts when merging d6eb5d5 into fe30166 - view on LGTM.com

new alerts:

• 3 for Unused import
• 1 for Mismatch in multiple assignment

[lgtm-com]

This pull request introduces 4 alerts when merging f7223b1 into fe30166 - view on LGTM.com

new alerts:

• 3 for Unused import
• 1 for Mismatch in multiple assignment

[lgtm-com]

This pull request introduces 4 alerts when merging 31e536f into fe30166 - view on LGTM.com

new alerts:

• 3 for Unused import
• 1 for Mismatch in multiple assignment

[lgtm-com]

This pull request introduces 5 alerts when merging 964a86a into fe30166 - view on LGTM.com

new alerts:

• 3 for Unused import
• 1 for Mismatch in multiple assignment
• 1 for Unused local variable

[MaximumEntropy]

LGTM! Like the model name property you've added!

[lgtm-com]

This pull request introduces 2 alerts and fixes 1 when merging 6abf603 into ff91628 - view on LGTM.com

new alerts:

• 2 for Mismatch in multiple assignment

fixed alerts:

• 1 for Mismatch in multiple assignment

[lgtm-com]

This pull request introduces 2 alerts and fixes 1 when merging 0b08d2a into 60f4c6c - view on LGTM.com

new alerts:

• 2 for Mismatch in multiple assignment

fixed alerts:

• 1 for Mismatch in multiple assignment

[lgtm-com]

This pull request introduces 2 alerts and fixes 1 when merging 6c6b7fd into 60f4c6c - view on LGTM.com

new alerts:

• 2 for Mismatch in multiple assignment

fixed alerts:

• 1 for Mismatch in multiple assignment

[lgtm-com]

This pull request introduces 2 alerts and fixes 1 when merging c23b980 into 60f4c6c - view on LGTM.com

new alerts:

• 2 for Mismatch in multiple assignment

fixed alerts:

• 1 for Mismatch in multiple assignment

[lgtm-com]

This pull request introduces 2 alerts and fixes 1 when merging d136dba into 60f4c6c - view on LGTM.com

new alerts:

• 2 for Mismatch in multiple assignment

fixed alerts:

• 1 for Mismatch in multiple assignment

[lgtm-com]

This pull request introduces 3 alerts and fixes 1 when merging 69f020c into 60f4c6c - view on LGTM.com

new alerts:

• 2 for Mismatch in multiple assignment
• 1 for `__init__` method calls overridden method

fixed alerts:

• 1 for Mismatch in multiple assignment

[lgtm-com]

This pull request introduces 3 alerts and fixes 1 when merging 1c94524 into 60f4c6c - view on LGTM.com

new alerts:

• 2 for Mismatch in multiple assignment
• 1 for `__init__` method calls overridden method

fixed alerts:

• 1 for Mismatch in multiple assignment

[ericharper]

LGTM. Thanks!

[lgtm-com]

This pull request introduces 3 alerts and fixes 1 when merging 81e5065 into e4a1b20 - view on LGTM.com

new alerts:

• 2 for Mismatch in multiple assignment
• 1 for `__init__` method calls overridden method

fixed alerts:

• 1 for Mismatch in multiple assignment