Skip to content

Expose GLU activations as arguments #69

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jaketae merged 15 commits into from
Aug 22, 2021
Merged

Expose GLU activations as arguments #69

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
3ba7882
feat: expose glu activations as argument
jaketae Aug 18, 2021
f004a5a
chore: rename activations -> glu_activations
jaketae Aug 18, 2021
c064e2e
refactor: use lookup dict instead of `getattr()`
jaketae Aug 18, 2021
2944fe4
refactor: mv lookup dict to `glu_activations.py`
jaketae Aug 21, 2021
7d04752
Merge branch 'main' into activation-args
jaketae Aug 21, 2021
3729ebc
chore: rm unnecessary default arg
jaketae Aug 21, 2021
04e397e
Merge branch 'activation-args' of github.com:bigscience-workshop/Mega…
jaketae Aug 21, 2021
b3c6bbe
test: add bf16 test; gelu in `test_training_all()`
jaketae Aug 22, 2021
ff248f0
Update megatron/testing_utils.py
jaketae Aug 22, 2021
0056ab7
refactor: use `require_torch_bf16` decorator
jaketae Aug 22, 2021
4097af6
chore: comment out bf16 test
jaketae Aug 22, 2021
dc61f89
consistent style
stas00 Aug 22, 2021
6798e81
fix look up table
stas00 Aug 22, 2021
ba8f404
better grouping
stas00 Aug 22, 2021
0f36f70
fix: replace hard coded options with `GLU_ACTIVATIONS`
jaketae Aug 22, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions megatron/arguments.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -313,6 +313,11 @@ def _add_network_size_args(parser):
default=PositionEmbeddingType.absolute,
help='Define position embedding type ("absolute" | "rotary"). "absolute" by default.'
)
group.add_argument('--glu-activation', type=str,
choices=["liglu", "geglu", "reglu", "swiglu"],
Comment thread
jaketae marked this conversation as resolved.
Outdated
default="",
Comment thread
jaketae marked this conversation as resolved.
Outdated
help='GLU activations to use.'
)

return parser

Expand Down
0 megatron/model/activations.py → megatron/model/glu_activations.py
View file
Open in desktop
File renamed without changes.
11 changes: 10 additions & 1 deletion megatron/model/transformer.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@

import deepspeed

from .glu_activations import geglu, liglu, reglu, swiglu
from .positional_embeddings import RotaryEmbedding, apply_rotary_pos_emb_torch, apply_rotary_pos_emb

# flags required to enable jit fusion kernels
Expand Down Expand Up @@ -76,7 +77,15 @@ def __init__(self, init_method, output_layer_init_method):

self.bias_gelu_fusion = args.bias_gelu_fusion
self.activation_func = F.gelu
if args.openai_gelu:
if args.glu_activation:
glu_lookup = {
Comment thread
jaketae marked this conversation as resolved.
Outdated
"gegelu": geglu,
"liglu": liglu,
"reglu": reglu,
"swiglu": swiglu,
}
self.activation_func = glu_lookup[args.glu_activation]
elif args.openai_gelu:
self.activation_func = openai_gelu
elif args.onnx_safe:
self.activation_func = erf_gelu
Expand Down