Feature request: GELU operator

[eric-haibin-lin]

Reference paper:
https://arxiv.org/abs/1606.08415

Reference python impl: https://github.com/huggingface/pytorch-openai-transformer-lm/blob/master/model_pytorch.py#L14-L15

Used in transformer language model:
https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf

[hendrycks]

This nonlinearity is also used in BERT, the state-of-the-art NLP model for various tasks: https://arxiv.org/abs/1810.04805
(Disclosure: I'm on the GELU paper.)

[haojin2]

Working on it now...

[haojin2]

PR #14449 should address this issue @hendrycks Please give a review if you have time, thanks!

[sandeep-krishnamurthy]

Closing as the PR is now merged. Thanks haojin