Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

Fp16 support for softmax #14072

Closed
eric-haibin-lin opened this issue Feb 6, 2019 · 2 comments
Closed

Fp16 support for softmax #14072

eric-haibin-lin opened this issue Feb 6, 2019 · 2 comments

Comments

@eric-haibin-lin
Copy link
Member

eric-haibin-lin commented Feb 6, 2019

Currently, given fp16 inputs, nd.softmax/sym.softmax perform reduction in fp16, which losses precision. The reduction should be done in fp32 instead.

https://github.com/apache/incubator-mxnet/blob/32c9ca74839ae4d275bcf9a027ea0a711373be81/src/operator/nn/softmax-inl.h#L164-L202
pytorch reference:
https://github.com/zdevito/ATen/blob/a6cc4156fe4abc9e31f62f2bba1a2f68c58b77b7/aten/src/ATen/native/cuda/SoftMax.cu#L43-L55

@mxnet-label-bot
Copy link
Contributor

Hey, this is the MXNet Label Bot.
Thank you for submitting the issue! I will try and suggest some labels so that the appropriate MXNet community members can help resolve it.
Here are my recommended labels: Feature

@eric-haibin-lin
Copy link
Member Author

fixed in #14098

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

2 participants