Added softplus FP32 FWD OneDNN kernel #36382
Conversation
|
Thanks for your contribution! |
|
@piotrekobiIntel please review this PR |
python/paddle/fluid/tests/unittests/mkldnn/test_softplus_mkldnn_op.py
Outdated
Show resolved
Hide resolved
|
@piotrekobiIntel Could you please continue your review? |
| dnnl::post_ops post_ops; | ||
| post_ops.append_eltwise(1.0f, dnnl::algorithm::eltwise_soft_relu, 0.0f, | ||
| 0.0f); | ||
| if (beta != 1.0f) { | ||
| post_ops.append_eltwise(1.0f, dnnl::algorithm::eltwise_linear, | ||
| 1.0f / beta, 0.0f); | ||
| } | ||
|
|
||
| dnnl::primitive_attr attrs; | ||
| attrs.set_post_ops(post_ops); | ||
|
|
||
| this->AcquireForwardPrimitiveDescriptor(attrs, dnnl::algorithm::binary_mul, | ||
| x_md, beta_md, x_md); |
There was a problem hiding this comment.
Something like this would allow to skip the multiplication by 1 if beta = 1, this exact code probably won't work but I hope you understand what I mean. I'm not sure if it's worth putting in the extra work though (it depends how often beta is equal to 1 in practice).
| dnnl::post_ops post_ops; | |
| post_ops.append_eltwise(1.0f, dnnl::algorithm::eltwise_soft_relu, 0.0f, | |
| 0.0f); | |
| if (beta != 1.0f) { | |
| post_ops.append_eltwise(1.0f, dnnl::algorithm::eltwise_linear, | |
| 1.0f / beta, 0.0f); | |
| } | |
| dnnl::primitive_attr attrs; | |
| attrs.set_post_ops(post_ops); | |
| this->AcquireForwardPrimitiveDescriptor(attrs, dnnl::algorithm::binary_mul, | |
| x_md, beta_md, x_md); | |
| if (beta == 1.0f) | |
| { | |
| this->AcquireForwardPrimitiveDescriptor(attrs, dnnl::algorithm::eltwise_soft_relu, x_md, x_md); | |
| } | |
| else | |
| { | |
| dnnl::post_ops post_ops; | |
| post_ops.append_eltwise(1.0f, dnnl::algorithm::eltwise_soft_relu, 0.0f, | |
| 0.0f); | |
| post_ops.append_eltwise(1.0f, dnnl::algorithm::eltwise_linear, | |
| 1.0f / beta, 0.0f); | |
| dnnl::primitive_attr attrs; | |
| attrs.set_post_ops(post_ops); | |
| this->AcquireForwardPrimitiveDescriptor(attrs, dnnl::algorithm::binary_mul, | |
| x_md, beta_md, x_md); | |
| } |
There was a problem hiding this comment.
It was done like that in the previous commits of this PR, but I have agreed with Jacek, that the overall change in performance was meaningless, and this way the code is unified and much more clear. Moreover, this operator will be fused with tanh activation(for ppyolov2_r50vd_365e model), so in that case binary operation must be done, because eltwise primitive does not support fusing with another eltwise primitive. But you've definitely got a point that execution time would be faster if there would be just soft_relu without binary_mul at the beginning
PR types
New features
PR changes
OPs
Describe
Added softplus FP32 FWD OneDNN kernel. It improves ppyolov2_r50vd_365e_coco model by 14%