【PaddlePaddle Hackathon 3 No.48】为 Paddle 新增 elementwise op support fp16 #45496

xiaohemaikoo · 2022-08-28T14:56:27Z

PR types

New features

PR changes

OPs

Describe

elementwise op support fp16

performance

Case No.	input_shape	FP32 Perf(us)	FP16 Perf(us)	diff
0 pow	[1000, 130, 17]	109.41	104.38	1.048
1 pow	[1000, 100, 10, 10]	474.33	432.76	1.096
2 pow	[1000, 100, 200]	937.03	854.88	1.096
3 pow	[100, 1000, 25, 40]	4456.1	3663.3	1.216
0 mod	[1000, 130, 17]	95.715	69.184	1.383
1 mod	[1000, 100, 10, 10]	472.66	295.51	1.599
2 mod	[1000, 100, 200]	963.66	578.45	1.665
3 mod	[100, 1000, 25, 40]	4432.7	2456.8	1.804
0 fmin	[1000, 130, 17]	95.847	55.266	1.734
1 fmin	[1000, 100, 10, 10]	471.40	221.61	2.127
2 fmin	[1000, 100, 200]	965.76	434.67	2.221
3 fmin	[100, 1000, 25, 40]	4393.4	1964.7	2.236
0 fmax	[1000, 130, 17]	97.508	52.384	1.861
1 fmax	[1000, 100, 10, 10]	478.41	219.75	2.177
2 fmax	[1000, 100, 200]	938.72	430.82	2.178
3 fmax	[100, 1000, 25, 40]	4443.8	1975.2	2.249
0 heaviside	[1000, 130, 17]	95.455	51.394	1.857
1 heaviside	[1000, 100, 10, 10]	481.63	221.41	2.175
2 heaviside	[1000, 100, 200]	952.66	434.75	2.191
3 heaviside	[100, 1000, 25, 40]	4438.8	2003.0	2.216

diff = FP32 / FP16

docs：PaddlePaddle/docs#5249

paddle-bot · 2022-08-28T14:56:33Z

你的PR提交成功，感谢你对开源项目的贡献!
请关注后续CI自动化测试结果，详情请参考Paddle-CI手册。
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

luotao1 · 2022-08-30T02:55:18Z

@xiaohemaikoo 可以先解决下代码冲突的问题，请 @zhangting2020 继续审核

paddle/phi/kernels/elementwise_kernel.cc

paddle/phi/kernels/funcs/elementwise_functor.h

paddle/phi/kernels/impl/elementwise_grad_kernel_impl.h

zhangting2020 · 2022-08-30T06:29:53Z

paddle/phi/kernels/impl/elementwise_grad_kernel_impl.h

+    float result = tmp_dout * tmp_y * std::pow(tmp_x, tmp_y - 1.0f);
+    return static_cast<dtype::float16>(result);
+  }
+};


这里可以参考你前向的修改方式，在原始的functor上进行修改就可以，不必要单独增加实现。

zhangting2020 · 2022-08-30T06:30:28Z

paddle/phi/kernels/impl/elementwise_grad_kernel_impl.h

+    float result = tmp_pow * tmp_dout * std::log(tmp_x);
+    return static_cast<dtype::float16>(result);
+  }
+};


zhangting2020 · 2022-08-30T07:08:37Z

python/paddle/fluid/tests/unittests/test_elementwise_pow_op.py

+            self.check_output(check_eager=True)
+
+    def test_check_grad(self):
+        if self.dtype not in [np.int32, np.int64]:


这个判断应该不需要，因为这个class是fp16类型的测试，其他类型不会涉及到这里面的逻辑

python/paddle/fluid/tests/unittests/test_mode_op.py

zhangting2020 · 2022-08-30T07:31:24Z

其他问题：

python api部分，需要将文档中的数据类型增加上float16。例如

Paddle/python/paddle/tensor/math.py

Lines 4668 to 4669 in 871e332

    
                   x (Tensor): The input tensor of Heaviside step function, it's data type should be float32, float64, int32 or int64. 
        
                   y (Tensor): The tensor that determines a Heaviside step function, it's data type should be float32, float64, int32 or int64.

中文文档也需要添加float16类型
PR描述中我已经增加了一个性能对比的模版，请自己设置几组shape，进行性能对比

paddle/phi/kernels/funcs/elementwise_functor.h

xiaohemaikoo · 2022-09-01T22:46:39Z

python api部分，中英文档中的数据类型均已提交PR增加float16类型。
性能对比测试数据正在收集最新数据，本地环境编译测试过程比较漫长。

最新代码已经修改大部分comments，可能部分comments修改不够完美。
因为以上comments讨论比较多。
希望有时间可以重新检视review一下最新提交代码的修改。
列出还有可以提升修改的地方。
感谢指导！！^_^

zhangting2020 · 2022-09-02T06:52:26Z

PR修改已经符合要求，感谢你的贡献！待性能数据补充验证没有问题，CI通过，我们会给予approve并合入。

Ligoml · 2022-09-02T07:00:10Z

请在对应的中文API文档中也添加支持fp16数据类型支持的说明哈~中文API文档放在 https://github.com/PaddlePaddle/docs/tree/develop/docs/api/paddle

xiaohemaikoo · 2022-09-03T12:37:47Z

我修改了math.py英文文档英文注释，PR-CI-Static-Check 需要approval。

另：
求review 修改中文API文档的PR，CI Docs-NEW也需要approval。
这是修改中文API文档的PR： PaddlePaddle/docs#5249

xiaohemaikoo · 2022-09-03T14:46:30Z

你好，性能对比测试数据已经更新到 PR描述性能对比测试数据表格中。
感谢指导！！^_^

zhangting2020

LGTM

xiaohemaikoo · 2022-09-06T00:15:17Z

LGTM

OK, thanks. CI 还有几个没通过的，好像和我没关系，我不需要关注了吧。

Ligoml

LGTM for docs，中文文档还有一个小问题需要修改下

paddle-bot bot added contributor External developers status: proposed labels Aug 28, 2022

xiaohemaikoo force-pushed the ma-elem branch from 8659a2f to ffce9bf Compare August 28, 2022 15:40

luotao1 assigned luotao1, zhangting2020 and Ligoml Aug 29, 2022

luotao1 added the PaddlePaddle Hackathon label Aug 29, 2022

paddle-bot bot removed the status: proposed label Aug 29, 2022

xiaohemaikoo force-pushed the ma-elem branch from ffce9bf to 56c241a Compare August 29, 2022 12:00

xiaohemaikoo mentioned this pull request Aug 29, 2022

【PaddlePaddle Hackathon 第三期】任务总览 #43938

Closed

This was referenced Aug 30, 2022

【PaddlePaddle Hackathon 3 No.48】为 Paddle 新增 elementwise pow raw 支持 float16 #45286

Closed

【PaddlePaddle Hackathon 3 No.48】为 Paddle 新增 elementwise mod support fp16 #45491

Closed

zhangting2020 reviewed Aug 30, 2022

View reviewed changes

xiaohemaikoo force-pushed the ma-elem branch from 2d44539 to 0ffd230 Compare August 31, 2022 02:27

zhangting2020 reviewed Sep 1, 2022

View reviewed changes

paddle/phi/kernels/funcs/elementwise_functor.h Outdated Show resolved Hide resolved

paddle/phi/kernels/funcs/elementwise_functor.h Show resolved Hide resolved

paddle/phi/kernels/funcs/elementwise_functor.h Show resolved Hide resolved

xiaohemaikoo force-pushed the ma-elem branch 2 times, most recently from 24c299d to aa4d996 Compare September 1, 2022 21:36

elementwise op support fp16

d356103

xiaohemaikoo force-pushed the ma-elem branch from aa4d996 to d356103 Compare September 1, 2022 22:35

zhangting2020 approved these changes Sep 5, 2022

View reviewed changes

luotao1 requested a review from Ligoml September 6, 2022 02:04

Ligoml approved these changes Sep 6, 2022

View reviewed changes

luotao1 merged commit f6d9ec2 into PaddlePaddle:develop Sep 6, 2022

Caozhou1995 pushed a commit to Caozhou1995/Paddle that referenced this pull request Sep 9, 2022

elementwise op support fp16 (PaddlePaddle#45496)

0e26c05

piDack mentioned this pull request Mar 13, 2023

[AMP OP&Test] Append bf16/fp16 support 4 elementwise_max #51151

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

【PaddlePaddle Hackathon 3 No.48】为 Paddle 新增 elementwise op support fp16 #45496

【PaddlePaddle Hackathon 3 No.48】为 Paddle 新增 elementwise op support fp16 #45496

xiaohemaikoo commented Aug 28, 2022 •

edited by Ligoml

Loading

paddle-bot bot commented Aug 28, 2022

luotao1 commented Aug 30, 2022

zhangting2020 Aug 30, 2022

zhangting2020 Aug 30, 2022

zhangting2020 Aug 30, 2022

xiaohemaikoo Aug 31, 2022

zhangting2020 commented Aug 30, 2022 •

edited

Loading

xiaohemaikoo commented Sep 1, 2022

zhangting2020 commented Sep 2, 2022

Ligoml commented Sep 2, 2022

xiaohemaikoo commented Sep 3, 2022

xiaohemaikoo commented Sep 3, 2022

zhangting2020 left a comment

xiaohemaikoo commented Sep 6, 2022

Ligoml left a comment

【PaddlePaddle Hackathon 3 No.48】为 Paddle 新增 elementwise op support fp16 #45496

【PaddlePaddle Hackathon 3 No.48】为 Paddle 新增 elementwise op support fp16 #45496

Conversation

xiaohemaikoo commented Aug 28, 2022 • edited by Ligoml Loading

PR types

PR changes

Describe

performance

paddle-bot bot commented Aug 28, 2022

luotao1 commented Aug 30, 2022

zhangting2020 Aug 30, 2022

Choose a reason for hiding this comment

zhangting2020 Aug 30, 2022

Choose a reason for hiding this comment

zhangting2020 Aug 30, 2022

Choose a reason for hiding this comment

xiaohemaikoo Aug 31, 2022

Choose a reason for hiding this comment

zhangting2020 commented Aug 30, 2022 • edited Loading

xiaohemaikoo commented Sep 1, 2022

zhangting2020 commented Sep 2, 2022

Ligoml commented Sep 2, 2022

xiaohemaikoo commented Sep 3, 2022

xiaohemaikoo commented Sep 3, 2022

zhangting2020 left a comment

Choose a reason for hiding this comment

xiaohemaikoo commented Sep 6, 2022

Ligoml left a comment

Choose a reason for hiding this comment

xiaohemaikoo commented Aug 28, 2022 •

edited by Ligoml

Loading

zhangting2020 commented Aug 30, 2022 •

edited

Loading