[Fluid] Move distributed_fused_lamb_init to phi #55993

zeroRains · 2023-08-05T07:59:42Z

PR types

Others

PR changes

Others

Description

将distributed_fused_lamb_init 迁移到phi下

paddle-bot · 2023-08-05T07:59:47Z

你的PR提交成功，感谢你对开源项目的贡献!
请关注后续CI自动化测试结果，详情请参考Paddle-CI手册。
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

zeroRains · 2023-08-08T12:26:29Z

@GhostScreaming 老师可以帮我看一下，这个enforce应该怎么改吗？

hitywt · 2023-08-09T09:34:02Z

paddle/phi/kernels/fusion/gpu/distributed_fused_lamb_init_kernel.cu

+    std::vector<DenseTensor *> master_param_out,
+    std::vector<DenseTensor *> grad_out,
+    DenseTensor *global_scale,
+    DenseTensor *step) {


这里代码风格建议，&和*都写在靠近类型的地方吧

@zeroRains 辛苦改一下~

额，这个应该是我的代码格式工具自动把我改好的风格又改回去了，等enforce的那个问题修好后我再一起push上来吧

@GhostScreaming 老师，这个我修改成靠近类型之后，用pre-commit时他又自动改回靠变量那边了。。。。这咋整

先这样吧，问题不大

hitywt · 2023-08-09T09:38:55Z

paddle/phi/kernels/fusion/gpu/distributed_fused_lamb_init_kernel.cu

+
+    PADDLE_ENFORCE_EQ(
+        param.size(),
+        param_out.size(),


这里param_out需要申请空间，否则size是0，可以参考fluid下MultiOutput的内部实现

…e output shape problem

zeroRains · 2023-08-18T02:02:57Z

@GhostScreaming 老师，pr可以review了

GhostScreaming · 2023-08-22T07:33:13Z

paddle/phi/kernels/fusion/cpu/distributed_fused_lamb_init_kernel.cc

+  kernel->OutputAt(7).SetDataType(phi::DataType::FLOAT32);
+  kernel->OutputAt(8).SetDataType(phi::DataType::INT32);
+  kernel->OutputAt(9).SetDataType(phi::DataType::INT32);
+  kernel->OutputAt(10).SetDataType(phi::DataType::INT32);


param_out, grad_out的类型应该与Input的param一致，（可能为phi::DataType::FLOAT32或者phi::DataType::FLOAT16）master_param_out的类型应该为phi::DataType::FLOAT32

GhostScreaming · 2023-08-22T07:44:29Z

paddle/phi/kernels/fusion/gpu/cast_with_ptr.h

paddle/fluid/operators/optimizers/cast_with_ptr.h可以删掉了，只有distributed_fused_lamb_init依赖这个文件

GhostScreaming · 2023-08-22T07:46:07Z

paddle/phi/kernels/fusion/gpu/distributed_fused_lamb_init_kernel.cu

+
+    CopyAndShareBufferForInitedTensor(
+        dev_ctx, info.param_t, fp16_p_t, info.numel_offset);
+    // PADDLE_ENFORCE_EQ(


为啥删掉这个检查？

@GhostScreaming 老师可以帮我看一下，这个enforce应该怎么改吗？

额之前我是试着改了一下，但是发生了编译错误，也问了一下老师您。您可能没看到，后面我尝试用一个指针变量接收分配结果，再进行检查也不行，所以就先注释了先改别的。老师麻烦您帮我看看这个应该怎么改呢？ @GhostScreaming

master_param_out已经和sliced_tensor share buffer了，再给它分配空间是不是有点问题？直接调master_param_out[info.idx]->data<float>()试试？

好像还是不行
@GhostScreaming

好像还是不行 @GhostScreaming

可以按下面的用法试下：
float* master_param_tmp = dev_ctx.template Alloc(master_param_out[info.idx]);
float* sliced_tensor_tmp = reinterpret_cast<float*>(sliced_tensor.data());
PADDLE_ENFORCE_EQ( master_param_tmp, sliced_tensor_tmp,
errors::InvalidArgument("Invalid master weight tensor pointer."));

好像还是不行 @GhostScreaming

可以按下面的用法试下： float* master_param_tmp = dev_ctx.template Alloc(master_param_out[info.idx]); float* sliced_tensor_tmp = reinterpret_cast<float*>(sliced_tensor.data()); PADDLE_ENFORCE_EQ( master_param_tmp, sliced_tensor_tmp, errors::InvalidArgument("Invalid master weight tensor pointer."));

解决了，谢谢老师！

GhostScreaming · 2023-08-22T07:54:45Z

paddle/phi/kernels/fusion/gpu/distributed_fused_lamb_init_kernel.cu

+  kernel->OutputAt(7).SetDataType(phi::DataType::FLOAT32);
+  kernel->OutputAt(8).SetDataType(phi::DataType::INT32);
+  kernel->OutputAt(9).SetDataType(phi::DataType::INT32);
+  kernel->OutputAt(10).SetDataType(phi::DataType::INT32);


param_out, grad_out的类型应该与Input的param一致，（可能为phi::DataType::FLOAT32或者phi::DataType::FLOAT16）master_param_out的类型应该为phi::DataType::FLOAT32

… and Input of

GhostScreaming

LGTM

GhostScreaming · 2023-08-30T12:07:33Z

paddle/phi/kernels/fusion/gpu/distributed_fused_lamb_init_kernel.cu

+    std::vector<DenseTensor *> master_param_out,
+    std::vector<DenseTensor *> grad_out,
+    DenseTensor *global_scale,
+    DenseTensor *step) {


先这样吧，问题不大

zeroRains added 2 commits August 4, 2023 10:14

add the .h and .cc file

56740c8

LanchCastKernel need to do

542da90

paddle-bot bot added contributor External developers status: proposed labels Aug 5, 2023

zeroRains added 2 commits August 5, 2023 08:00

have some little bug

02fea97

fix pre-commit

ed4d115

luotao1 added the HappyOpenSource 快乐开源活动issue与PR label Aug 7, 2023

paddle-bot bot removed the status: proposed label Aug 7, 2023

luotao1 assigned hitywt, luotao1 and GhostScreaming Aug 7, 2023

zeroRains added 3 commits August 7, 2023 04:35

fix the bug in matching function

c4d5d51

change the output type

161048e

change point to instance

a07ae08

luotao1 added HappyOpenSource Pro 进阶版快乐开源活动，更具挑战性的任务 and removed HappyOpenSource 快乐开源活动issue与PR labels Aug 7, 2023

luotao1 mentioned this pull request Aug 7, 2023

[Fluid] Migrate Fluid Distributed Kernels to PHI #55716

Closed

zeroRains added 3 commits August 7, 2023 08:32

fix the bug in register and it can make successfully

db5d13e

remove a mutable_data

dc34413

final change the register name

8d59a3e

hitywt reviewed Aug 9, 2023

View reviewed changes

zeroRains added 5 commits August 12, 2023 10:36

fix the segmentation fault bug

f99e1a2

remove some LOG_FIRST_N which use to debug and meet a new bug that th…

6f29f47

…e output shape problem

fix the bug in shape but the max_diff is large than atol

9467d45

fix the final bug in accuracy

c4d0026

register the output dtype

9b1e497

GhostScreaming reviewed Aug 22, 2023

View reviewed changes

zeroRains added 3 commits August 22, 2023 08:57

fix the output datatype

9530b2c

remove cast_with_ptr.h and k samethe data tyamong param_out, grad_out…

d30f7c6

… and Input of

modify the .cu

62c4c63

AndSonder mentioned this pull request Aug 22, 2023

Remove need_move_to_phi #56371

Merged

zeroRains added 2 commits August 23, 2023 03:08

remove mutable_data

71b67dc

fix the format and the enforce

5eb780e

GhostScreaming approved these changes Aug 30, 2023

View reviewed changes

luotao1 approved these changes Aug 31, 2023

View reviewed changes

luotao1 merged commit 0bc369e into PaddlePaddle:develop Aug 31, 2023

BeingGod pushed a commit to BeingGod/Paddle that referenced this pull request Sep 9, 2023

[Fluid] Move distributed_fused_lamb_init to phi (PaddlePaddle#55993)

ff5b555

zeroRains deleted the distributed branch September 25, 2023 08:34

AndSonder mentioned this pull request Oct 3, 2023

静态图执行器预分析性能优化 #55299

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fluid] Move distributed_fused_lamb_init to phi #55993

[Fluid] Move distributed_fused_lamb_init to phi #55993

zeroRains commented Aug 5, 2023

paddle-bot bot commented Aug 5, 2023

zeroRains commented Aug 8, 2023 •

edited

Loading

hitywt Aug 9, 2023

GhostScreaming Aug 24, 2023

zeroRains Aug 24, 2023

zeroRains Aug 29, 2023

GhostScreaming Aug 30, 2023

hitywt Aug 9, 2023

zeroRains commented Aug 18, 2023

GhostScreaming Aug 22, 2023

zeroRains Aug 22, 2023

GhostScreaming Aug 22, 2023

zeroRains Aug 22, 2023

GhostScreaming Aug 22, 2023

zeroRains Aug 22, 2023 •

edited

Loading

GhostScreaming Aug 24, 2023

zeroRains Aug 24, 2023 •

edited

Loading

hitywt Aug 29, 2023

zeroRains Aug 29, 2023

GhostScreaming Aug 22, 2023

zeroRains Aug 22, 2023

GhostScreaming left a comment

GhostScreaming Aug 30, 2023

[Fluid] Move distributed_fused_lamb_init to phi #55993

[Fluid] Move distributed_fused_lamb_init to phi #55993

Conversation

zeroRains commented Aug 5, 2023

PR types

PR changes

Description

paddle-bot bot commented Aug 5, 2023

zeroRains commented Aug 8, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zeroRains commented Aug 18, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zeroRains Aug 22, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zeroRains Aug 24, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GhostScreaming left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zeroRains commented Aug 8, 2023 •

edited

Loading

zeroRains Aug 22, 2023 •

edited

Loading

zeroRains Aug 24, 2023 •

edited

Loading