[Dy2St][PIR] Hold backward program in GradNode #63694

gouzil · 2024-04-19T07:48:31Z

PR Category

Execute Infrastructure

PR Types

Bug fixes

Description

将forward_program和 backward_program 存入 attrs 保证反向 program 的 shared_ptr 能够 +1 不会在跑完正向后被释放

NOTE: 需要使用 pybind11 的 2.12 版本

相关链接:

paddle-bot · 2024-04-19T07:48:36Z

你的PR提交成功，感谢你对开源项目的贡献!
请关注后续CI自动化测试结果，详情请参考Paddle-CI手册。
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

SigureMo · 2024-04-19T07:51:42Z

paddle/fluid/pybind/op_function_common.cc

+  std::shared_ptr<::pir::Program> program =
+      reinterpret_cast<std::shared_ptr<::pir::Program>&>(vh[0]);


能拿出来吗？

SigureMo · 2024-04-19T07:52:01Z

paddle/fluid/pybind/op_function_common.cc

+      reinterpret_cast<std::shared_ptr<::pir::Program>&>(vh[0]);
+  // TODO(gouzil): 试一下pybind11能不能使用智能指针作为参数
+  // pir::IrMapping mapper;
+  attrs[key] = program.get();


为什么要转回裸指针？

SigureMo · 2024-04-19T07:52:46Z

paddle/fluid/pybind/op_function_common.cc

+  void** vh = inst->simple_layout ? inst->simple_value_holder
+                                  : &inst->nonsimple.values_and_holders[0];
+
+  ::pybind11::handle(obj).inc_ref();


在这 inc，在哪 dec？

…IRProgram

SigureMo

其余 LGTM，完美

SigureMo · 2024-04-24T11:40:21Z

paddle/fluid/eager/to_static/run_program_op_node.h


  // Clear out and middles to avoid hold memory until backward finish.
  out.clear();
  middles.clear();
+  VLOG(1) << "out and middles clear end";


这个忘清了？

SigureMo · 2024-04-24T11:41:33Z

paddle/fluid/framework/type_defs.h

@@ -40,6 +41,7 @@ class InferShapeContext;
 class InferVarTypeContext;
 class VarDesc;
 class BlockDesc;
+class ProgramDesc;


这个有必要嘛？下面只添加了 pir::Program，为啥这里要前向声明老 IR ProgramDesc？

SigureMo · 2024-04-24T11:41:56Z

paddle/fluid/pybind/op_function_common.cc

@@ -998,6 +1010,7 @@ void ConstructAttrMapForRunProgram(
                        attr_end));

  PyObject* obj = nullptr;
+  attrs["testkey"] = std::string("testvalue");


忘清了？

SigureMo

LGTMeow

…IRProgram

…IRProgram # Conflicts: # python/paddle/jit/pir_translated_layer.py

SigureMo · 2024-05-06T07:15:23Z

test/cpp/prim/CMakeLists.txt

@@ -23,7 +23,7 @@ if(NOT (NOT WITH_PYTHON AND ON_INFER))
  cc_library(init_env_utils SRCS init_env_utils.cc)
  target_compile_definitions(init_env_utils PUBLIC PADDLE_DLL_EXPORT)

-  paddle_test(test_comp_eager SRCS test_eager_prim.cc DEPS init_env_utils)
+  paddle_test(test_comp_eager SRCS test_eager_prim.cc init_env_utils.cc)


#56691 看样子是为了减小单测体积才这么搞的，这样改是不是又变回去了？

是的，但是它们之间目前存在重复依赖的问题，会导致windows LNK2005错误，所以就先直接不拆分了

重复依赖是指？如果这样的话，24 行是不是不需要了？还是说后续有计划优化这里呢？

重复依赖可以看13c89cf，以及它的PR-CI-Windows-OPENBLAS，大概就是它俩都依赖了phi导致重复依赖了。也参考了一些：paddle_test 的文档还是没能拆出来。最好的情况当然是拆分它，缩小体积，后续优化吧。

最好的情况当然是拆分它，缩小体积，后续优化吧。

ok

SigureMo

LGTMeow

SigureMo · 2024-05-06T08:21:17Z

test/cpp/prim/CMakeLists.txt

@@ -23,7 +23,7 @@ if(NOT (NOT WITH_PYTHON AND ON_INFER))
  cc_library(init_env_utils SRCS init_env_utils.cc)
  target_compile_definitions(init_env_utils PUBLIC PADDLE_DLL_EXPORT)

-  paddle_test(test_comp_eager SRCS test_eager_prim.cc DEPS init_env_utils)
+  paddle_test(test_comp_eager SRCS test_eager_prim.cc init_env_utils.cc)


最好的情况当然是拆分它，缩小体积，后续优化吧。

ok

Co-authored-by: xiongkun <[email protected]> Co-authored-by: Nyakku Shigure <[email protected]>

add int4_1 int4_2 FLAGS_logging_pir_py_code (PaddlePaddle#63981) * FLAGS_logging_pir_py_code * FLAGS_logging_pir_py_code_dir --------- Co-authored-by: jiahy0825 <[email protected]> [Cleanup] Remove Flake8 config in `.editorconfig` (PaddlePaddle#64027) 【PIR Dist Op Reg No.19】 reg pull_box_sparse (PaddlePaddle#62982) * fix * fix * fix * fix * fix * fix * add test * add * fix * fix * add out * fix * codestyle * fix * fix backward * merge [Dy2St][PIR] Hold backward program in GradNode (PaddlePaddle#63694) Co-authored-by: xiongkun <[email protected]> Co-authored-by: Nyakku Shigure <[email protected]> split test.cmake: add new test_cases.cmake (PaddlePaddle#64007) [PIR] Support sparse_slice and sparse_sum in pt (PaddlePaddle#64009) * support sparse_slice and sparse_sum in pt * support sparse_slice and sparse_sum in pt * support sparse_slice and sparse_sum in pt option for WITH_CPP_TEST (PaddlePaddle#63896) * option for WITH_CPP_TEST * fix * Fix * Fix [PIR] Fix `attributes_num` of `SliceArrayOp` (PaddlePaddle#64013) [Dy2St] Use `full_graph=True` outside dy2st uts (part1) (PaddlePaddle#64058) [Dy2St] Use `full_graph=True` outside dy2st uts (part2) (PaddlePaddle#64059) fix typo (PaddlePaddle#64060) Co-authored-by: jiahy0825 <[email protected]> update (PaddlePaddle#64042) Replace paddle/fluid/platform/device/gpu/gpu_dnn.h (PaddlePaddle#63819) * Fix * Fix * Fix Clean lookup_table_v2_op.h lookup_table_v2_op.cu (PaddlePaddle#64020) * Fix * ci refine GetTensorListFromArgs (PaddlePaddle#64045) Revert "【Hackathon 6th Fundable Projects 3 No.60】Remove fluid operator chunk_…" (PaddlePaddle#64050) This reverts commit 88b1a6e. [Prim][PIR] support floor_divide op forward in prim pir (PaddlePaddle#64023) * floor-div-dev * update test [CINN] Reconstruct shape_analysis (PaddlePaddle#63790) * reconstruct shape_analysis * fix input value shape infer * fix merge bugs * fix concat and gather op InferSymbolicShape * fix merge bug * fix value_to_shape_or_data hash error and add some checks * fix set shape for null value * fix group op lazy infer * add IsStaticShape check * fix merge bug * support static dim check and set for VectorType * change auto to detail type [XPU] fix bugs in processing of attention_mask and fix_seed_offset on XPU (PaddlePaddle#64003) * [XPU] fix segmentfault caused by setting fix_seed_offset on XPU * cast attention_mask to float32 when necessary fix merge bug (PaddlePaddle#64069) 【Fix PIR Unittest No.125、147、481】Fix some 0D uts in PIR mode (part1) (PaddlePaddle#64064) [Prim][VJP]support autogen to remove unused composite in .yaml (PaddlePaddle#64054) * support autogen to remove unused composite in .yaml * fix bug [PIR] Fix typo `set_pit_tests_properties` -> `set_pir_tests_properties` (PaddlePaddle#64063) [Dy2St] Use `full_graph=True` outside dy2st uts (part3) (PaddlePaddle#64066) [PIR save/load] Open more tests for paddle.save and paddle.load (PaddlePaddle#64044) * open more tests for paddle.save and paddle.load * fix API Improvement for paddle.nn.functional.group_norm and paddle.nn.GroupNorm (PaddlePaddle#63881) * update group_norm * update trt plugin * update trt plugin * fix trt plugin * fix trt plugin * fix test * fix test * fix ci windows inference * update kernel function names and add v2 test * fix * fix fp16 test Revert "【Hackathon 6th Fundable Projects 3 No.81】Remove fluid operators ctc_a…" (PaddlePaddle#64049) This reverts commit 2134ead. Clean paddle/fluid/operators/fused/attention_layer_norm.h (PaddlePaddle#64051) * Fix * Fix Replace operators::math to phi::math in fluid/operators (PaddlePaddle#63854) [CINN]Clean usless loop_reorder_aligment tactic (PaddlePaddle#63998) * [CINN]Clean usless loop_reorder_aligment tactic * fix source 【Hackathon 6th Fundable Projects 3 No.396】fluid operator yolo_box_head (PaddlePaddle#63783) * Fix * Fix * Fix * Fix * Fix 【Hackathon 6th Fundable Projects 3 No.240】fluid operator moe (PaddlePaddle#63929) 【Hackathon 6th Fundable Projects 3 No.82】fluid operator cudnn_lstm (PaddlePaddle#63936) * Fix * Fix * Fix * Fix [CINN] Remove useless log (PaddlePaddle#64052) [pir_save_load] add pir for test_jit_save_load.py (PaddlePaddle#63958) * add jit load.train * modify backward program lost * modify * combine eval and train * modify 8 case of jit.save.load * modify jit_save_load case * rename jit_save_load * change name all * modify timeout * modify new case * modify TestJitSaveLoadMultiMethods * modify cpu tensor no holder bug Flashattention support qkvpacked and varlen (PaddlePaddle#63289) * Flashattention support qkvpacked and varlen * fix codestyle * fix codestyle * FlashAttention kvReduceGQA Performance Optimization * Fix problem with windows * code clean * update third_party/flashattn * update errormsg and docs * update api * update doc * update doctest * update doc, test=document_fix * update doc, test=document_fix * Update python/paddle/nn/functional/flash_attention.py Co-authored-by: zachary sun <[email protected]> * Update python/paddle/nn/functional/flash_attention.py Co-authored-by: zachary sun <[email protected]> * update doc --------- Co-authored-by: zachary sun <[email protected]> 【PIR Dist Op Reg No.20】 reg global_gather (PaddlePaddle#63867) * reg global_gather * reg global_gather * reg_global_gather * fix * fix * fix * fix conflict * fix conflict * Update ops_api_gen.py * Update ops_api_gen.py Fix backward program kwargs error when process inplace value (PaddlePaddle#63939) 【Hackathon 6th No.35】support kwargs for recompute when use_reentrant == True fix (PaddlePaddle#63880) * support kwargs for recompute when use_reentrant == True * recover third party merge main lint delete printf change flash attn version

Co-authored-by: xiongkun <[email protected]> Co-authored-by: Nyakku Shigure <[email protected]>

gouzil added 4 commits April 18, 2024 22:13

add log

3223492

add log

c1398d1

add log

2a7f67d

add handle

733ac03

paddle-bot bot added the contributor External developers label Apr 19, 2024

gouzil requested a review from SigureMo April 19, 2024 07:48

SigureMo reviewed Apr 19, 2024

View reviewed changes

gouzil added 3 commits April 19, 2024 17:47

remove breakpoint

5c90eac

use std::shared_ptr<::pir::Program>>

64a56b0

Merge branch 'develop' of github.com:gouzil/Paddle into fix_backward_…

b48eaa0

…IRProgram

gouzil mentioned this pull request Apr 22, 2024

[cmake] update pybind11 v2.10.3 to v2.12.0 #63741

Merged

2 tasks

gouzil added 4 commits April 23, 2024 13:29

Merge branch 'develop' of github.com:gouzil/Paddle into fix_backward_…

5534c70

…IRProgram

copy PaddlePaddle#58180

8aba5ef

clean log and open test

23bbd58

Rollback execution sequence

31be3b9

SigureMo reviewed Apr 24, 2024

View reviewed changes

gouzil added 3 commits April 24, 2024 19:42

rm log

e005597

rm class ProgramDesc

6277bc2

rm test[key]

4972090

SigureMo previously approved these changes Apr 24, 2024

View reviewed changes

SigureMo changed the title ~~[WIP][Dy2St][PIR] Hold backward program in GradNode~~ [Dy2St][PIR] Hold backward program in GradNode Apr 24, 2024

rm include

4bcbcc6

gouzil dismissed SigureMo’s stale review via 4bcbcc6 April 24, 2024 15:54

SigureMo changed the title ~~[Dy2St][PIR] Hold backward program in GradNode~~ [WIP][Dy2St][PIR] Hold backward program in GradNode Apr 25, 2024

gouzil force-pushed the fix_backward_IRProgram branch from 5378583 to 4bcbcc6 Compare April 25, 2024 07:58

gouzil added 4 commits April 25, 2024 16:26

Merge branch 'develop' of github.com:gouzil/Paddle into fix_backward_…

f5c6e0b

…IRProgram

fix jit.load for pir

414fbda

[ci][test] ignore test

1e68e62

[ci][test] open CompareOperantsTest

cb6cdab

gouzil added 6 commits April 25, 2024 23:06

[ci][test] ignore hook_utils

a9946ee

[ci][test] ignore dygraph_functions and open hook_utils

d83476b

Merge branch 'develop' of github.com:gouzil/Paddle into fix_backward_…

0bf8a6a

…IRProgram # Conflicts: # python/paddle/jit/pir_translated_layer.py

fix test_comp_eager

462cdf7

rollback test_eager_prim.cc

c7e9825

fix

de20f12

gouzil changed the title ~~[WIP][Dy2St][PIR] Hold backward program in GradNode~~ [Dy2St][PIR] Hold backward program in GradNode May 6, 2024

gouzil requested a review from SigureMo May 6, 2024 07:08

SigureMo reviewed May 6, 2024

View reviewed changes

SigureMo approved these changes May 6, 2024

View reviewed changes

SigureMo merged commit 4cd5384 into PaddlePaddle:develop May 6, 2024
30 checks passed

SigureMo mentioned this pull request May 6, 2024

PIR 动转静理想态单测推全验证任务列表（二期）🥳 #60131

Closed

co63oc pushed a commit to co63oc/Paddle that referenced this pull request May 7, 2024

[Dy2St][PIR] Hold backward program in GradNode (PaddlePaddle#63694)

c32740e

Co-authored-by: xiongkun <[email protected]> Co-authored-by: Nyakku Shigure <[email protected]>

yinfan98 pushed a commit to yinfan98/Paddle that referenced this pull request May 7, 2024

[Dy2St][PIR] Hold backward program in GradNode (PaddlePaddle#63694)

f05e65c

Co-authored-by: xiongkun <[email protected]> Co-authored-by: Nyakku Shigure <[email protected]>

yinfan98 pushed a commit to yinfan98/Paddle that referenced this pull request May 7, 2024

[Dy2St][PIR] Hold backward program in GradNode (PaddlePaddle#63694)

1e8436d

Co-authored-by: xiongkun <[email protected]> Co-authored-by: Nyakku Shigure <[email protected]>

gouzil deleted the fix_backward_IRProgram branch May 9, 2024 00:37

co63oc pushed a commit to co63oc/Paddle that referenced this pull request May 10, 2024

[Dy2St][PIR] Hold backward program in GradNode (PaddlePaddle#63694)

d24270f

Co-authored-by: xiongkun <[email protected]> Co-authored-by: Nyakku Shigure <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Dy2St][PIR] Hold backward program in GradNode #63694

[Dy2St][PIR] Hold backward program in GradNode #63694

gouzil commented Apr 19, 2024 •

edited by SigureMo

Loading

paddle-bot bot commented Apr 19, 2024

SigureMo Apr 19, 2024

SigureMo Apr 19, 2024

SigureMo Apr 19, 2024

SigureMo left a comment

SigureMo Apr 24, 2024

SigureMo Apr 24, 2024

SigureMo Apr 24, 2024

gouzil Apr 24, 2024

SigureMo left a comment

SigureMo May 6, 2024

gouzil May 6, 2024

SigureMo May 6, 2024

gouzil May 6, 2024

SigureMo May 6, 2024

SigureMo left a comment

SigureMo May 6, 2024

		std::shared_ptr<::pir::Program> program =
		reinterpret_cast<std::shared_ptr<::pir::Program>&>(vh[0]);

[Dy2St][PIR] Hold backward program in GradNode #63694

[Dy2St][PIR] Hold backward program in GradNode #63694

Conversation

gouzil commented Apr 19, 2024 • edited by SigureMo Loading

PR Category

PR Types

Description

paddle-bot bot commented Apr 19, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SigureMo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SigureMo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SigureMo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gouzil commented Apr 19, 2024 •

edited by SigureMo

Loading