Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[xdoctest][task 248-249,266-267,269] reformat example code with google style in incubate/distributed/fleet/*,incubate/nn/layer/* #56772

Merged
merged 4 commits into from
Sep 6, 2023
Merged

[xdoctest][task 248-249,266-267,269] reformat example code with google style in incubate/distributed/fleet/*,incubate/nn/layer/* #56772

merged 4 commits into from
Sep 6, 2023

Conversation

ooooo-create
Copy link
Contributor

PR types

Others

PR changes

Others

Description

修改如下文件的示例代码，使其通过 xdoctest 检查：

  • python/paddle/incubate/distributed/fleet/fleet_util.py
  • python/paddle/incubate/distributed/fleet/parameter_server/ir/ps_dispatche.py
  • python/paddle/incubate/nn/layer/fused_ec_moe.py
  • python/paddle/incubate/nn/layer/fused_linear.py
  • python/paddle/incubate/nn/layer/io.py

预览：

Related links

@sunzhongkai588 @SigureMo @megemini

@paddle-bot
Copy link

paddle-bot bot commented Aug 29, 2023

你的PR提交成功，感谢你对开源项目的贡献!
请关注后续CI自动化测试结果，详情请参考Paddle-CI手册。
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

@paddle-bot paddle-bot bot added the contributor External developers label Aug 29, 2023
@luotao1 luotao1 added the HappyOpenSource Pro 进阶版快乐开源活动，更具挑战性的任务 label Aug 30, 2023
megemini
megemini reviewed Aug 30, 2023
View reviewed changes
Copy link
Contributor

@megemini megemini left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

# doctest: +REQUIRES(env:DISTRIBUTED) 并不是直接跳过，改的时候还是要注意一下。

python/paddle/incubate/distributed/fleet/fleet_util.py Outdated
>>> # doctest: +REQUIRES(env:DISTRIBUTED)
>>> from paddle.incubate.distributed.fleet.fleet_util import FleetUtil
>>> fleet_util = FleetUtil()
>>> fleet_util.load_fleet_model("hdfs:/my/model/path", table_id=1)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里应该是 load_fleet_model_one_table ，另外，参数也要改一下 ～

python/paddle/incubate/distributed/fleet/fleet_util.py Outdated
Comment on lines 424 to 430
>>> fleet_util.write_model_donefile(output_path="hdfs:/my/output",
... model_path="hdfs:/my/model",
... day=20190723,
... pass_id=66,
... xbox_base_key=int(time.time()),
... hadoop_fs_name="hdfs://xxx",
... hadoop_fs_ugi="user,passwd")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

参数不对 ～ 可以比对方法定义或者单测改一下。

python/paddle/incubate/distributed/fleet/fleet_util.py Outdated
Comment on lines 527 to 536
>>> fleet_util.write_xbox_donefile(
... output_path="hdfs:/my/output/",
... model_path="hdfs:/my/output/20190722/01",
... day=20190722,
... pass_id=1,
... xbox_base_key=int(time.time()),
... data_path="hdfs:/my/data/",
... hadoop_fs_name="hdfs://xxx",
... hadoop_fs_ugi="user,passwd",
... monitor_data={})
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

参数不对～

python/paddle/incubate/distributed/fleet/fleet_util.py Outdated
>>> # doctest: +REQUIRES(env:DISTRIBUTED)
>>> from paddle.incubate.distributed.fleet.fleet_util import FleetUtil
>>> fleet_util = FleetUtil()
>>> fleet_util.save_batch_model("hdfs:/my/path", 20190722, 88)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

方法名不对 ～

python/paddle/incubate/distributed/fleet/fleet_util.py Outdated
>>> # doctest: +REQUIRES(env:DISTRIBUTED)
>>> from paddle.incubate.distributed.fleet.fleet_util import FleetUtil
>>> fleet_util = FleetUtil()
>>> fleet_util.save_xbox_base_model("hdfs:/my/path", 20190722, 88)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

参数不对～

python/paddle/incubate/distributed/fleet/fleet_util.py Outdated
Comment on lines 1987 to 1990
>>> fleet_util.write_model_donefile(output_path="hdfs:/my/output",
... model_path="hdfs:/my/model",
... day=20190723,
... pass_id=66,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

参数不对 ～

python/paddle/incubate/distributed/fleet/fleet_util.py Outdated
Comment on lines 2087 to 2094
>>> fleet_util.write_xbox_donefile(
... output_path="hdfs:/my/output/",
... model_path="hdfs:/my/output/20190722/01",
... day=20190722,
... pass_id=1,
... xbox_base_key=int(time.time()),
... data_path="hdfs:/my/data/",
... monitor_data={})
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

参数不对 ～

python/paddle/incubate/distributed/fleet/parameter_server/ir/ps_dispatcher.py Outdated
Comment on lines 58 to 64
>>> from paddle.incubate.distributed.fleet.parameter_server.ir.ps_dispatcher import RoundRobin

rr = RoundRobin(pserver_endpoints)
rr.dispatch(vars)
>>> pserver_endpoints = ["127.0.0.1:6007", "127.0.0.1:6008"]
>>> vars = ["var1","var2","var3","var4","var5"]

>>> rr = RoundRobin(pserver_endpoints)
>>> rr.dispatch(vars)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里不是 HashName 的示例～ 可以参考单测重新写一下 ～

python/paddle/incubate/nn/layer/fused_linear.py Outdated
>>> x = paddle.randn([3, 4])
>>> linear = FusedLinear(4, 5)
>>> y = linear(x)
>>> print(y.shape) # [3, 5]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

输出要写一下 ～

python/paddle/incubate/nn/layer/io.py Outdated
Comment on lines 58 to 82
>>> import paddle.fluid as fluid
>>> from paddle.incubate.nn.layer.io import ListenAndServ
>>> import paddle
>>> paddle.enable_static()
>>> place = fluid.CPUPlace()
>>> main = fluid.Program()
>>> with fluid.program_guard(main):
... serv = ListenAndServ(
... "127.0.0.1:6", ["X"], optimizer_mode=False)
... with serv.do():
... out_var = main.global_block().create_var(
... name="scale_0.tmp_0",
... psersistable=True,
... dtype="float32",
... shape=[32, 32],
... )
... x = paddle.static.data(
... shape=[32, 32],
... dtype='float32',
... name="X")
... paddle.nn.initializer.Constant(value=1.0)(x, main.global_block())
... paddle.scale(x=x, scale=10.0, out=out_var)

>>> exe = fluid.Executor(place)
>>> exe.run(main)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fluid 改一下 ～

@ooooo-create
Copy link
Contributor Author

单测文件是test\legacy_test\test_fleet_util.py吗，里面几乎找不到之前定义的函数😕，不知道参数应该是什么样的，那变量问题我都skip?

@megemini
Copy link
Contributor

单测文件是test\legacy_test\test_fleet_util.py吗，里面几乎找不到之前定义的函数😕，不知道参数应该是什么样的，那变量问题我都skip?

参数不对的，可以看一下方法的定义 ～

变量问题的，如果不清楚怎么定义，也没有单测参考，skip 并说明依赖自定义变量就行 ～

SigureMo
SigureMo reviewed Sep 1, 2023
View reviewed changes
python/paddle/incubate/nn/layer/io.py Outdated
@@ -55,7 +55,7 @@ class ListenAndServ:
Examples:
.. code-block:: python

>>> import paddle.fluid as fluid
>>> import paddle.base as fluid
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

现在不可以这样用的，还没有 paddle.base 呢，paddle.base 得 rename 之后才会有

下面的 API 可分别尝试使用

fluid.CPUPlace -> paddle.CPUPlace
fluid.Program -> paddle.static.Program
fluid.program_guard -> paddle.static.program_guard

SigureMo
SigureMo reviewed Sep 6, 2023
View reviewed changes
python/paddle/incubate/nn/layer/io.py Outdated

exe = fluid.Executor(place)
exe.run(main)
>>> # doctest: +SKIP('ValueError: Operator "listen_and_serv" has not been registered.')
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

我这边跑了下是可以跑的，也许是需要编译 WITH_DISTRIBUTED？使用 # doctest: +REQUIRES(env:DISTRIBUTED) 吧

SigureMo
SigureMo approved these changes Sep 6, 2023
View reviewed changes
Copy link
Member

@SigureMo SigureMo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTMeow 🐾

@luotao1 luotao1 merged commit c958ba7 into PaddlePaddle:develop Sep 6, 2023
@luotao1 luotao1 mentioned this pull request Sep 6, 2023
Closed
BeingGod pushed a commit to BeingGod/Paddle that referenced this pull request Sep 9, 2023
@ooooo-create @BeingGod
[xdoctest][task 248-249,266-267,269] reformat example code with googl…
3fb2983 
…e style in `incubate/distributed/fleet/*`,`incubate/nn/layer/*` (PaddlePaddle#56772)

* [Doctest]fix No.248-249,266-267,269, test=docs_preview

* fix style

* fix

* add env:DISTRIBUTED
@ooooo-create ooooo-create deleted the ooooo/xdoctest249 branch September 23, 2023 05:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@megemini megemini megemini left review comments

@SigureMo SigureMo SigureMo approved these changes

Assignees

@luotao1 luotao1

@SigureMo SigureMo

@sunzhongkai588 sunzhongkai588

Labels
contributor External developers HappyOpenSource Pro 进阶版快乐开源活动，更具挑战性的任务
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@ooooo-create @megemini @SigureMo @luotao1 @sunzhongkai588