Add Support Layer List to ASP #40253

mingxu1067 · 2022-03-08T05:58:42Z

PR types

New features

PR changes

APIs

Describe

Added a function to allow user to add their own layer and correspond pruning way to ASP support.
Changed naming rule of asp masks to make it clear.
Usage example

import numpy as np
import paddle
from paddle.static import sparsity

paddle.enable_static()

def special_pruning(tensor, m, n, mask_algo, param_name):
    static_tensor = np.random.rand(*tensor.shape).astype(np.float32)
    static_tensor_mask = np.random.rand(*tensor.shape).astype(np.float32)
    return static_tensor, static_tensor_mask

main_program = paddle.static.Program()
startup_program = paddle.static.Program()

with paddle.static.program_guard(main_program, startup_program):
    input_data = paddle.static.data(name='data', shape=[None, 128])
    label = paddle.static.data(name='label', shape=[None, 10])
    hidden = paddle.static.nn.fc(x=input_data, num_flatten_dims=-1, 
                                 size=32, activation=None, name="need_special_prune")
    hidden = paddle.static.nn.fc(x=hidden, num_flatten_dims=-1, size=32, activation=None)
    prob = paddle.static.nn.fc(x=hidden, num_flatten_dims=-1, size=10, activation=None)
    loss = paddle.mean(paddle.nn.functional.square_error_cost(prob, label))


sparsity.add_supported_layer("need_special_prune", special_pruning)

place = paddle.CPUPlace()
exe = paddle.static.Executor(place)
exe.run(startup_program)

sparsity.prune_model(main_program, mask_algo="mask_1d", with_mask=False)

1. Added a function to allow user to add their own layer and correspond pruning way to ASP support. 2. Changed naming rule to asp masks.

paddle-bot-old · 2022-03-08T05:58:46Z

Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

Xreki · 2022-03-14T02:57:02Z

python/paddle/fluid/contrib/sparsity/__init__.py

-# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
-# Copyright (c) 2021 NVIDIA Corporation.  All rights reserved.
+# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
+# Copyright (c) 2022 NVIDIA Corporation.  All rights reserved.


该文件不是本PR新增，不用修改CopyRight年份，下同。这是误改？

Done, 2022 -> 2021

Xreki · 2022-03-14T02:58:00Z

python/paddle/fluid/contrib/sparsity/__init__.py


 __all__ = [
    'calculate_density', 'check_mask_1d', 'get_mask_1d', 'check_mask_2d',
    'get_mask_2d_greedy', 'get_mask_2d_best', 'create_mask', 'check_sparsity',
    'MaskAlgo', 'CheckMethod', 'decorate', 'prune_model', 'set_excluded_layers',
-    'reset_excluded_layers'
+    'reset_excluded_layers', 'add_supported_layer'


add_supported_layer、set_excluded_layers、reset_excluded_layers，这三个接口有点令人迷惑了。

這三個接口功能分別如下
add_supported_layer: 讓使用者增加自己客製化的nn.Layer的參數到ASP的支援清單，預設只會支援FC, Conv2D, Linear.
set_excluded_layers: 在支援的nn.Layer的參數中，針對特定的幾層不執行ASP, 比如一個網絡有三層FC, 但其中一層不需要ASP，怎可以使用此API來將該層排除
reset_excluded_layers: 將 set_excluded_layers 所添加的Layer全部清空。

Xreki · 2022-03-14T02:58:00Z

python/paddle/fluid/contrib/sparsity/__init__.py


 __all__ = [
    'calculate_density', 'check_mask_1d', 'get_mask_1d', 'check_mask_2d',
    'get_mask_2d_greedy', 'get_mask_2d_best', 'create_mask', 'check_sparsity',
    'MaskAlgo', 'CheckMethod', 'decorate', 'prune_model', 'set_excluded_layers',
-    'reset_excluded_layers'
+    'reset_excluded_layers', 'add_supported_layer'


add_supported_layer、set_excluded_layers、reset_excluded_layers，这三个接口有点令人迷惑了。

Xreki · 2022-03-14T03:00:11Z

python/paddle/fluid/contrib/sparsity/asp.py

@@ -293,7 +295,7 @@ class ASPHelper(object):
    """

    MASK_APPENDDED_NAME = '_asp_mask'
-    SUPPORTED_LAYERS = {'fc': 'w_0', 'linear': 'w_0', 'conv2d': 'w_0'}
+    PADDLE_WEIGHT_SUFFIX = "w_"


这样改的好处，是可以支持更多的layer，不限定于fc、linear、conv2d？

是的，此次改動將支援的Layer改為統一由supported_layer_list來管理

Xreki · 2022-03-14T03:00:11Z

python/paddle/fluid/contrib/sparsity/asp.py

@@ -293,7 +295,7 @@ class ASPHelper(object):
    """

    MASK_APPENDDED_NAME = '_asp_mask'
-    SUPPORTED_LAYERS = {'fc': 'w_0', 'linear': 'w_0', 'conv2d': 'w_0'}
+    PADDLE_WEIGHT_SUFFIX = "w_"


这样改的好处，是可以支持更多的layer，不限定于fc、linear、conv2d？

Xreki · 2022-03-14T04:32:20Z

python/paddle/fluid/contrib/sparsity/asp.py

@@ -384,7 +379,7 @@ def _get_mask_name(param_name):
        Returns:
            string: The mask name of :attr:`param_name`.
        """
-        return param_name + ASPHelper.MASK_APPENDDED_NAME
+        return param_name + "." + ASPHelper.MASK_APPENDDED_NAME


这里加了.，还需要保留_吗？

Done, remoe _.

Xreki · 2022-03-14T04:32:20Z

python/paddle/fluid/contrib/sparsity/asp.py

@@ -384,7 +379,7 @@ def _get_mask_name(param_name):
        Returns:
            string: The mask name of :attr:`param_name`.
        """
-        return param_name + ASPHelper.MASK_APPENDDED_NAME
+        return param_name + "." + ASPHelper.MASK_APPENDDED_NAME


这里加了.，还需要保留_吗？

Xreki · 2022-03-14T04:33:55Z

python/paddle/static/sparsity/__init__.py


 __all__ = [     #noqa
    'calculate_density',
    'decorate',
    'prune_model',
    'set_excluded_layers',
-    'reset_excluded_layers'
+    'reset_excluded_layers',
+    'add_supported_layer'


这里后续PR再加？add_supported_layer、set_excluded_layers、reset_excluded_layers，这三个接口有点令人迷惑了。

同上解釋後，是否還是要暫緩對外公開呢？

Xreki · 2022-03-14T04:33:55Z

python/paddle/static/sparsity/__init__.py


 __all__ = [     #noqa
    'calculate_density',
    'decorate',
    'prune_model',
    'set_excluded_layers',
-    'reset_excluded_layers'
+    'reset_excluded_layers',
+    'add_supported_layer'


这里后续PR再加？add_supported_layer、set_excluded_layers、reset_excluded_layers，这三个接口有点令人迷惑了。

同上解釋後，是否還是要暫緩對外公開呢？

如線下討論，暫緩公開

1. Modified year of copyright. 2. Changed asp mask name from _asp_mask to asp_mask.

1. Hide add_support_layer and wait API confirmation to release. 2. Changed year of copyright.

Xreki

LGTM

mingxu1067 added 2 commits March 8, 2022 10:48

Added Support List to ASP

906766b

1. Added a function to allow user to add their own layer and correspond pruning way to ASP support. 2. Changed naming rule to asp masks.

Fixed typo.

9f33e3c

mingxu1067 added the NVIDIA label Mar 8, 2022

Fix issues in test_fleet_sharding_meta_optimizer.py

9aa5b23

Xreki reviewed Mar 14, 2022

View reviewed changes

mingxu1067 added 2 commits March 14, 2022 13:42

Minor changes

f59b826

1. Modified year of copyright. 2. Changed asp mask name from _asp_mask to asp_mask.

Hide add_support_layer.

920111e

1. Hide add_support_layer and wait API confirmation to release. 2. Changed year of copyright.

mingxu1067 force-pushed the mingh/asp_add_support_layer branch from 44d0b07 to 920111e Compare March 14, 2022 06:27

Remove add_supported_list from import of static/sparsity.

5e78f1f

Xreki approved these changes Mar 15, 2022

View reviewed changes

Xreki requested a review from lanxianghit March 16, 2022 06:03

lanxianghit approved these changes Mar 16, 2022

View reviewed changes

TCChenlong approved these changes Mar 16, 2022

View reviewed changes

Xreki merged commit c040bbd into PaddlePaddle:develop Mar 16, 2022

liqitong-a pushed a commit to liqitong-a/Paddle that referenced this pull request Mar 17, 2022

Add Support Layer List to ASP (PaddlePaddle#40253)

e0d10d8

mingxu1067 mentioned this pull request Apr 1, 2022

Dynamic Graph Support to ASP #38517

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Support Layer List to ASP #40253

Add Support Layer List to ASP #40253

mingxu1067 commented Mar 8, 2022

paddle-bot-old bot commented Mar 8, 2022

Xreki Mar 14, 2022

mingxu1067 Mar 14, 2022

Xreki Mar 14, 2022

mingxu1067 Mar 14, 2022

Xreki Mar 14, 2022

mingxu1067 Mar 14, 2022

Xreki Mar 14, 2022

mingxu1067 Mar 14, 2022

Xreki Mar 14, 2022

mingxu1067 Mar 14, 2022

Xreki Mar 14, 2022

mingxu1067 Mar 14, 2022

Xreki Mar 14, 2022

mingxu1067 Mar 14, 2022

Xreki Mar 14, 2022

mingxu1067 Mar 14, 2022

Xreki Mar 14, 2022

mingxu1067 Mar 14, 2022

mingxu1067 Mar 14, 2022

Xreki left a comment

Add Support Layer List to ASP #40253

Add Support Layer List to ASP #40253

Conversation

mingxu1067 commented Mar 8, 2022

PR types

PR changes

Describe

paddle-bot-old bot commented Mar 8, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Xreki left a comment

Choose a reason for hiding this comment