Fixing some parameters in NN

[ngkim823]

Anyone can help me how to fix the parameters in the specific blocks?

For example, I don't want to update parameters in the the below while other blocks run as usual.
(side): Dense(1178 -> 100, Activation(relu))

(0): Residual_Nick(
(main): Sequential(
(0): Conv2D(1 -> 1, kernel_size=(2, 2), stride=(1, 1), padding=(1, 1), Activation(relu))
(1): AvgPool2D(size=(2, 2), stride=(1, 1), padding=(0, 0), ceil_mode=False, global_pool=False, pool_type=avg, layout=NCHW)
(2): Conv2D(1 -> 1, kernel_size=(2, 2), stride=(1, 1), Activation(relu))
(3): AvgPool2D(size=(2, 2), stride=(1, 1), padding=(0, 0), ceil_mode=False, global_pool=False, pool_type=avg, layout=NCHW)
(4): Dense(1044 -> 100, Activation(relu))
(5): Dense(100 -> 256, Activation(relu))
(6): Dense(256 -> 100, Activation(relu))
)
(side): Dense(1178 -> 100, Activation(relu))
)

[leezu]

You can collect all parameters of the Block with collect_params() and then set p.grad_req = 'null' for every parameter p that you want to fix / freeze.

https://github.com/apache/incubator-mxnet/blob/25ab7a61bfc59d2968c447a99427f29ff5158cf0/python/mxnet/gluon/parameter.py#L95-L98

[ngkim823]

Thanks a million, it works.
However, unlike my expectation, the adaptation of the remaining block seems to be a little bit slow.
Is there any side-effect of fixing some parts of the parameters?

[leezu]

What kind of side-effect do you mean?

[ngkim823]

Hi Leezu,

Thanks for the help.
I mean slow adaptation of the remaining blocks.
For example, when the parameters are fixed for the "side" block of Dense network, it seems the parameter update for "main" block of Conv & Dense block seems to be slow.

(0): Residual_Nick(
(main): Sequential(
(0): Conv2D(1 -> 1, kernel_size=(2, 2), stride=(1, 1), padding=(1, 1), Activation(relu))
(1): AvgPool2D(size=(2, 2), stride=(1, 1), padding=(0, 0), ceil_mode=False, global_pool=False, pool_type=avg, layout=NCHW)
(2): Conv2D(1 -> 1, kernel_size=(2, 2), stride=(1, 1), Activation(relu))
(3): AvgPool2D(size=(2, 2), stride=(1, 1), padding=(0, 0), ceil_mode=False, global_pool=False, pool_type=avg, layout=NCHW)
(4): Dense(1044 -> 100, Activation(relu))
(5): Dense(100 -> 256, Activation(relu))
(6): Dense(256 -> 100, Activation(relu))
)
(side): Dense(1178 -> 100, Activation(relu))
)