Gluon Trainer does not handle non-deterministic parameter order for distributed training

[eric-haibin-lin]

Description

Currently gluon Trainer iterates over the parameter dict and assign indices for multi-machine training. The index is used to identify the gradient/parameters. This relies on a deterministic order of param dict iteration and deterministic order of parameter creation. However, that is not be true if the user's code defines parameters in a random order (e.g. https://github.com/dmlc/gluon-nlp/blob/v0.9.x/src/gluonnlp/model/attention_cell.py#L223)