On custom ENVs

[alietestep]

hi
tanks a lot for sharing code,
i'm trying to make this working on custom ENvs (robotic simulators) and i edited network for supporting image states and every things sounds good except 2:

1- first problem arises just in utils/network.py/line 123 , where Actor calculates scaled gradient in code:
def train_step(...)
....
self.grads_scaled = list(map(lambda x: tf.divide(x, batch_size), self.grads))
i got unsupported operation NoneType/Int error and for couple of days i couldn't solve this and i just commented this line and used unscaled grads and it worked.
but i'm pretty sure some things wrong with this. ... .

2-with this training starts but no learning can be seen during training as:
`
loss 0.0001 avg_return=-1000.00 0% 1/1000000 ....
loss 0.0001 avg_return=-1000.00 0% 1/1000000 ....
loss 0.0001 avg_return=-1000.00 0% 3/1000000 ....
.....
loss 0.0001 avg_return=-1000.00 0% 2050/1000000 ....
loss 0.0001 avg_return=-1000.00 0% 2051/1000000 ....
......
loss 0.0001 avg_return=-1000.00 0% 5000/1000000 ....
loss 0.0001 avg_return=-1000.00 0% 5001/1000000 ....

`
from above result, model loss is same and near zero and average return not changing during training (may not learns).
i didn't continue training after 5000 steps for lack of confidence of learning and training in my particular ENV needs heavy resources.

so case 1 is OK? or not and how can i fix it?
so is it natural in case 2 and should i continue training?

so any help or suggestion can be great.
thanks a lot again ,
best regards.