Fix multi discrete

[stefanpantic]

What do these changes do?

• Fixes a bug with calculating log_rhos in VTrace.
• Arguments for the function get_log_rhos were switched and instead of calculating target_action_log_probs - behaviour_action_log_probs, the reverse was being done, behaviour_action_log_probs - target_action_log_probs. I swithed the arguments and also replaced the elementwise list comprehension with an equivalent tensor operation.
• Reverts Revert "[wingman -> rllib] IMPALA MultiDiscrete changes (#3967)" #4332

Related issue number

• Fixes [rllib] IMPALA can't converge on cluster with Ray 0.6.4 #4329

[AmplabJenkins]

Can one of the admins verify this patch?

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/12795/
Test FAILed.

[ericl]

Looks good.

• IMPALA_BreakoutNoFrameskip-v4_0_env=BreakoutNoFrameskip-v4: RUNNING, [33 CPUs, 1 GPUs], [pid=35223], 1029 s, 82 iter, 5238000 ts, 326 rew

This is exactly in line with the reference result on rl-experiments repo.

[ericl]

Pushed a lint fix.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/12830/
Test FAILed.