Pytest fails randomly on VecNormalize tests

[araffin]

See https://github.com/DLR-RM/stable-baselines3/pull/53/checks?check_run_id=758580860
and https://gitlab.com/araffin/stable-baselines3/-/jobs/589877156

Since #50 was merged.

Traceback:

 2020-06-10T16:29:13.7480849Z =================================== FAILURES ===================================
2020-06-10T16:29:13.7482359Z ___________________________ test_sync_vec_normalize ____________________________
2020-06-10T16:29:13.7482761Z 
2020-06-10T16:29:13.7484480Z     def test_sync_vec_normalize():
2020-06-10T16:29:13.7484937Z         env = DummyVecEnv([make_env])
2020-06-10T16:29:13.7485318Z     
2020-06-10T16:29:13.7485746Z         assert unwrap_vec_normalize(env) is None
2020-06-10T16:29:13.7486142Z     
2020-06-10T16:29:13.7486588Z         env = VecNormalize(env, norm_obs=True, norm_reward=True, clip_obs=10., clip_reward=10.)
2020-06-10T16:29:13.7487019Z     
2020-06-10T16:29:13.7487477Z         assert isinstance(unwrap_vec_normalize(env), VecNormalize)
2020-06-10T16:29:13.7488198Z     
2020-06-10T16:29:13.7488624Z         env = VecFrameStack(env, 1)
2020-06-10T16:29:13.7489719Z     
2020-06-10T16:29:13.7490133Z         assert isinstance(unwrap_vec_normalize(env), VecNormalize)
2020-06-10T16:29:13.7490566Z     
2020-06-10T16:29:13.7490939Z         eval_env = DummyVecEnv([make_env])
2020-06-10T16:29:13.7491374Z         eval_env = VecNormalize(eval_env, training=False, norm_obs=True, norm_reward=True, clip_obs=10., clip_reward=10.)
2020-06-10T16:29:13.7491818Z         eval_env = VecFrameStack(eval_env, 1)
2020-06-10T16:29:13.7492237Z     
2020-06-10T16:29:13.7494143Z         env.reset()
2020-06-10T16:29:13.7494637Z         # Initialize running mean
2020-06-10T16:29:13.7495040Z         latest_reward = None
2020-06-10T16:29:13.7495440Z         for _ in range(100):
2020-06-10T16:29:13.7495931Z             _, latest_reward, _, _ = env.step([env.action_space.sample()])
2020-06-10T16:29:13.7496345Z     
2020-06-10T16:29:13.7496769Z         # Check that unnormalized reward is same as original reward
2020-06-10T16:29:13.7497213Z         original_latest_reward = env.get_original_reward()
2020-06-10T16:29:13.7497681Z         assert np.allclose(original_latest_reward, env.unnormalize_reward(latest_reward))
2020-06-10T16:29:13.7498090Z     
2020-06-10T16:29:13.7498462Z         obs = env.reset()
2020-06-10T16:29:13.7498870Z         dummy_rewards = np.random.rand(10)
2020-06-10T16:29:13.7499270Z         original_obs = env.get_original_obs()
2020-06-10T16:29:13.7499712Z         # Check that unnormalization works
2020-06-10T16:29:13.7500113Z >       assert np.allclose(original_obs, env.unnormalize_obs(obs))
2020-06-10T16:29:13.7500514Z E       assert False
2020-06-10T16:29:13.7501878Z E        +  where False = <function allclose at 0x7fc839db05f0>(array([[ 0.24142723,  0.97041893, -0.96644473]], dtype=float32), array([[-0.48370542,  0.9704189 , -0.96644476]]))
2020-06-10T16:29:13.7502405Z E        +    where <function allclose at 0x7fc839db05f0> = np.allclose
2020-06-10T16:29:13.7503978Z E        +    and   array([[-0.48370542,  0.9704189 , -0.96644476]]) = <bound method VecNormalize.unnormalize_obs of <stable_baselines3.common.vec_env.vec_normalize.VecNormalize object at 0x7fc814804d50>>(array([[10.        ,  3.0962322 , -0.74936515]], dtype=float32))
2020-06-10T16:29:13.7505725Z E        +      where <bound method VecNormalize.unnormalize_obs of <stable_baselines3.common.vec_env.vec_normalize.VecNormalize object at 0x7fc814804d50>> = <stable_baselines3.common.vec_env.vec_frame_stack.VecFrameStack object at 0x7fc814804450>.unnormalize_obs
2020-06-10T16:29:13.7506161Z 
2020-06-10T16:29:13.7506578Z tests/test_vec_normalize.py:167: AssertionError

[araffin]

unnormalization may not always work because of the clipping, fixed in #53