Using "env.render(mode='rgb_array')" and "env.render(mode='depth_array')" simultaneously is strange in Mujoco environment

[ryota-mo]

When I use two different size of env.render(mode='rgb_array') and env.render(mode='depth_array' , such as (width, height) = (64, 64) in depth_array and (256, 256) in rgb_array, output np.array is too strange. However, When I use both render as rgb_array, I can get expected images.

For example,

from PIL import Image
import gym


def main():
    env = gym.make("HalfCheetah-v2")
    env.reset()
    for i in range(2):
        env.step([0]*6)
        depth_array_small = env.render(
            mode="depth_array",
            width=64,
            height=64,
        )
        rgb_array_large = env.render(
            mode="rgb_array",
            width=256,
            height=256,
        )
        Image.fromarray(rgb_array_large).save(f"rgb_array_large_{i}.png")


if __name__ == "__main__":
    main()

There are output files.

rgb_array_large_0.png	rgb_array_large_1.png

When I change the order of rgb_array_large and rgb_array_small, I get these pictures.

rgb_array_large_change_order_0.png	rgb_array_large_change_order_1.png

When I change the last code to the next one, I can get exact images.

from PIL import Image
import gym


def main():
    env = gym.make("HalfCheetah-v2")
    env.reset()
    for i in range(2):
        env.step([0]*6)
        depth_array_small = env.render(
            mode="depth_array",
            width=64,
            height=64,
        )
        env.viewer.update_offscreen_size(width=256, height=256)
        rgb_array_large = env.render(
            mode="rgb_array",
            width=256,
            height=256,
        )
        env.viewer.update_offscreen_size(width=64, height=64)

        Image.fromarray(rgb_array_large).save(
            f"rgb_array_large_update_offscreen_size_{i}.png")


if __name__ == "__main__":
    main()

rgb_array_large_update_offscreen_size_0.png	rgb_array_large_update_offscreen_size_1.png

I think https://github.com/openai/mujoco-py/blob/master/mujoco_py/mjrendercontext.pyx#L142 makes these strange behaviors. Updating the size of images occurs only when the sizes of images are smaller than the old image (render).

I think updating MujocoEnv.render is needed. Please give it some consideration.

[TigerStone93]

Is there any update?

[jkterry1]

PR #2762 is about to be merged, introducing V4 MuJoCo environments using new bindings and a dramatically newer version of the engine. If this issue still persists with the V4 ones, please create a new issue for it.