MetaDrive: Rendering images into the 3D environment

[alexander-toch]

Hi,

First of all, thanks for this great project!

I'm using MetaDrive for robustness testing of DNN-based lane detection algorithms.
I would need to place an image statically onto the road for an attack evaluation. The image should then also be visible in the RGB Camera so that my inference pipeline can pick it up.

Is it possible to do this with existing functions in the project?
Or do I need to create a Panda3D texture from the image (similar to here) and find a way to render it into the terrain?

Thanks for any tips!

[QuanyiLi]

Ahh, I see. That's interesting. First, You need to modify this function:

metadrive/metadrive/engine/core/terrain.py

Line 590 in 233a3a1

def get_terrain_semantics(self, center_point):

You have to put a new semantic class to the generated semantic_map, which tells the terrain about which texture to place at a given place. Then, you need to input your new texture to the shader program so it can be accessed in terrain.frag.glsl. You also new to modify this shader program by

1. declaring new textures
2. modify the semantic parse code so your new texture is selected when the semantic class is your target texture class. Just modify here:

  if ((attri.r > 0.01) && (terrain_uv.x>=r_min) && (terrain_uv.y >= r_min) && (terrain_uv.x<=r_max) && (terrain_uv.y<=r_max)){
    float value = attri.r; // Assuming it's a red channel texture
    if (value < 0.11) {
        // yellow
        diffuse=texture(yellow_tex, terrain_uv * road_tex_ratio).rgb;
    } else if (value < 0.21) {
        // road
        diffuse = texture(road_tex, terrain_uv * road_tex_ratio).rgb;
    } else if (value < 0.31) {
        // white
        diffuse = texture(white_tex, terrain_uv * road_tex_ratio).rgb;
    }  else if (value > 0.3999 ||  value < 0.760001) {
        // crosswalk
        float theta=(value-0.39999) * 1000/180 * 3.1415926535;
        vec2 new_terrain_uv = vec2(cos(theta)*terrain_uv.x - sin(theta)*terrain_uv.y, sin(theta)*terrain_uv.x+cos(theta)*terrain_uv.y);
        diffuse = texture(crosswalk_tex, new_terrain_uv * road_tex_ratio).rgb;
    } else{
        // Semantics for value 4
        diffuse = texture(white_tex, terrain_uv * road_tex_ratio).rgb;
    }
    tex_normal_world = get_normal(diffuse, road_normal,  road_rough, road_tex_ratio, tbn);
  }

[alexander-toch]

Thanks!

It's working already with onscreen rendering, but when using headless rendering (use_render=False), the Texture keeps being black when capturing images via the RGB Camera. Is there anything I need to add additionally for headless rendering?

see the diff to the current main

[alexander-toch]

Found my issue, I needed to change the dummy env to env = MetaDriveEnv({"use_render": False, "image_observation": True, "sensors": {"rgb_camera": (RGBCamera, 100, 100)}}) so that I can use the AssetLoader in base_block.py for assigning the texture to the node path.

[alexander-toch]

Hey, I have a follow-up question. I got most of the rendering working, but I still struggle with one thing: the texture appearance in the render for the main+RGB cams:

In the best case, the rendered model would be as bright as the original texture image.

As a first step, I disabled the SimplePBR Pipeline in the engine_core, so I don't have to take care of the materials (I don't need the PBR for the sidewalk for now).

And for placing the patch, I'm using a canvas .egg model with the following code:

scale = 0.02
mod = self.loader.loadModel(AssetLoader.file_path_dirty_road_patch("canvas.egg"))
tex = self.loader.loadTexture(AssetLoader.file_path_dirty_road_patch("test_patch.jpg"))
mod.set_scale(1, tex.getXSize()*scale, tex.getYSize()*scale)
mod.setTwoSided(True)
mod.set_texture(tex)
fr = mod.find("**/frame")
fr.setTextureOff(1)
fr.set_transparency(1) # hide the frame
mod.flatten_light()
mod.reparent_to(self.dirty_road_patch_node_path)
mod.set_pos(20, 0, 0.05)

I already experimented a little bit with the tonemap.frag.glsl, but couldn't produce better results.

Any tips would be very appreciated (and a big thanks for the help so far).

Update:

I used a similar technique as used in the semantic map by creating a "magic number" that I can use in the shader to differentiate my custom texture. I assigned a material with roughness = 0.42 to the dirty_road_patch_node_path. In the simplepbr.frag.glsl I then check:

if (p3d_Material.roughness > 0.419 && p3d_Material.roughness < 0.43) {
    is_dirty_road_patch = true;
}

And can then adjust the color of the texture so it matches the original texture image. I use a similar approach in the tonemap shader.