Eye Alignment

[emlcpfx]

@MichaelJBlack @radekd91 @TimoBolkart Can you please weigh in on this issue with eye alignment in DECA that has made its way into EMOCA?

yfeng95/DECA#29

[emlcpfx]

I found a hacky workaround to this problem. You can move around vertices on the FLAME model before they get encoded (or whatever it's called.)

in DECA.py you'll find the trans_verts variable on line 970

` # 1) Reconstruct the face mesh
# FLAME - world space
if not isinstance(self.deca.flame, FLAME_mediapipe):
verts, landmarks2d, landmarks3d = self.deca.flame(shape_params=shapecode, expression_params=expcode,
pose_params=posecode)
landmarks2d_mediapipe = None
else:
verts, landmarks2d, landmarks3d, landmarks2d_mediapipe = self.deca.flame(shapecode, expcode, posecode)

    # world to camera
    trans_verts = util.batch_orth_proj(verts, cam)

    def move_vertices_up(trans_verts, center_vertex_index, threshold_distance, upward_translation):
        # Calculate the Euclidean distance between each vertex and the center vertex
        distances = torch.norm(trans_verts - trans_verts[:, center_vertex_index:center_vertex_index+1, :], dim=2)

        # Create a mask indicating which vertices are within the threshold distance
        mask_nearby = distances < threshold_distance

        # Apply the mask to select the nearby vertices for each batch
        selected_vertex_indices = torch.nonzero(mask_nearby, as_tuple=True)

        # Update the Y coordinate of the selected vertices to move them up
        trans_verts[selected_vertex_indices[0], selected_vertex_indices[1], 1] += upward_translation

    # SCREEN RIGHT EYE
    center_vertex_index = 4043
    threshold_distance = 0.15
    upward_translation = 0.04
    move_vertices_up(trans_verts, center_vertex_index, threshold_distance, upward_translation)

    # SCREEN LEFT EYE
    center_vertex_index = 4587
    threshold_distance = 0.1
    upward_translation = 0.025
    move_vertices_up(trans_verts, center_vertex_index, threshold_distance, upward_translation)



    predicted_landmarks = util.batch_orth_proj(landmarks2d, cam)[:, :, :2]

    # camera to image space
    trans_verts[:, :, 1:] = -trans_verts[:, :, 1:]
    predicted_landmarks[:, :, 1:] = - predicted_landmarks[:, :, 1:]

`

1. Open emoca\assets\FLAME\geometry\head_template_watertight.obj in Blender.
2. Follow these directions to get the index number of the vertices you want to move around: https://blender.stackexchange.com/questions/3249/show-mesh-vertices-id
3. Play with the threshold_distance to select nearby vertices and the upward_translation for how much up/down you want to move the vertices. (Note: This only moves them in the Y axis, because that's what I needed.)

I'd love to hear if anyone has a better way to implement this.

[emlcpfx]

Ultimately, this is a bad idea, because the soft selection creates jittering vertices. It’s better to scale and move up all of the verts.