Project: Humanoid Robot Imitation of Human Motion from Instructional Videos

Project for the MVA "Object Recognition and Computer Vision" class (https://www.di.ens.fr/willow/teaching/recvis19/).

Clone the repository

git clone https://github.com/ManifoldFR/recvis-project

Dependencies

The initial HMR repository uses Python 2.7 and TensorFlow 1.3. We patched HMR to be able to use more recent versions of Python and TensorFlow 1.x (tested on TensorFlow 1.15 and Python 3.7).

We reverse-engineered motion-reconstruction to use with AlphaPose (included in the repository) and Python 3 instead of Openpose.

The patched versions of the repos are included in this repository.

DeepMimic requires a specific format for motion capture files. They can be obtained from BVH (BioVision Hierarchy Animation) files. Converting from BVH to DeepMimic requires the BvhToDeepMimic package

pip install bvhtodeepmimic

Use

Getting MoCap data from a video

Put the videos in a directory (by default data/vid) and call

python -m run_alphapose

Then run

python -m refine_hmr

The output will be in the refined directory.

Conversion of motion to DeepMimic JSON format

Use https://github.com/BartMoyaers/BvhToDeepMimic to convert the BVH files to DeepMimic-formatted files.

We repurposed MoCap conversion files from the PyBullet reimplementation of DeepMimic (credit to Erwin Coumans and Yihang Yin), inverse_kinematics.py and transformation.py. We wrote a wrapper for this code that you can modify and call as

python ik_hmr_deepmimic.py

For information, the HMR output joints are as follow (from a comment inside of the HMR source):

0. Right ankle
1. Right knee
2. Right hip
3. Left hip
4. Left knee
5. Left ankle
6. Right wrist
7. Right elbow
8. Right shoulder
9. Left shoulder
10. Left elbow
11. Left wrist
12. Neck
13. Head top
14. nose
15. left_eye
16. right_eye
17. left_ear
18. right_ear

Bibliography

• SFV: Reinforcement Learning of Physical Skills from Videos, Peng, Kanazawa, Malik, Abbeel and Levine. In ACM Transactions on Graphics (Proc. SIGGRAPH Asia 2018). https://xbpeng.github.io/projects/SFV/2018_TOG_SFV.pdf | GitHub project page GitHub code repo
• DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills: https://xbpeng.github.io/projects/DeepMimic/index.html
• Estimating 3D Motion and Forces of Person-Object Interactionsfrom Monocular Video. https://arxiv.org/pdf/1904.02683.pdf | Project page
• End-to-end Recovery of Human Shape and Pose: https://arxiv.org/pdf/1712.06584.pdf | GitHub repo
• RMPE: Regional Multi-person Pose Estimation: https://arxiv.org/abs/1612.00137.pdf | Project page