Queations about imitation inference

[PeterWangyi]

Hello author, thank you for your excellent work.

I have some questions about the inference stage. In some of my previous experiences, such as Deepmimic and AMP methods, a separate model is trained for each motion. During inference, only need to specify the init state, and the rest is left to the model itself. In PHC, the model learns a lot of motions. In my understanding, the inference stage should only specify the first frame action, and let the model infer the rest.

After I ran
python phc/run_hydra.py learning=im_mcp_big exp_name=phc_comp_3 env=env_im_getup_mcp robot=smpl_humanoid env.zero_out_far=False robot.real_weight_porpotion_boxes=False env.num_prim=3 env.motion_file=sample_data/amass_isaac_standing_upright_slim.pkl env.models=['output/HumanoidIm/phc_3/Humanoid.pth'] env.num_envs=1 headless=False epoch=-1 test=True
, and I replaced it with my own motion file.

In the following video, the input action is a walking motion with sliding. I observed that the model inference tends to fit the input motion frame by frame, rather than being able to correct the sliding problem.

So I am curious about how inference is done in PHC, and why it can imitate such a motion frame by frame?

Looking forward to your reply!

phc_comp_kp_2-2024-11-06-14.05.53.mp4

[dbdxnuliba]

Hello
@PeterWangyi @ZhengyiLuo
@kexul
when I run the command,

python phc/run_hydra.py learning=im_mcp_big exp_name=phc_comp_3 env=env_im_getup_mcp robot=smpl_humanoid env.zero_out_far=False robot.real_weight_porpotion_boxes=False env.num_prim=3 env.motion_file=sample_data/amass_isaac_standing_upright_slim.pkl env.models=['output/HumanoidIm/phc_3/Humanoid.pth'] env.num_envs=1 headless=False epoch=-1 test=True

the agent motion is unnormal, here is the gif

could you please help me to fix it .
and my pc is RTX4070Ti

and my conda env pip list :
(isaac) rob@rob:~/rl/PHC$ pip list

absl-py 2.1.0
actionlib 1.14.0
addict 2.4.0
aiohappyeyeballs 2.4.3
aiohttp 3.10.10
aiosignal 1.3.1
angles 1.9.13
antlr4-python3-runtime 4.9.3
asttokens 2.4.1
async-timeout 4.0.3
attrs 24.2.0
autograd 1.7.0
backcall 0.2.0
beautifulsoup4 4.12.3
blinker 1.8.2
bondpy 1.8.6
Brotli 1.0.9
cachetools 5.5.0
camera-calibration 1.17.0
camera-calibration-parsers 1.12.0
catkin 0.8.10
cchardet 2.1.7
certifi 2024.8.30
cffi 1.17.1
chardet 5.2.0
charset-normalizer 3.3.2
chumpy 0.70
click 8.1.7
cloudpickle 3.1.0
comm 0.2.2
ConfigArgParse 1.7
configer 1.3.1
configparser 7.1.0
contourpy 1.1.1
controller-manager 0.20.0
controller-manager-msgs 0.20.0
cv-bridge 1.16.2
cvxopt 1.3.2
cycler 0.12.1
Cython 3.0.11
dash 2.18.2
dash-core-components 2.0.0
dash-html-components 2.0.0
dash-table 5.0.0
decorator 5.1.1
diagnostic-analysis 1.11.0
diagnostic-common-diagnostics 1.11.0
diagnostic-updater 1.11.0
docker-pycreds 0.4.0
dynamic-reconfigure 1.7.3
easydict 1.13
etils 1.3.0
executing 2.1.0
Farama-Notifications 0.0.4
fasteners 0.19
fastjsonschema 2.20.0
filelock 3.13.1
Flask 3.0.3
fonttools 4.54.1
freetype-py 2.5.1
frozenlist 1.5.0
fsspec 2024.10.0
gazebo_plugins 2.9.2
gazebo_ros 2.9.2
gdown 5.2.0
gencpp 0.7.0
geneus 3.0.0
genlisp 0.4.18
genmsg 0.6.0
gennodejs 2.0.2
genpy 0.6.15
gitdb 4.0.11
GitPython 3.1.43
glfw 2.7.0
gmpy2 2.1.2
google-auth 2.35.0
google-auth-oauthlib 1.0.0
grpcio 1.64.0
gym 0.26.2
gym-notices 0.0.8
gymnasium 1.0.0
human_body_prior 0.8.5.0
hydra-core 1.3.2
idna 3.7
image-geometry 1.16.2
imageio 2.35.1
imageio-ffmpeg 0.5.1
importlib_metadata 8.5.0
importlib-resources 3.0.0
interactive-markers 1.12.0
ipdb 0.13.13
ipython 8.12.3
ipywidgets 8.1.5
isaacgym 1.0rc4 /home/rob/rl/IsaacGym_Preview_4_Package/isaacgym/python
itsdangerous 2.2.0
jedi 0.19.1
Jinja2 3.1.4
joblib 1.4.2
joint-state-publisher 1.15.1
joint-state-publisher-gui 1.15.1
jsonschema 4.23.0
jsonschema-specifications 2023.12.1
jupyter_core 5.7.2
jupyterlab_widgets 3.0.13
kiwisolver 1.4.7
laser_geometry 1.6.7
lazy_loader 0.4
lightning-utilities 0.11.8
lxml 5.3.0
Markdown 3.7
MarkupSafe 2.1.3
matplotlib 3.6.3
matplotlib-inline 0.1.7
mediapy 1.2.2
message-filters 1.16.0
mkl-fft 1.3.8
mkl-random 1.2.4
mkl-service 2.4.0
moveit-commander 1.1.13
moveit-core 1.1.13
moveit-python 0.4.5
moveit-ros-planning-interface 1.1.13
moveit-ros-visualization 1.1.13
moveit-task-constructor-core 0.1.3
mpmath 1.3.0
msgpack 1.1.0
mujoco 3.2.3
mujoco-py 2.1.2.14
multidict 6.1.0
nbformat 5.10.4
nest-asyncio 1.6.0
networkx 3.1
ninja 1.11.1.1
numpy 1.22.4
numpy-stl 3.1.2
nvidia-cublas-cu11 11.10.3.66
nvidia-cuda-nvrtc-cu11 11.7.99
nvidia-cuda-runtime-cu11 11.7.99
nvidia-cudnn-cu11 8.5.0.96
oauthlib 3.2.2
omegaconf 2.3.0
open3d 0.18.0
opencv-python 4.6.0.66
packaging 24.1
pandas 2.0.3
parso 0.8.4
patchelf 0.17.2.1
pexpect 4.9.0
pickleshare 0.7.5
pillow 10.4.0
pip 24.2
pkgutil_resolve_name 1.3.10
platformdirs 4.3.6
plotly 5.24.1
prompt_toolkit 3.0.48
propcache 0.2.0
protobuf 5.28.3
psutil 6.1.0
ptyprocess 0.7.0
pure_eval 0.2.3
pyasn1 0.6.1
pyasn1_modules 0.4.1
pycparser 2.22
pyglet 2.0.18
Pygments 2.18.0
PyOpenGL 3.1.7
pyparsing 3.1.4
pyquaternion 0.9.9
pyrender 0.1.45
PySocks 1.7.1
python-dateutil 2.9.0.post0
python-qt-binding 0.4.4
python-utils 3.8.2
pytorch-lightning 2.4.0
pytz 2024.2
PyVirtualDisplay 3.0
PyWavelets 1.4.1
PyYAML 6.0.2
qt-dotgraph 0.4.2
qt-gui 0.4.2
qt-gui-cpp 0.4.2
qt-gui-py-common 0.4.2
ray 2.10.0
referencing 0.35.1
requests 2.32.3
requests-oauthlib 2.0.0
resource_retriever 1.12.7
retrying 1.3.4
rl-games 1.1.4
rosbag 1.16.0
rosboost-cfg 1.15.8
rosclean 1.15.8
roscreate 1.15.8
rosgraph 1.16.0
roslaunch 1.16.0
roslib 1.15.8
roslint 0.12.0
roslz4 1.16.0
rosmake 1.15.8
rosmaster 1.16.0
rosmsg 1.16.0
rosnode 1.16.0
rosparam 1.16.0
rospy 1.16.0
rosservice 1.16.0
rostest 1.16.0
rostopic 1.16.0
rosunit 1.15.8
roswtf 1.16.0
rpds-py 0.20.1
rqt_action 0.4.9
rqt_bag 0.5.1
rqt_bag_plugins 0.5.1
rqt-console 0.4.12
rqt_dep 0.4.12
rqt_graph 0.4.14
rqt_gui 0.5.3
rqt_gui_py 0.5.3
rqt-image-view 0.4.17
rqt_launch 0.4.9
rqt-logger-level 0.4.12
rqt-moveit 0.5.11
rqt_msg 0.4.10
rqt_nav_view 0.5.7
rqt_plot 0.4.13
rqt_pose_view 0.5.11
rqt_publisher 0.4.10
rqt_py_common 0.5.3
rqt_py_console 0.4.10
rqt-reconfigure 0.5.5
rqt-robot-dashboard 0.5.8
rqt-robot-monitor 0.5.15
rqt_robot_steering 0.5.12
rqt-runtime-monitor 0.5.10
rqt-rviz 0.7.0
rqt_service_caller 0.4.10
rqt_shell 0.4.11
rqt_srv 0.4.9
rqt-tf-tree 0.6.4
rqt_top 0.4.10
rqt_topic 0.4.13
rqt_web 0.4.10
rsa 4.9
rviz 1.14.20
scikit-image 0.21.0
scikit-learn 1.3.2
scipy 1.10.1
sensor-msgs 1.13.1
sentry-sdk 2.18.0
setproctitle 1.3.3
setuptools 75.1.0
six 1.16.0
smach 2.5.2
smach-ros 2.5.2
smclib 1.8.6
smmap 5.0.1
smpl_sim 0.0.1
smplx 0.1.28
soupsieve 2.6
srdfdom 0.6.4
stack-data 0.6.3
sympy 1.13.2
tenacity 9.0.0
tensorboard 2.14.0
tensorboard-data-server 0.7.2
tensorboardX 2.6.2.2
termcolor 2.4.0
tf 1.13.2
tf-conversions 1.13.2
tf2-geometry-msgs 0.7.7
tf2-kdl 0.7.7
tf2-py 0.7.7
tf2-ros 0.7.7
threadpoolctl 3.5.0
tifffile 2023.7.10
tomli 2.0.2
topic-tools 1.16.0
torch 1.13.1
torchaudio 2.4.1
torchgeometry 0.1.2
torchmetrics 1.5.1
torchvision 0.14.1
tqdm 4.66.6
trac_ik_python 1.6.6
traitlets 5.14.3
transforms3d 0.4.2
trimesh 4.5.2
typing_extensions 4.11.0
tzdata 2024.2
urdfdom-py 0.4.6
urllib3 2.2.3
vtk 9.3.1
wandb 0.18.5
wcwidth 0.2.13
Werkzeug 3.0.6
wheel 0.44.0
widgetsnbextension 4.0.13
xacro 1.14.18
yarl 1.15.2
zipp 3.20.2

[ZhengyiLuo]

Hi! Thanks for your interest in my work.

PHC is a motion-tracking policy that imitates per-frame reference motion. This formulation allows it to imitate many different types of motion (unlike Deepmimic/AMP, which is tethered to one type of motion). The reward for training PHC is to get as close to the reference motion as possible.

As a result, PHC does not really try to "fix" foot sliding as its primary goal is to "imitate" the reference motion. It will try to imitate the foot sliding motion as much as it can using physically plausible actions. The foot jittering and high-frequency motion you are observing is PHC trying to achieve balance while fitting the reference motion that has the foot sliding.

As for your second video it looks like the agent is imitating the reference motion decently? The reference (red dots) is just standing still without moving.

[PeterWangyi]

@dbdxnuliba
This is normal. There is env=env_im_getup_mcp in the command. During initialization, the character will fall down in the first round, and the reference state init will be executed in the second round of initialization. So it falls down first and then stands up.

You can try pressing the "M" key and then "J" in the window, which is also described in the readme, and you can see the motion of character getup.

[PeterWangyi]

PHC is a motion-tracking policy that imitates per-frame reference motion. This formulation a

@ZhengyiLuo
Thanks for your reply!

I also realized this problem: the physics-based frame-by-frame imitate method does not seem to be suitable for use as a post-processing module to optimize non-physical phenomena in motion.

I read some other papers, such as PhysDiff, which added physics modules to the training process of motion generation, but the efficiency was very low. In addition, do you have any other suggestions?

Looking forward to your insights and replies.

[kexul]

@PeterWangyi Michael Black had a paper about that. https://www.youtube.com/watch?v=Dufvp_O0ziU

[dbdxnuliba]

@PeterWangyi ,thanks for your answer ,and may I ask you another quesition about how can you get your motion file , and what's the full command do you use like abc.pkl

python phc/run_hydra.py learning=im_mcp_big exp_name=phc_comp_3 env=env_im_getup_mcp robot=smpl_humanoid env.zero_out_far=False robot.real_weight_porpotion_boxes=False env.num_prim=3 env.motion_file=sample_data/abc.pkl env.models=['output/HumanoidIm/phc_3/Humanoid.pth'] env.num_envs=1 headless=False epoch=-1 test=True

and could you please send me your comand and motion file ,just as you shown in the gif
Many thanks

Hello author, thank you for your excellent work.

I have some questions about the inference stage. In some of my previous experiences, such as Deepmimic and AMP methods, a separate model is trained for each motion. During inference, only need to specify the init state, and the rest is left to the model itself. In PHC, the model learns a lot of motions. In my understanding, the inference stage should only specify the first frame action, and let the model infer the rest.

After I ran python phc/run_hydra.py learning=im_mcp_big exp_name=phc_comp_3 env=env_im_getup_mcp robot=smpl_humanoid env.zero_out_far=False robot.real_weight_porpotion_boxes=False env.num_prim=3 env.motion_file=sample_data/amass_isaac_standing_upright_slim.pkl env.models=['output/HumanoidIm/phc_3/Humanoid.pth'] env.num_envs=1 headless=False epoch=-1 test=True , and I replaced it with my own motion file.

In the following video, the input action is a walking motion with sliding. I observed that the model inference tends to fit the input motion frame by frame, rather than being able to correct the sliding problem.

So I am curious about how inference is done in PHC, and why it can imitate such a motion frame by frame?

Looking forward to your reply!

phc_comp_kp_2-2024-11-06-14.05.53.mp4