Animation retargeting leads to weird looking animations

[Lunatix89]

Tested versions

• Reproducable in v4.3-dev

System information

Godot v4.3.dev.mono (28d895f32) - Windows 10.0.19045 - Vulkan (Forward+) - dedicated AMD Radeon RX 6700 XT (Advanced Micro Devices, Inc.; 31.0.14043.7000) - AMD Ryzen 9 3900X 12-Core Processor (24 Threads)

Issue description

I'm currently trying to set up a 3D character with 8-directional idle/walk/jog/run animations.
For this, I bought a model from cgtrader as a placeholder model and several locomotion animation packs:

• Mocap Online
• Synty Studios
• Movement_MocapAnimPack_2.0
• Mixamo

And all of them except the Mixamo one are being animated really weird, like the animations are added to their rest pose somehow:

I've also tried to go down the path of importing them into blender, combining all the animations into one file and then reexport as GLTF but that also proved to be error prone because blenders skeleton system isn't compatible with the one from FBX so most of the animations are working but I got all sort of problems with it like twisted foot / hand, crossed arms or even still the same output sometimes.

The thing is - I and many of us game devs are simply no blender / animation pros, neither do we own a license for expensive Autodesk software so we have to rely on assets from third parties which mostly target Unity and Unreal.
So while I fully understand that the Godot way is to provide already animated characters with correct rest poses to the engine, in practise that's causing a lot of trouble (and wasted money) for the developers.

From what I've seen when importing several animations and fiddling around with them, most if not all of the animation files won't have their rest pose set to "T" or "A", they just use the same pose as the first animation frame.
However, the Mixamo character has all it's bones properly set up, nothing twisted after import and also all separate animation files have their rest pose set to "T" so that's why I'm guessing that the rest pose is the root cause here.

In Unity there is this avatar which will be reused across all the imported animations which is usually set up for the base character model.
This might roughly be the same as setting up a BoneMap in Godot and reusing that one for all animation files so maybe there should be an option to use the rest pose of a reference model to calculate the relative poses when retargeting animations.

Steps to reproduce

• Create a blank 3D project
• Grab some locomotion animations from one of the many asset creators that mostly target Unity/Unreal
• Import the base model into Godot
  • Import Mode: Scene
  • Trimming: Yes (Some animations don't start at "0" so they have little jumps in them when looping)
  • Skeleton3D: Setup bone map with humanoid profile and save to disk
• Import animation like "run forward"
  • Import Mode: Animation Library
  • Trimming: Yes
  • Skeleton3D: Use bone map from base model
  • Animation: Set loop mode to "Linear"
• Drag base model into scene
• Right click on the node and enable "Editable Children"
• Select animation player
  • Add one if none was imported - for me, the Synty Robot did not have any animations so the animation player was not added
• Click "Animation -> Manage animations"
• Click "Load Library" and select the desired animation
• Select the animation
• Press "Play"

Minimal reproduction project (MRP)

Unfurtunately I can't provide an MVP since all of the assets are locked behind commercial licenses.
However, Mocap Online offers a free demo pack which can be downloaded for testing purposes so I've set up a small test project but excluded all the FBX files.

One can just download the demo package from Mocap Online here. extract and copy MCO_Demo_Pack_v2/FBX to MocapOnlineDemo/MCO_Demo_Pack_v2/FBX before opening the project and it should work.

MocapOnlineDemo.zip

[fire]

We are working on adding the template pose feature #88824

[Lunatix89]

This looks promising and hopefully gets into Godot soon as it would make live for developers coming from unity so much easier because we could simply reuse our animations we purchased in the asset store.

And I second this comment from @Illauriel:

Here's how I see my specific workflow with this in my imagination:

• I bought a pack of animations from some store and want to retarget them to my character that has vastly different proportions to Unreal Mannequin or whatever the animation creators used as their base model.
• I drop the folder with animation FBX files in the project hierarchy, open the file that says T-pose, set BoneMapHumanoid, reimport open it as a scene and export a SkeletonProfile or whatever it will be called from it.
• I open each of the other animation files' advanced import dialogues, set BoneMapHumanoid, set the exported SkeletonProfile, set a destination to export the animations, reimport.
• Then I delete the FBX source files and forget about their existence. I don't include them in the version control, I don't bloat my project folder, and feel happy about it.

This sounds almost as in Unity - create an avatar from the base model and reuse that in all the split animations and get a nice library of animations when creating a blend tree for the model.

[lyuma]

Wow, thank you for the amazing example project. This is exactly the sort of scenario that #88824 is intended to solve.

With the latest version of the PR, I now use animations to transfer the pose from one model to another.
I tested the workflow and confirmed that it solves the issue here:

Initial version of jog animation, showing the problem:

I now open up the good t-pose version and export the t-pose animation to a file:

This is best done using the "Export Skeleton Rest Pose" field on the skeleton, which works in case the model does not contain an explicit t-pose animation and avoids several pitfalls when referencing an AnimationLibrary or other extracted animation.

After reimporting the t-pose model, I can now use the extracted rest-pose animation in another model:

And tada! We now have the correct jogging animation, in Godot's humanoid format:

I will re-post this in the PR as well to show the workflow.

[Lunatix89]

Wow, you gave me hope back to finally use all those expensive assets I purchased to get nice looking animations in Godot.
I think I'll cherry-pick this into my custom build to test it out :)

[lyuma]

(You're also welcome to do Unidot, which does this internally as part of converting to gltf... but I'm hoping to have native it using native FBX soon with Godot 4.3)

By the way, I wanted to share some of the failure cases when using a normal animation for this posing feature. This is why I added an explicit "Export Skeleton Rest Pose" box.

It is also possible to export individual t-pose animation using Actions -> Save all animations, or exporting the T-Pose animation individually. Or, you can import the t-pose model as AnimationLibrary. However, using this workflow, you will run into three pitfalls, which I will illustrate:
Pitfall 1. If you're using the t-posed model in a scene, changing it to AnimationLibrary will break all references to the asset, and you will be forced to duplicate the FBX. this is not ideal:

Pitfall 2. if you export the t-pose animation from the t-posed model, it will be a post-retargeted animation, which will have identity poses for all bones, since that is the definition of a t-pose in Godot's humanoid retargeter.

I have a warning to detect this case and warn the user, so they do not make this mistake.

Pitfall 3. if you export the t-pose animation without using retargeting (clear the BoneMap), you will also have an invalid animation, since Godot by default excludes animation tracks which do not change the model from the rest pose. Luckily, I am able to detect that not all bones were animated so once again I am able to display a warning:

This is solved by removing the "Keep Custom Tracks" in the import dock:

So to summarize, if you want to use a T-Pose animation but not use the Export Skeleton Rest Pose feature, you must ensure the following:

1. You need to make sure that your t-pose reference model is not imported with a BoneMap.
2. The t-pose reference model should be imported as AnimationLibrary or it should extract the animation from it.
3. The model must be imported with "Remove Immutable Tracks" unchecked.

This is a lot to think about, and part of why I originally wanted to avoid supporting animations at all since it's easy to get wrong, but I think the approach I picked is an okay compromise and the warning box should help I hope.