[rmodels] Incorrect animation of glTF model

[paulmelis]

Please, before submitting a new issue verify and check:

• I tested it on latest raylib version from master branch
• I checked there is no similar issue already reported
• I checked the documentation on the wiki
• My code has no errors or misuse of raylib

By the way, the FAQ currently reads:

GLTF: This file format supports animation but raylib is not able to load them at this moment. GLTF animations loading was implemented in the past but GLTF is a quite complex file format and could implement animations in many ways, raylib used to fail a lot on loading them and support was removed.

But there is LoadModelAnimations() which supports GLTF? So seems the FAQ entry is out of date?

Issue description

While trying to figure out how to make #4037 work well together with animated models, I noticed current support for animations isn't entirely bug-free (hence making the glTF transform work on top of it somewhat challenging).

For example, with https://github.com/KhronosGroup/glTF-Sample-Models/blob/main/2.0/BrainStem/glTF-Binary/BrainStem.glb (which needs #4053 to handle the missing bone names) the initial load of the model, without applying UpdateModelAnimation() is okay (which is not surprising, as there's only a single mesh):

But when applying UpdateModelAnimation() in the render loop, the transforms based on the armature make parts fly all over the place incorrectly:

But then a very similar model like https://github.com/KhronosGroup/glTF-Sample-Models/blob/main/2.0/CesiumMan/glTF-Binary/CesiumMan.glb does animate correctly, and I don't immediately see the big difference between the two files causing the issues.

Environment

Arch Linux, GTX TITAN

Code Example

#include <string>

#define SUPPORT_FILEFORMAT_JPG
#include "raylib.h"

#define PLATFORM_DESKTOP

#if defined(PLATFORM_DESKTOP)
#define GLSL_VERSION            330
#else   // PLATFORM_ANDROID, PLATFORM_WEB
#define GLSL_VERSION            120
#endif

int main(int argc, char *argv[])
{
    std::string modelfile("../models/robot.glb");
    if (--argc == 1)
        modelfile = argv[1];

    const int screenWidth = 950;
    const int screenHeight = 540;

    SetConfigFlags(FLAG_MSAA_4X_HINT);
    InitWindow(screenWidth, screenHeight, "play animation");

    Model model = LoadModel(modelfile.c_str());

    BoundingBox bbox = GetModelBoundingBox(model);
    printf("Model bbox min: %.6f, %.6f, %.6f\n", bbox.min.x, bbox.min.y, bbox.min.z);
    printf("Model bbox max: %.6f, %.6f, %.6f\n", bbox.max.x, bbox.max.y, bbox.max.z);

    int animCount = 0;
    ModelAnimation* modelAnimations = LoadModelAnimations(modelfile.c_str(), &animCount);
    printf("Loaded %d animation(s)\n", animCount);

    Camera3D cam = (Camera3D){ 0 };
    //cam.position = (Vector3) { bbox.max.x*2.5f, bbox.max.y*2.5f, bbox.max.z*2.5f };
    //cam.target = (Vector3) { 0.5f*(bbox.min.x+bbox.max.x), 0.5f*(bbox.min.y+bbox.max.y), 0.5f*(bbox.min.z+bbox.max.z) };
    cam.position = (Vector3){ 3.0f, 3.0f, 3.0f };
    cam.projection = CAMERA_PERSPECTIVE;
    cam.up = (Vector3){ 0.0f, 1.0f, 0.0f };
    cam.fovy = 45.0f;

    bool animating = false;

    SetTargetFPS(60);
    
    int fc = 0;

    // Main game loop
    while (!WindowShouldClose())
    {
        float dt = GetFrameTime();

        UpdateCamera(&cam, CAMERA_ORBITAL);

        if (modelAnimations && animating)
        {
            fc++;
            fc %= (modelAnimations[0].frameCount);
            UpdateModelAnimation(model, modelAnimations[0], fc);
        }

        BeginDrawing();
            ClearBackground(RAYWHITE);
            BeginMode3D(cam);
                DrawModelEx(model, (Vector3) { 0.0f, 0.5f, 0.0f }, (Vector3) { 0.0f, 1.0f, 0.0f }, 0.0f, (Vector3) { 1.0f, 1.0f, 1.0f }, WHITE);
                DrawGrid(10, 1.0f);    
            EndMode3D();
        EndDrawing();

        if (IsKeyPressed(KEY_S)) TakeScreenshot("animation.png");
        if (IsKeyPressed(KEY_A)) animating = !animating;
    }

    UnloadModel(model);
    CloseWindow();

    return 0;
}

Edit: first linked glb was wrong, should be BrainStem.glb, now fixed

[raysan5]

@paulmelis Thanks for reporting about the FAQ, just reviewed it.

About the different models animations working or not, I don't know the reason but I can guess is related to the bones hierarchy and its transformations. Is it possible to export a GLTF file with the transformations "baked" by bone?

[ed-mart]

Were these models exported by blockbench?

[paulmelis]

Were these models exported by blockbench?

I don't know. They're part of the glTF sample repo, but no idea how they were created.

[paulmelis]

Hmm, one difference I see that might be of influence is that CesiumMan.glb only has one mesh with one primitive, while BrainStem.glb also only has one mesh but that mesh has a few dozen primitives (each with their own JOINT_0 and WEIGHTS_0 arrays). And primitives are converted into separate raylib Mesh's during loading.

[Arcod7]

Were these models exported by blockbench?

Hello, I tested exported models by blockbench of the cobblemon's pokemon. In general, feet and body are placed correctly during animation, but the head and other bones like leaves on a ivysaur go right in the center. I have faced a similar issue with the Fox of the glTF sample repo.

[raysan5]

@paulmelis @Arcod7 thanks for the further investigation of this issue! It seems issue could be related how meshes are exported by blockbench but also how they are loaded by raylib...

[Arcod7]

Here's an illustrated example : The Torterra model in blockbench exported to glb works great in blender, but breaks in raylib (on the latest github dev version).


Here's a short clip

In Blockbench, we can see that the parts breaking are the ones indented more than 1 in the body. For example the head is affected (body->torso->head) and the feet too while the leg is well placed (legs are not part of torso).
During the animation, I noticed that while the head and others are placed in the center of the model, the feet seems to be at the center of the leg and not the body.

I'm highly motivated to try to solve the bug, do you have any idea that could help me ?

[raysan5]

@Arcod7 Mmmh... interesting, it seems that node transformations are only considered for the first hierarchy level but not applied down to the other hierarchy levels.

[Arcod7]

Okay, do you think you will have time for this soon ? If there's something else I can provide to help, you can tell me !

[raysan5]

@Arcod7 I'm afraid I don't have time to review it. I neither implemented current loader (only vertex data loading) so it would take me a lot of time that unfortunately I don't have at the moment.

[VitoTringolo]

Hey I think I quite fixed this one! I need some more time to check it but it seems to work quite well!
The only thing that is not managed yet is the right rotation for the models with the Z-UP

Before

After

Before

After

Before

After

I'll create a merge request soon :)

[raysan5]

@VitoTringolo Wow! It looks great!!!

[MrScautHD]

@VitoTringolo

Can you maybe try this model to?
Player-Female.zip

[Arcod7]

Hello @VitoTringolo, you did great, it looks stunning !
I have a project with raylib that ends in 2 days and I would love to try out your fix, could you push it on your fork ?

Also did you only touched the translation part or you also fixed rotations ?

On the model provided by @MrScautHD (and a lot of my models), parts of the body disappear with the rotation applied.
For exemple, the head of the Player-Female model is only present at the frame first and last frame of the run animation. It's the same thing with the arm of the player with the idle animation.

Run animation: (only scale and rotation applied, I disabled transformations)

Idle animation: (only scale and rotation applied, I disabled transformations)

[raysan5]

@MrScautHD @Arcod7 is this model created with Blockbench?

[MrScautHD]

@MrScautHD @Arcod7 is this model created with Blockbench?

yea

[Arcod7]

@MrScautHD @Arcod7 is this model created with Blockbench?

My models are also coming from blockbench (bbmodel exported to glb). They are working great with blender (not a single issue).

[VitoTringolo]

@VitoTringolo

Can you maybe try this model to? Player-Female.zip

Sure thing!
This is the result that seems to be correct!
If you have other models to test, please share :)

[VitoTringolo]

Hello @VitoTringolo, you did great, it looks stunning ! I have a project with raylib that ends in 2 days and I would love to try out your fix, could you push it on your fork ?

Also did you only touched the translation part or you also fixed rotations ?

On the model provided by @MrScautHD (and a lot of my models), parts of the body disappear with the rotation applied. For exemple, the head of the Player-Female model is only present at the frame first and last frame of the run animation. It's the same thing with the arm of the player with the idle animation.

Run animation: (only scale and rotation applied, I disabled transformations)

Idle animation: (only scale and rotation applied, I disabled transformations)

Before we were completely not considering the transform matrix when the animation channel was NULL! This was the main issue! So to answer your question, yes I am now considering the translation the rotation and also the scaling factor for each joint

[Arcod7]

It looks so nice 😊 !
Can you try the second and third animations of this model ? This is where I was getting issues with head and arms which disappear

My models are those from cobblemon
These are .bbmodel, it can be exported to .glb by blockbench (make sure to check the "Export Groups as Armature" when exporting)

If you want a sample of exported models I can provide it.

[VitoTringolo]

It seems to work well also for these animations.

[Arcod7]

This fixes almost everything on the cobblemon models, thanks a lot !
There is juste one last thing, sometimes a bbmodel animation has an animation length of 0 which means that there's no end to the animation but it still playable.
Do you have an idea of what's going on with raylib ?

[VitoTringolo]

This fixes almost everything on the cobblemon models, thanks a lot ! There is juste one last thing, sometimes a bbmodel animation has an animation length of 0 which means that there's no end to the animation but it still playable. Do you have an idea of what's going on with raylib ?

I'm not sure I understand what you described, what do you mean with "animation has an animation length of 0 "? Can you share a model with these properties?

[Arcod7]

Sure ! Here's a model with a ground_walk working at index 1 (animation length is set to 2 in blockbench) and a ground_idle that's not working as expected at index 0 (animation length is set to 0 in blockbench)
torterra.glb.zip

https://github.com/raysan5/raylib/assets/77203393/cb7a6135-09f9-4f23-98ae-6356668612f9
I link a video to illustrate the issue

Have a good day !

[VitoTringolo]

Oh OK now I understood, sorry 😅

We have the same behavior of Blender with this model.
In my opinion the solution for this is to set a value different then zero before exporting the model from Blockbench.
I tried it and the result looks good

[raysan5]

In my opinion the solution for this is to set a value different then zero before exporting the model from Blockbench.

I think that is the right solution. Otherwise, it should be checked if the glTF official specs allow other behaviours.

[raysan5]

@MrScautHD What is the license of the Player-Female.glb model? Is it free?

[MrScautHD]

@MrScautHD What is the license of the Player-Female.glb model? Is it free?

Its mine, so feel free to use it. 😊

[MrScautHD]

@VitoTringolo and @raysan5 What about this error?
It really spamms the console like 1000 lines and the model still work.

It is coming from:

[VitoTringolo]

@VitoTringolo and @raysan5 What about this error? It really spamms the console like 1000 lines and the model still work.

It is coming from:

Oh yeah! That's because of this line of code:

if (FloatEquals(tend, tstart)) return false;

I think I can change to return true, without any problem to avoid flooding!