Skip to content

fix segfault in EmbreeScene.__dealloc__, due to use-after-free of EmbreeDevice#7

Merged
mikedh merged 1 commit intotrimesh:release/314from
pbsds:fix-user-after-free-2025-04-23
Oct 22, 2025
Merged

fix segfault in EmbreeScene.__dealloc__, due to use-after-free of EmbreeDevice#7
mikedh merged 1 commit intotrimesh:release/314from
pbsds:fix-user-after-free-2025-04-23

Conversation

@pbsds
Copy link

@pbsds pbsds commented Apr 23, 2025

(ported from e64a2fa / adam-grant-hendry#2, as it seems to apply here too)

pytest + gdb backtrace output, with embree v2.17.7 compiled with -DCMAKE_BUILD_TYPE=RelWithDebInfo:

============================= test session starts ==============================
platform linux -- Python 3.12.9, pytest-8.3.5, pluggy-1.5.0 -- /nix/store/fqm9bqqlmaqqr02qbalm1bazp810qfiw-python3-3.12.9/bin/python3.12
cachedir: .pytest_cache
rootdir: /build/source
configfile: pyproject.toml
collected 13 items

tests/test_intersection.py::TestPyEmbree::test_pyembree_should_be_able_to_create_a_device_if_not_provided PASSED [  7%]
tests/test_intersection.py::TestPyEmbree::test_pyembree_should_be_able_to_create_a_scene PASSED [ 15%]
tests/test_intersection.py::TestPyEmbree::test_pyembree_should_be_able_to_create_several_scenes PASSED [ 23%]
tests/test_intersection.py::TestPyEmbree::test_pyembree_should_be_able_to_display_embree_version PASSED [ 30%]
tests/test_intersection.py::TestIntersectionTriangles::test_intersect PASSED [ 38%]
Thread 1 "python3.12" received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()
#0  0x0000000000000000 in ?? ()
#1  0x00007fffb1e7e76f in embree::APIBuffer<embree::Vec3fa>::free (this=0x8cb7b0) at /build/source/kernels/common/buffer.h:212
#2  embree::APIBuffer<embree::Vec3fa>::~APIBuffer (this=0x8cb7b0, __in_chrg=<optimized out>) at /build/source/kernels/common/buffer.h:137
#3  std::__new_allocator<embree::APIBuffer<embree::Vec3fa> >::destroy<embree::APIBuffer<embree::Vec3fa> > (this=0x9357a8, __p=0x8cb7b0) at /nix/store/qs54xir5n4vhhbi22aydbkvyyq4v8p0l-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/new_allocator.h:198
#4  embree::vector_t<embree::APIBuffer<embree::Vec3fa>, std::allocator<embree::APIBuffer<embree::Vec3fa> > >::clear (this=0x9357a8) at /build/source/kernels/common/../../common/sys/vector.h:148
        i = 0
#5  embree::vector_t<embree::APIBuffer<embree::Vec3fa>, std::allocator<embree::APIBuffer<embree::Vec3fa> > >::~vector_t (this=0x9357a8, __in_chrg=<optimized out>) at /build/source/kernels/common/../../common/sys/vector.h:42
#6  embree::TriangleMesh::~TriangleMesh (this=0x9356b0, __in_chrg=<optimized out>) at /build/source/kernels/common/scene_triangle_mesh.h:25
#7  embree::avx::TriangleMeshISA::~TriangleMeshISA (this=0x9356b0, __in_chrg=<optimized out>) at /build/source/kernels/common/scene_triangle_mesh.h:220
#8  embree::avx::TriangleMeshISA::~TriangleMeshISA (this=0x9356b0, __in_chrg=<optimized out>) at /build/source/kernels/common/scene_triangle_mesh.h:220
#9  0x00007fffb174744f in embree::Scene::~Scene (this=0xb45dc0, __in_chrg=<optimized out>) at /build/source/kernels/common/scene.cpp:594
        i = 0
        i = <optimized out>
#10 0x00007fffb17475e9 in embree::Scene::~Scene (this=0xb45dc0, __in_chrg=<optimized out>) at /build/source/kernels/common/scene.cpp:599
        i = <optimized out>
#11 0x00007ffff3565de8 in __pyx_pf_8pyembree_12rtcore_scene_11EmbreeScene_4__dealloc__ (__pyx_v_self=0x7fffb3055e00) at pyembree/rtcore_scene.cpp:3957

…reeDevice

pytest + gdb backtrace output, with embree v2.17.7 compiled with `-DCMAKE_BUILD_TYPE=RelWithDebInfo`:

    ============================= test session starts ==============================
    platform linux -- Python 3.12.9, pytest-8.3.5, pluggy-1.5.0 -- /nix/store/fqm9bqqlmaqqr02qbalm1bazp810qfiw-python3-3.12.9/bin/python3.12
    cachedir: .pytest_cache
    rootdir: /build/source
    configfile: pyproject.toml
    collected 13 items

    tests/test_intersection.py::TestPyEmbree::test_pyembree_should_be_able_to_create_a_device_if_not_provided PASSED [  7%]
    tests/test_intersection.py::TestPyEmbree::test_pyembree_should_be_able_to_create_a_scene PASSED [ 15%]
    tests/test_intersection.py::TestPyEmbree::test_pyembree_should_be_able_to_create_several_scenes PASSED [ 23%]
    tests/test_intersection.py::TestPyEmbree::test_pyembree_should_be_able_to_display_embree_version PASSED [ 30%]
    tests/test_intersection.py::TestIntersectionTriangles::test_intersect PASSED [ 38%]
    Thread 1 "python3.12" received signal SIGSEGV, Segmentation fault.
    0x0000000000000000 in ?? ()
    #0  0x0000000000000000 in ?? ()
    adam-grant-hendry#1  0x00007fffb1e7e76f in embree::APIBuffer<embree::Vec3fa>::free (this=0x8cb7b0) at /build/source/kernels/common/buffer.h:212
    adam-grant-hendry#2  embree::APIBuffer<embree::Vec3fa>::~APIBuffer (this=0x8cb7b0, __in_chrg=<optimized out>) at /build/source/kernels/common/buffer.h:137
    scopatz#3  std::__new_allocator<embree::APIBuffer<embree::Vec3fa> >::destroy<embree::APIBuffer<embree::Vec3fa> > (this=0x9357a8, __p=0x8cb7b0) at /nix/store/qs54xir5n4vhhbi22aydbkvyyq4v8p0l-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/new_allocator.h:198
    scopatz#4  embree::vector_t<embree::APIBuffer<embree::Vec3fa>, std::allocator<embree::APIBuffer<embree::Vec3fa> > >::clear (this=0x9357a8) at /build/source/kernels/common/../../common/sys/vector.h:148
            i = 0
    scopatz#5  embree::vector_t<embree::APIBuffer<embree::Vec3fa>, std::allocator<embree::APIBuffer<embree::Vec3fa> > >::~vector_t (this=0x9357a8, __in_chrg=<optimized out>) at /build/source/kernels/common/../../common/sys/vector.h:42
    scopatz#6  embree::TriangleMesh::~TriangleMesh (this=0x9356b0, __in_chrg=<optimized out>) at /build/source/kernels/common/scene_triangle_mesh.h:25
    scopatz#7  embree::avx::TriangleMeshISA::~TriangleMeshISA (this=0x9356b0, __in_chrg=<optimized out>) at /build/source/kernels/common/scene_triangle_mesh.h:220
    scopatz#8  embree::avx::TriangleMeshISA::~TriangleMeshISA (this=0x9356b0, __in_chrg=<optimized out>) at /build/source/kernels/common/scene_triangle_mesh.h:220
    scopatz#9  0x00007fffb174744f in embree::Scene::~Scene (this=0xb45dc0, __in_chrg=<optimized out>) at /build/source/kernels/common/scene.cpp:594
            i = 0
            i = <optimized out>
    scopatz#10 0x00007fffb17475e9 in embree::Scene::~Scene (this=0xb45dc0, __in_chrg=<optimized out>) at /build/source/kernels/common/scene.cpp:599
            i = <optimized out>
    scopatz#11 0x00007ffff3565de8 in __pyx_pf_8pyembree_12rtcore_scene_11EmbreeScene_4__dealloc__ (__pyx_v_self=0x7fffb3055e00) at pyembree/rtcore_scene.cpp:3957

(ported from e64a2fa)
@pbsds pbsds force-pushed the fix-user-after-free-2025-04-23 branch from 6e54d82 to c6b0472 Compare April 23, 2025 11:18
@pbsds
Copy link
Author

pbsds commented Apr 23, 2025

I've now reproduced the issue with embreex and verified the fix works:

============================= test session starts ==============================
platform linux -- Python 3.12.9, pytest-8.3.5, pluggy-1.5.0 -- /nix/store/fqm9bqqlmaqqr02qbalm1bazp810qfiw-python3-3.12.9/bin/python3.12
cachedir: .pytest_cache
rootdir: /build/source
configfile: pyproject.toml
collected 13 items

tests/test_intersection.py::Testembreex::test_embreex_should_be_able_to_create_a_device_if_not_provided PASSED [  7%]
tests/test_intersection.py::Testembreex::test_embreex_should_be_able_to_create_a_scene PASSED [ 15%]
tests/test_intersection.py::Testembreex::test_embreex_should_be_able_to_create_several_scenes PASSED [ 23%]
tests/test_intersection.py::Testembreex::test_embreex_should_be_able_to_display_embree_version PASSED [ 30%]
tests/test_intersection.py::TestIntersectionTriangles::test_intersect PASSED [ 38%]
Thread 1 "python3.12" received signal SIGSEGV, Segmentation fault.
0x00007ffff66f6d30 in ?? ()
#0  0x00007ffff66f6d30 in ?? ()
#1  0x00007fff8658be1e in embree::APIBuffer<embree::Vec3fa>::free (this=0x7f4a00) at /build/source/kernels/common/buffer.h:212
#2  0x00007fff8658c5c6 in embree::APIBuffer<embree::Vec3fa>::~APIBuffer (this=0x7f4a00, __in_chrg=<optimized out>) at /build/source/kernels/common/buffer.h:137
#3  0x00007fff8658984a in std::__new_allocator<embree::APIBuffer<embree::Vec3fa> >::destroy<embree::APIBuffer<embree::Vec3fa> > (this=0x9333b8, __p=0x7f4a00) at /nix/store/qs54xir5n4vhhbi22aydbkvyyq4v8p0l-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/new_allocator.h:198
#4  embree::vector_t<embree::APIBuffer<embree::Vec3fa>, std::allocator<embree::APIBuffer<embree::Vec3fa> > >::clear (this=0x9333b8) at /build/source/kernels/common/../../common/sys/vector.h:148
        i = 0
#5  embree::vector_t<embree::APIBuffer<embree::Vec3fa>, std::allocator<embree::APIBuffer<embree::Vec3fa> > >::~vector_t (this=0x9333b8, __in_chrg=<optimized out>) at /build/source/kernels/common/../../common/sys/vector.h:42
#6  embree::TriangleMesh::~TriangleMesh (this=0x9332c0, __in_chrg=<optimized out>) at /build/source/kernels/common/scene_triangle_mesh.h:25
#7  0x00007fff8a4b775a in embree::avx::TriangleMeshISA::~TriangleMeshISA (this=0x9332c0, __in_chrg=<optimized out>) at /build/source/kernels/common/scene_triangle_mesh.h:220
#8  0x00007fff8a4b777c in embree::avx::TriangleMeshISA::~TriangleMeshISA (this=0x9332c0, __in_chrg=<optimized out>) at /build/source/kernels/common/scene_triangle_mesh.h:220
#9  0x00007fff8655cd06 in embree::Scene::~Scene (this=0xaf1c80, __in_chrg=<optimized out>) at /build/source/kernels/common/scene.cpp:594
        i = 0
        i = <optimized out>
#10 0x00007fff8655ce86 in embree::Scene::~Scene (this=0xaf1c80, __in_chrg=<optimized out>) at /build/source/kernels/common/scene.cpp:599
        i = <optimized out>
#11 0x00007fff864af6f1 in embree::rtcDeleteScene (hscene=0xaf1c80) at /build/source/kernels/common/rtcore.cpp:971
        scene = 0xaf1c80
#12 0x00007ffff3562149 in __pyx_pf_7embreex_12rtcore_scene_11EmbreeScene_4__dealloc__ (__pyx_v_self=0x7fff93050b00) at embreex/rtcore_scene.cpp:7149
#13 __pyx_pw_7embreex_12rtcore_scene_11EmbreeScene_5__dealloc__ (__pyx_v_self=0x7fff93050b00) at embreex/rtcore_scene.cpp:7136
        __pyx_kwvalues = 0x0
#14 __pyx_tp_dealloc_7embreex_12rtcore_scene_EmbreeScene (o=0x7fff93050b00) at embreex/rtcore_scene.cpp:7491
        etype = 0x0
        eval = 0x0
        etb = 0x0
        p = 0x7fff93050b00

@mikedh mikedh changed the base branch from main to release/314 October 22, 2025 17:37
@mikedh mikedh merged commit 3b5fc64 into trimesh:release/314 Oct 22, 2025
@mikedh mikedh mentioned this pull request Oct 22, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants