Skip to content

fix segfault in EmbreeScene.__dealloc__, due to use-after-free of EmbreeDevice#2

Open
pbsds wants to merge 1 commit intoadam-grant-hendry:masterfrom
pbsds:fix-user-after-free-2025-04-21
Open

fix segfault in EmbreeScene.__dealloc__, due to use-after-free of EmbreeDevice#2
pbsds wants to merge 1 commit intoadam-grant-hendry:masterfrom
pbsds:fix-user-after-free-2025-04-21

Conversation

@pbsds
Copy link

@pbsds pbsds commented Apr 21, 2025

pytest + gdb backtrace output, with embree v2.17.7 compiled with -DCMAKE_BUILD_TYPE=RelWithDebInfo:

============================= test session starts ==============================
platform linux -- Python 3.12.9, pytest-8.3.5, pluggy-1.5.0 -- /nix/store/fqm9bqqlmaqqr02qbalm1bazp810qfiw-python3-3.12.9/bin/python3.12
cachedir: .pytest_cache
rootdir: /build/source
configfile: pyproject.toml
collected 13 items

tests/test_intersection.py::TestPyEmbree::test_pyembree_should_be_able_to_create_a_device_if_not_provided PASSED [  7%]
tests/test_intersection.py::TestPyEmbree::test_pyembree_should_be_able_to_create_a_scene PASSED [ 15%]
tests/test_intersection.py::TestPyEmbree::test_pyembree_should_be_able_to_create_several_scenes PASSED [ 23%]
tests/test_intersection.py::TestPyEmbree::test_pyembree_should_be_able_to_display_embree_version PASSED [ 30%]
tests/test_intersection.py::TestIntersectionTriangles::test_intersect PASSED [ 38%]
Thread 1 "python3.12" received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()
#0  0x0000000000000000 in ?? ()
#1  0x00007fffb1e7e76f in embree::APIBuffer<embree::Vec3fa>::free (this=0x8cb7b0) at /build/source/kernels/common/buffer.h:212
#2  embree::APIBuffer<embree::Vec3fa>::~APIBuffer (this=0x8cb7b0, __in_chrg=<optimized out>) at /build/source/kernels/common/buffer.h:137
#3  std::__new_allocator<embree::APIBuffer<embree::Vec3fa> >::destroy<embree::APIBuffer<embree::Vec3fa> > (this=0x9357a8, __p=0x8cb7b0) at /nix/store/qs54xir5n4vhhbi22aydbkvyyq4v8p0l-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/new_allocator.h:198
#4  embree::vector_t<embree::APIBuffer<embree::Vec3fa>, std::allocator<embree::APIBuffer<embree::Vec3fa> > >::clear (this=0x9357a8) at /build/source/kernels/common/../../common/sys/vector.h:148
        i = 0
#5  embree::vector_t<embree::APIBuffer<embree::Vec3fa>, std::allocator<embree::APIBuffer<embree::Vec3fa> > >::~vector_t (this=0x9357a8, __in_chrg=<optimized out>) at /build/source/kernels/common/../../common/sys/vector.h:42
#6  embree::TriangleMesh::~TriangleMesh (this=0x9356b0, __in_chrg=<optimized out>) at /build/source/kernels/common/scene_triangle_mesh.h:25
#7  embree::avx::TriangleMeshISA::~TriangleMeshISA (this=0x9356b0, __in_chrg=<optimized out>) at /build/source/kernels/common/scene_triangle_mesh.h:220
#8  embree::avx::TriangleMeshISA::~TriangleMeshISA (this=0x9356b0, __in_chrg=<optimized out>) at /build/source/kernels/common/scene_triangle_mesh.h:220
#9  0x00007fffb174744f in embree::Scene::~Scene (this=0xb45dc0, __in_chrg=<optimized out>) at /build/source/kernels/common/scene.cpp:594
        i = 0
        i = <optimized out>
#10 0x00007fffb17475e9 in embree::Scene::~Scene (this=0xb45dc0, __in_chrg=<optimized out>) at /build/source/kernels/common/scene.cpp:599
        i = <optimized out>
#11 0x00007ffff3565de8 in __pyx_pf_8pyembree_12rtcore_scene_11EmbreeScene_4__dealloc__ (__pyx_v_self=0x7fffb3055e00) at pyembree/rtcore_scene.cpp:3957

Which indicates the segfault happens in

https://github.com/RenderKit/embree/blob/8b2ad69f899ca0710f94d10303a2f02d7426faaf/kernels/common/buffer.h#L212

…reeDevice

pytest + gdb backtrace output, with embree v2.17.7 compiled with `-DCMAKE_BUILD_TYPE=RelWithDebInfo`:

    ============================= test session starts ==============================
    platform linux -- Python 3.12.9, pytest-8.3.5, pluggy-1.5.0 -- /nix/store/fqm9bqqlmaqqr02qbalm1bazp810qfiw-python3-3.12.9/bin/python3.12
    cachedir: .pytest_cache
    rootdir: /build/source
    configfile: pyproject.toml
    collected 13 items

    tests/test_intersection.py::TestPyEmbree::test_pyembree_should_be_able_to_create_a_device_if_not_provided PASSED [  7%]
    tests/test_intersection.py::TestPyEmbree::test_pyembree_should_be_able_to_create_a_scene PASSED [ 15%]
    tests/test_intersection.py::TestPyEmbree::test_pyembree_should_be_able_to_create_several_scenes PASSED [ 23%]
    tests/test_intersection.py::TestPyEmbree::test_pyembree_should_be_able_to_display_embree_version PASSED [ 30%]
    tests/test_intersection.py::TestIntersectionTriangles::test_intersect PASSED [ 38%]
    Thread 1 "python3.12" received signal SIGSEGV, Segmentation fault.
    0x0000000000000000 in ?? ()
    #0  0x0000000000000000 in ?? ()
    adam-grant-hendry#1  0x00007fffb1e7e76f in embree::APIBuffer<embree::Vec3fa>::free (this=0x8cb7b0) at /build/source/kernels/common/buffer.h:212
    adam-grant-hendry#2  embree::APIBuffer<embree::Vec3fa>::~APIBuffer (this=0x8cb7b0, __in_chrg=<optimized out>) at /build/source/kernels/common/buffer.h:137
    scopatz#3  std::__new_allocator<embree::APIBuffer<embree::Vec3fa> >::destroy<embree::APIBuffer<embree::Vec3fa> > (this=0x9357a8, __p=0x8cb7b0) at /nix/store/qs54xir5n4vhhbi22aydbkvyyq4v8p0l-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/new_allocator.h:198
    scopatz#4  embree::vector_t<embree::APIBuffer<embree::Vec3fa>, std::allocator<embree::APIBuffer<embree::Vec3fa> > >::clear (this=0x9357a8) at /build/source/kernels/common/../../common/sys/vector.h:148
            i = 0
    scopatz#5  embree::vector_t<embree::APIBuffer<embree::Vec3fa>, std::allocator<embree::APIBuffer<embree::Vec3fa> > >::~vector_t (this=0x9357a8, __in_chrg=<optimized out>) at /build/source/kernels/common/../../common/sys/vector.h:42
    scopatz#6  embree::TriangleMesh::~TriangleMesh (this=0x9356b0, __in_chrg=<optimized out>) at /build/source/kernels/common/scene_triangle_mesh.h:25
    scopatz#7  embree::avx::TriangleMeshISA::~TriangleMeshISA (this=0x9356b0, __in_chrg=<optimized out>) at /build/source/kernels/common/scene_triangle_mesh.h:220
    scopatz#8  embree::avx::TriangleMeshISA::~TriangleMeshISA (this=0x9356b0, __in_chrg=<optimized out>) at /build/source/kernels/common/scene_triangle_mesh.h:220
    scopatz#9  0x00007fffb174744f in embree::Scene::~Scene (this=0xb45dc0, __in_chrg=<optimized out>) at /build/source/kernels/common/scene.cpp:594
            i = 0
            i = <optimized out>
    scopatz#10 0x00007fffb17475e9 in embree::Scene::~Scene (this=0xb45dc0, __in_chrg=<optimized out>) at /build/source/kernels/common/scene.cpp:599
            i = <optimized out>
    scopatz#11 0x00007ffff3565de8 in __pyx_pf_8pyembree_12rtcore_scene_11EmbreeScene_4__dealloc__ (__pyx_v_self=0x7fffb3055e00) at pyembree/rtcore_scene.cpp:3957
pbsds added a commit to pbsds/pyembree that referenced this pull request Apr 23, 2025
…reeDevice

pytest + gdb backtrace output, with embree v2.17.7 compiled with `-DCMAKE_BUILD_TYPE=RelWithDebInfo`:

    ============================= test session starts ==============================
    platform linux -- Python 3.12.9, pytest-8.3.5, pluggy-1.5.0 -- /nix/store/fqm9bqqlmaqqr02qbalm1bazp810qfiw-python3-3.12.9/bin/python3.12
    cachedir: .pytest_cache
    rootdir: /build/source
    configfile: pyproject.toml
    collected 13 items

    tests/test_intersection.py::TestPyEmbree::test_pyembree_should_be_able_to_create_a_device_if_not_provided PASSED [  7%]
    tests/test_intersection.py::TestPyEmbree::test_pyembree_should_be_able_to_create_a_scene PASSED [ 15%]
    tests/test_intersection.py::TestPyEmbree::test_pyembree_should_be_able_to_create_several_scenes PASSED [ 23%]
    tests/test_intersection.py::TestPyEmbree::test_pyembree_should_be_able_to_display_embree_version PASSED [ 30%]
    tests/test_intersection.py::TestIntersectionTriangles::test_intersect PASSED [ 38%]
    Thread 1 "python3.12" received signal SIGSEGV, Segmentation fault.
    0x0000000000000000 in ?? ()
    #0  0x0000000000000000 in ?? ()
    adam-grant-hendry#1  0x00007fffb1e7e76f in embree::APIBuffer<embree::Vec3fa>::free (this=0x8cb7b0) at /build/source/kernels/common/buffer.h:212
    adam-grant-hendry#2  embree::APIBuffer<embree::Vec3fa>::~APIBuffer (this=0x8cb7b0, __in_chrg=<optimized out>) at /build/source/kernels/common/buffer.h:137
    scopatz#3  std::__new_allocator<embree::APIBuffer<embree::Vec3fa> >::destroy<embree::APIBuffer<embree::Vec3fa> > (this=0x9357a8, __p=0x8cb7b0) at /nix/store/qs54xir5n4vhhbi22aydbkvyyq4v8p0l-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/new_allocator.h:198
    scopatz#4  embree::vector_t<embree::APIBuffer<embree::Vec3fa>, std::allocator<embree::APIBuffer<embree::Vec3fa> > >::clear (this=0x9357a8) at /build/source/kernels/common/../../common/sys/vector.h:148
            i = 0
    scopatz#5  embree::vector_t<embree::APIBuffer<embree::Vec3fa>, std::allocator<embree::APIBuffer<embree::Vec3fa> > >::~vector_t (this=0x9357a8, __in_chrg=<optimized out>) at /build/source/kernels/common/../../common/sys/vector.h:42
    scopatz#6  embree::TriangleMesh::~TriangleMesh (this=0x9356b0, __in_chrg=<optimized out>) at /build/source/kernels/common/scene_triangle_mesh.h:25
    scopatz#7  embree::avx::TriangleMeshISA::~TriangleMeshISA (this=0x9356b0, __in_chrg=<optimized out>) at /build/source/kernels/common/scene_triangle_mesh.h:220
    scopatz#8  embree::avx::TriangleMeshISA::~TriangleMeshISA (this=0x9356b0, __in_chrg=<optimized out>) at /build/source/kernels/common/scene_triangle_mesh.h:220
    scopatz#9  0x00007fffb174744f in embree::Scene::~Scene (this=0xb45dc0, __in_chrg=<optimized out>) at /build/source/kernels/common/scene.cpp:594
            i = 0
            i = <optimized out>
    scopatz#10 0x00007fffb17475e9 in embree::Scene::~Scene (this=0xb45dc0, __in_chrg=<optimized out>) at /build/source/kernels/common/scene.cpp:599
            i = <optimized out>
    scopatz#11 0x00007ffff3565de8 in __pyx_pf_8pyembree_12rtcore_scene_11EmbreeScene_4__dealloc__ (__pyx_v_self=0x7fffb3055e00) at pyembree/rtcore_scene.cpp:3957

(ported from e64a2fa)
pbsds added a commit to pbsds/pyembree that referenced this pull request Apr 23, 2025
…reeDevice

pytest + gdb backtrace output, with embree v2.17.7 compiled with `-DCMAKE_BUILD_TYPE=RelWithDebInfo`:

    ============================= test session starts ==============================
    platform linux -- Python 3.12.9, pytest-8.3.5, pluggy-1.5.0 -- /nix/store/fqm9bqqlmaqqr02qbalm1bazp810qfiw-python3-3.12.9/bin/python3.12
    cachedir: .pytest_cache
    rootdir: /build/source
    configfile: pyproject.toml
    collected 13 items

    tests/test_intersection.py::TestPyEmbree::test_pyembree_should_be_able_to_create_a_device_if_not_provided PASSED [  7%]
    tests/test_intersection.py::TestPyEmbree::test_pyembree_should_be_able_to_create_a_scene PASSED [ 15%]
    tests/test_intersection.py::TestPyEmbree::test_pyembree_should_be_able_to_create_several_scenes PASSED [ 23%]
    tests/test_intersection.py::TestPyEmbree::test_pyembree_should_be_able_to_display_embree_version PASSED [ 30%]
    tests/test_intersection.py::TestIntersectionTriangles::test_intersect PASSED [ 38%]
    Thread 1 "python3.12" received signal SIGSEGV, Segmentation fault.
    0x0000000000000000 in ?? ()
    #0  0x0000000000000000 in ?? ()
    adam-grant-hendry#1  0x00007fffb1e7e76f in embree::APIBuffer<embree::Vec3fa>::free (this=0x8cb7b0) at /build/source/kernels/common/buffer.h:212
    adam-grant-hendry#2  embree::APIBuffer<embree::Vec3fa>::~APIBuffer (this=0x8cb7b0, __in_chrg=<optimized out>) at /build/source/kernels/common/buffer.h:137
    scopatz#3  std::__new_allocator<embree::APIBuffer<embree::Vec3fa> >::destroy<embree::APIBuffer<embree::Vec3fa> > (this=0x9357a8, __p=0x8cb7b0) at /nix/store/qs54xir5n4vhhbi22aydbkvyyq4v8p0l-gcc-14.2.1.20250322/include/c++/14.2.1.20250322/bits/new_allocator.h:198
    scopatz#4  embree::vector_t<embree::APIBuffer<embree::Vec3fa>, std::allocator<embree::APIBuffer<embree::Vec3fa> > >::clear (this=0x9357a8) at /build/source/kernels/common/../../common/sys/vector.h:148
            i = 0
    scopatz#5  embree::vector_t<embree::APIBuffer<embree::Vec3fa>, std::allocator<embree::APIBuffer<embree::Vec3fa> > >::~vector_t (this=0x9357a8, __in_chrg=<optimized out>) at /build/source/kernels/common/../../common/sys/vector.h:42
    scopatz#6  embree::TriangleMesh::~TriangleMesh (this=0x9356b0, __in_chrg=<optimized out>) at /build/source/kernels/common/scene_triangle_mesh.h:25
    scopatz#7  embree::avx::TriangleMeshISA::~TriangleMeshISA (this=0x9356b0, __in_chrg=<optimized out>) at /build/source/kernels/common/scene_triangle_mesh.h:220
    scopatz#8  embree::avx::TriangleMeshISA::~TriangleMeshISA (this=0x9356b0, __in_chrg=<optimized out>) at /build/source/kernels/common/scene_triangle_mesh.h:220
    scopatz#9  0x00007fffb174744f in embree::Scene::~Scene (this=0xb45dc0, __in_chrg=<optimized out>) at /build/source/kernels/common/scene.cpp:594
            i = 0
            i = <optimized out>
    scopatz#10 0x00007fffb17475e9 in embree::Scene::~Scene (this=0xb45dc0, __in_chrg=<optimized out>) at /build/source/kernels/common/scene.cpp:599
            i = <optimized out>
    scopatz#11 0x00007ffff3565de8 in __pyx_pf_8pyembree_12rtcore_scene_11EmbreeScene_4__dealloc__ (__pyx_v_self=0x7fffb3055e00) at pyembree/rtcore_scene.cpp:3957

(ported from e64a2fa)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant