LineSegmentIntersector cannot be applied to a single point in a point cloud

[mingkangzh]

My requirement is this: I can select a point on the point cloud, display the index, world coordinates, and node path of the point. Then use these points to calculate distances or other things.

But I found that using LineSegmentIntersector cannot achieve such functionality on point clouds. It seems that it can only be used on 3D models.

When I zoom in on the point cloud until the sphere occupies more pixels, ensuring that I can select a certain point, there is still no intersection generated.

I conducted a test and found that when the point cloud and a 3D model of a teapot appear in the scene simultaneously, the model can work normally, while the point cloud cannot.

Here are my test results:

1. The 3D model in the bottom left corner can be obtained
2. I couldn't find the point in the upper right corner, even though I aimed at the sphere

May I ask how to obtain the coordinates, index, and nodes path of the point cloud corresponding to the pixel points?

[mingkangzh]

I have found the problem.

It is determined here that topology must be VK_ PRIMITIVE_ TOPOLOGY_ TRIANGLE_ LIST and vertexCount must be less than 3.

I don't understand the source code part. How can I make modifications to make LineSegmentIntersector work on a point cloud?

I will try to modify it tomorrow. This is Asia, I should go to bed now.

[robertosfield]

LineSegmentIntersector does not handle line and point datasttets, it crrently only support triangle meshes.

Intersecting points and lines which are infinitely small with a line segment that is infinity thin is not something that can be made to work, to handle these types of data you have do something different than is done for triangle mesh.

There are several ways one tackle this issue, essentially they all revolve around give the intersector primitive a thickness, and/or given the point or line a radius or thickness. One approach would be to have a polytope, cone or cylinder intersector that you check with a point or line segment is within the cone or cylinder.

Another approach would be to give the point or line radius/thickness so the line segment can then check if the line segment intersects it.

What approach to tackle will depend upon what problem you are trying to tackle. Is it picking points/lines on screen? Doing general intersection test?

[mingkangzh]

Thank you for your answer. I believe the method you provided can solve my problem.

Due to the fact that point clouds are unordered and unstructured, I will use the latter method to determine the distance from a point to a straight line, which is the line segment with a radius that you proposed.

What I am currently doing is picking points and lines, as well as corresponding nodes, in order to achieve user interaction, camera center reset, point cloud center reset, model creation at specified locations, point cloud registration, point cloud stitching, and other functions.

The first method you proposed will be something I will consider using in the future. On a structured (octree) point cloud, perform noise removal, filtering, stitching, and then encapsulate it into a polyhedron to achieve transformation from the point cloud to the 3D model.

I will create a class for LinePointIntersector, modeled after LineSegmentIntersector, and convert VK_ PRIMITIVE_ TOPOLOGY_ TRIANGLE_ LIST to VK_PRIMITIVE_TOPOLOGY_POINT_LIST, and then implement something similar to TriangleIntersectorto determine whether a point intersects a line segment with a radius.

I have another question, how can I achieve the selection of some points in the point cloud?

What I am currently thinking of is:

1. I draw an area on the window;
2. Obtain the model matrix of the point cloud through recursion;
3. Use the model matrix, camera's viewMatrix, and projectionMatrix to map the width and height of the window for each point in the point cloud, simulate the process of projecting the point cloud onto the window, and determine whether it can be projected onto the area I drew in the first step to achieve the selection function.

Excuse me, are these steps working? Or do you have any other good ideas?

[mingkangzh]

I have completed the point picking and worked like the LineSegmentIntersector to obtain all the data,

But instead of using line segments with radii, I simulated the process of projecting a point cloud onto a window to determine if the point was around my mouse.

The reason for doing so is:

When using a line segment with a radius, I must constantly zoom in on the point cloud to ensure that the mouse can click on it. If the point cloud is not enlarged, then the radius needs to be increased, resulting in obtaining many incorrect points.

The method of projecting a point cloud onto a window to determine whether it falls around the mouse does not require zooming in on the point cloud to select one of the points, or even requiring the mouse to point the midpoint. Judging the radius of pixels rather than the radius of line segments can also ensure that I get the correct point.

Thank you. Thanks to this issue, I should be able to easily implement box selection or region selection for some points in the point cloud.

[robertosfield]

Good to hear you've got a solution.

Longer term I'd like to see the VSG have additional Intersector implementation like the OSG has. I would also like to add acceleration structures such as KdTree or similar scheme. There are some parallels with the needs of ray tracing on the GPU, while there wouldn't be a full overlap in functionality I am curious whether there is enough overalp to share some of the public interface for settings these things up.

I have plenty of other tasks on my plate right now so I'm not in position to start experimenting with any of the above, hopefully over time I or others will be able to start tackling them for the core VSG.

[jamierobertson1]

For your second question about region selection, an alternate way is to create a vsg::Intersector subclass and create a series of convex polytopes in eye space by projecting the selection points to the the cameras near and far planes). Then maintain a stack of the polytopes transformed into the nodes local coordinate system using the pushTransform and popTransform functions, and then test whether the individual points are inside the local polytope.

[mingkangzh]

Great idea! It's like LineSegmentIntersector.
But determining whether a point is within a convex polytope is a mathematical problem that I find difficult to understand.

[jamierobertson1]

Checking whether a point is within a convex polytope is pretty simple, so long as the edges are defined in order, if the points are all on the same side of each edge plane (i.e. the distance from point to plane is the same sign), then the point is inside.

What's more difficult is dealing with concave polygons, self intersecting polygons etc., although It's possible to decompose these into multiple convex polygons.

[mingkangzh]

Thank you for your reply. I have completed the click and box selection using Qt.

I encountered some problems during this process.

I customized and inherited a QWidget, converted vsgQt (QWindow) into a QWidget, and then handed it over to me as a custom QWidget object for layout.

When I rewrite paintEvent to draw a polygon composed of mouse movement points, it always does not display this polygon.

I think this is because the rendering of vsgQt is controlled by the QTimer timer, which will clear the content of my paintEvent to display the vsgQt window.

My current implementation is to take a screenshot of the vsgQt window when clicking on the box selection, and display the result of the screenshot in another window; After completing the bounding box selection, calculate whether the point cloud falls in the bounding box area and switch the window back to the vsgQt window.

It seems that there is no need for a vsgQt window. When needed, I can obtain real-time off screen rendered images and submit them to the QWidget for display, making it easier for me to use Qt's functions to display and draw other things, such as box selection, based on this image.

So, is there any way to remove the window of vsgQt and instead render it in real-time as an image?

Preserve everything except the window, turn the window display into a image, and users can implement some functions of the window through the interface. For example, interfaces for mouse or keyboard events, resizing, and other UI operations can be provided. Keep the default three frame buffers, reconstruct the swapchains, and so on.

[robertosfield]

The vsgQt::Viewer uses a timer in order to have a easy way of implementing continuous rendering. It's a pretty lighweight class so you should be able to write your own equivalent without timer if that suits you application.

vsgQt::Window uses the VSG to provide Vulkan support because Qt's own Vulkan support is pretty poor, I did try to utilize Qt's Vulkan support but found it broken in too many ways to use for proper applications, so had to utilize the VSG for creating the Vulkan Instance and Surface. The VSG's is all about exposing the full capabilities of Vulkan, while Qt support for Vulkan looks to be a quick hack done without full understanding of Vulkan.

You could use the VSG/Vulkan using off screen rendering and then copy the image to the Qt window but this would add an performance overhead.

As for selection areas etc. You may be better off implementing these features in VSG/Vulkan rather than Qt.

[mingkangzh]

How to draw pixel by pixel? For example, if I want to draw the path of my mouse movement in VSG, is there any good way?

After rendering the nodes of the VSG scene graph into textures, and then drawing the selected pixels onto the texture?

Is it possible to use utility functions such as drawing lines, circles, ellipses, paths, and polygons on the scene after rendering?

[robertosfield]

One doesn't normal pixel by pixel with 3D graphics, but I guess you could use a compute shader to do it, it would be rather inefficient.

The normal way to draw something like the path of the mouse is create a line and render this as an overlay after the main 3D window contents have been drawn. The overlay would typically be done as a orthographic View. vsgortho and vsgoverlay would be good examples to look at for inspiration for this.

GPUs like statically allocated data so one would create a vertex array with enough points to contain all the points you need, then update the number of vertices to draw within this array as you add new points. A vsg::VertexDraw could provide the scene graph level object for managing the vertex array and the number of vertices to draw in the line. Unfortunately there isn't a VSG example that illustrates this but others have successfully used this approach in their applications.

This overlay approach can be used for all object types - you can rendering points, lines, 3d mesh etc. all the things you can do in 3d window as illustrated by the vsgoverlay example.

One can also do techniques like creating a mesh from your selection polygon, render this mesh (and another over details) to texture, then when rendering the 3d view use this rendered texture as an input texture. There is a huge number of possibilities of how to do things.

With selection operation you potentially you use occlussion query, where you draw the selection region then when you render the 3d scene you test the objects to see if they are occluded by the what has been rendered into the depth buffer. There is an ovsgoverlayquery example that illustrates this Vulkan feature.

Selection on the CPU can be done and the Polytope approach that @jamierobertson1 mentioned is actually quite straight forward. A Polytope is typically represented by a std::vectorvsg::plane / std::vectorvsg::dplane, each plane is set up so that it''s aligned to the edge of the polygon and the viewing direction with the normal pointing inwards. You then test each vertex distance above each plane in the Polytope and if all are positive then it's inside the Polytope.

The OSG has a PolytopeIntersector that uses this approach, we just haven't yet implemented such a technique in the VSG, the implementation would be in principle the same.

[mingkangzh]

Thank you for your answer. I have placed many transparent windows for layer Qt on the QWindow of vsgQt, which supports me to draw and display Qt Widegt on the transparent window.

[robertosfield]

If you embracing Qt for this type of functionality rather than the VSG then it's not area of expertise and can't help you.

Personally I'd be wary of relying upon Qt too much as you could end up just trying implement things is ways that fit with Qt rather than what are the most efficient ways to do it.