Grabbing

[bbohlender]

Since this library contains the pointing interaction, I think a grabbing interaction would be a very good fit to implement in here. Especially cause both Interaction paradigms should possibly coexist regardless of the input (Controller, Hand, Mouse, ...).

Maybe we can discuss If Grabbing would fit into this library and how it can interoperate with the current pointing interaction.

(I've already started a POC for two- & one-handed grabbing but It would need some refinement to be submitted as a PR)

[CodyJasonBennett]

I'm curious as to what this would look like. We have the <RayGrab /> component, but it's not very flexible.

[bbohlender]

Me too :D
And the POC I was talking about is 1 year old. Anyways here's gist that shows the code we used a year ago.
https://gist.github.com/bbohlender/d7531b86a1e0899d879ed287b13171b7

However, today I would prefer event abstractions like onGrabStart, onGrabMove, onGrabEnd. Especially for Multiuser a differentiation between state and events is much more helpful. I think in general react-xr should provide a good event abstraction like what react-gesture does for mouse and touch events, but now for mouse, touch and controllers. Additionally there could be something like react-xr-drei that has convenience components like SingleHandGrabbable / MultiHandGrabbable that simply transform the object / group.

Unfortunately I do not yet have time to get my hands dirty on this

[micmania1]

I've been playing about with grabbing too! Before I jumped into it I had a look at what was already provided by existing interactions. My take was that they all expect some form of ray to be intersecting the object rather then the user interacting with the object directly. In short: Good for raycasting events, but not extensible beyond that.

I've copied a couple of files (warts and all) to a gist from my project to demonstrate my approach. The key file being XRInteractions.tsx . Level1.tsx is kinda like user-land implementation but does have some of the grabbing logic - that part needs a bit more thought & cleanup.

Code Outline

A little bit of commentry on what I've done here: https://gist.github.com/micmania1/6a189a51691b8239826a0a3ecad7f0d5

XRInteractions.tsx
intersect - a simple helper function to check for collisions between two boxes - no support for different shapes yet.

useXRInteractionManager - A zustand store used to track interaction states as well as controller states (ie. a controller can only be doing so much at once).
This fires two events on the interactable objects when appropriate: grabstart and grabend

XRInteraction - A react component which is reliant upon userXRInteractionManager. This is responsible for making objects interactable by translating standard XR events into human gestures (ie. squeeze event into a grab).

Level1.tsx
In my project this contains all of the context for a single level, but in this gist the important part is in the Ball.tsx which has been plonked at the top of the file.

The ball uses physics through cannon-es which is why the grab events are being handled by the component - ie. any library wouldn't know if physics was available or not, so grabbing & positioning the ball may require different steps. Maybe any library implementation would depend on physics?

Requirements

I'm still discovering 3D development so I tried to set the barrier pretty low for what I was trying to achieve to begin with. No fancy double hand grabs or grabbing multiple items at once.

Here's my basic requirements:

• A single controller can only pick up a single item at a time
• A grab is detected by a squeezestart event and triggers when a registered object is intersecting with the controller model in 3D space - This triggers the grabstart event
• A grab is ended when the squeezeend event is triggered on a controller which is already grabbing an object - This triggers the grabend event.

Some things I purposely ignored were:

• multi-hand grabbing
• grabbing multiple items at once (although two separate hands should work)
• prioritising which item to grab if there are two within distance - it'll pick one and release the other in whichever order the evnts occur in
• throwing the object, although that's what I'll be tackling next!

Example Usage

const ref = useRef();
useEffect(() => {
  const grabStart = () => console.log('grab start!');
  ref.current.addEventListener('ongrabstart', grabStart);
  // ... etc.
  return () => {
    ref.current.removeEventListener('ongrabstart', grabEnd);
    // ... etc.
  }
});

<XRInteraction ref={ref}>
  <MyObject />
</XRInteraction>

I could extend this to add onGrabStart/End props too to make it a little nicer to use.

Video example

com.oculus.shellenv-20220727-222329_Trim.mp4

[taeuscherpferd]

I would love to see this implemented as well. I really enjoy working with this framework, but it's far too lacking to be able to make anything serious in VR. Adding built in grabbing support would go a long way to boosting the usefulness of the framework. Is anybody against grabbing? If someone were to submit a pull request, would it be shot down out of principle or does it have a good chance of going through?

[CodyJasonBennett]

I'd be happy to accept a PR. We could use some work on interactions, and I'd have to concur as to why.

[bbohlender]

After 3 years I'll close this issue myself :D
The groundwork regarding interactions has been laid so that controls like Grabbing and more complex, cross-platform, extensible interactions can be implemented. As shown in the roadmap of the new Readme, a library called @react-three/controls is planned, which will allow highly configurable and versatile Grabbing, as well as other controls such as cross platform OrbitControls that respect the event system of the scene.

[geyang]

For my application, I am planning on implementing the following gestures today:

• grab: this is equivalent to squeeze on the MotionController. I detect three minor finger squeeze.
• pinch: this is using the index finger and the thumb
• rayGrab
• rayPinch

Having grab different from pinch is needed for manipulating a robot gripper.