Worklet Frame Processor Crash Android 🐛

[pvedula7]

What were you trying to do?

I am using a custom frame processor plugin that does pose detection. On IOS it works with no issues. On Android there are two issues I am facing.

1. The front camera does not trigger the worklet whatsoever, only the back camera triggers the if statement of isActive and modalVisible.
2. The scanSKRNMLKitPose will successfully detect poses from the native library. However, setting any form of state with runOnJS causes the app to crash with no errors. I have double checked that the output on IOS and Android is exactly the same.

const [poses, setPoses] = useState<any[]>([]);
const frameProcessor = useFrameProcessor((frame) => {
    "worklet";
    if (isActive && modalVisible) {
      const pose = scanSKRNMLKitPose(frame);
      runOnJS(setPoses)(pose);
      runOnJS(updateFrameSize)(frame.width, frame.height);
    }
  }, []);

The variable pose is an object such as this

Array [
  Object {
    "LeftAnkle": Object {
      "inFrameLikelihood": 0.9808423519134521,
      "position": Object {
        "x": 1131.0230712890625,
        "y": 312.59228515625,
        "z": -645.5625,
      },
      "type": "LeftAnkle",
    },
    "LeftEar": Object {
      "inFrameLikelihood": 0.989250898361206,
      "position": Object {
        "x": 1070.3919677734375,
        "y": 793.01220703125,
        "z": -1249.582763671875,
      },
      "type": "LeftEar",
    },
    "LeftElbow": Object {
      "inFrameLikelihood": 0.9963029623031616,
      "position": Object {
        "x": 956.1546630859375,
        "y": 540.278564453125,
        "z": -1312.270263671875,
      },
      "type": "LeftElbow",
    },
    "LeftEye": Object {
      "inFrameLikelihood": 0.9696589708328247,
      "position": Object {
        "x": 1087.89990234375,
        "y": 774.898193359375,
        "z": -1475.7806396484375,
      },
      "type": "LeftEye",
    },
    "RightWrist": Object {
      "inFrameLikelihood": 0.9745367765426636,
      "position": Object {
        "x": 1263.23291015625,
        "y": 461.745849609375,
        "z": -1935.74267578125,
      },
      "type": "RightWrist",
    },
  },
]

Is there something that needs to be done differently for Android?

Reproduceable Code

The reanimated camera view is as shown below.

{device != null && isActive && modalVisible && (
        <PinchGestureHandler onGestureEvent={onPinchGesture} enabled={isActive}>
          <Reanimated.View style={StyleSheet.absoluteFill}>
            <ReanimatedCamera
              ref={camera}
              style={StyleSheet.absoluteFill}
              device={device}
              format={format}
              fps={60}
              hdr={enableHdr}
              lowLightBoost={device.supportsLowLightBoost && enableNightMode}
              isActive={isActive}
              onInitialized={onInitialized}
              onError={onError}
              enableZoomGesture={false}
              animatedProps={cameraAnimatedProps}
              photo={true}
              video={true}
              audio={hasMicrophonePermission}
              frameProcessor={
                device.supportsParallelVideoProcessing && isActive
                  ? frameProcessor
                  : undefined
              }
              orientation="portrait"
              frameProcessorFps={60}
            />
          </Reanimated.View>
        </PinchGestureHandler>
      )}

What happened instead?

App crashes when running camera.

Relevant log output

No response

Device

Samsung Galaxy S21

VisionCamera Version

2.15.2

Additional information

• I am using Expo
• I have read the Troubleshooting Guide
• I agree to follow this project's Code of Conduct
• I searched for similar issues in this repository and found none.

[myselfuser1]

Try this https://www.youtube.com/playlist?list=PLQhQEGkwKZUrempLnmxjt7ZCZJu1W3p2i

[mrousavy]

Hey! I've rewritten the entire Android codebase of VisionCamera from CameraX to Camera2 in the efforts of ✨ VisionCamera V3.

I just now completed the Camera2 rewrite and I believe the core structure is running, but there might be some edge cases to iron out. Can you try and test the PR #1674 for me to see if you can still reproduce this issue here?

Here's an instruction on how you can test that: #1674 (comment)

If the issue cannot be reproduced with that version/PR anymore, then hoorayy, I fixed it! 🎉
Otherwise please let me know and I'll keep this issue open to keep track of it.

Thank you!