🐛 runAsync has memory problem

[xulihang]

What's happening?

Using runAsync will consume a lot of memory.

Reproduceable Code

const frameProcessor = useFrameProcessor(frame => {
    'worklet'
    runAsync(frame, () => {
      'worklet'
      let results;
      if (engine.value === "ZXing") {
        results = zxing(frame,{multiple:true});
      }else if (engine.value === "Dynamsoft") {
        results = decode(frame,{rotateImage:false});
      }else{
        results = scanBarcodes(frame);
      }
      
      console.log(results);
      if (results) {
        convertAndSetResultsJS(results as Record<string,object>,frame.width,frame.height,frame.orientation);
      }
      
    })
  }, [])

Relevant log output

2024-09-19 17:49:59.123554+0800 BarcodeScanner[2259:110842] Successfully registered Frame Processor Plugin "scanBarcodes"!
2024-09-19 17:49:59.124328+0800 BarcodeScanner[2259:110842] Successfully registered Frame Processor Plugin "decode"!
2024-09-19 17:49:59.124397+0800 BarcodeScanner[2259:110842] Successfully registered Frame Processor Plugin "zxing"!

Camera Device

Not needed in this case.

Device

iPhone SE (iOS 15)

VisionCamera Version

4.5.3

Can you reproduce this issue in the VisionCamera Example app?

No, I cannot reproduce the issue in the Example app

Additional information

• I am using Expo
• I have enabled Frame Processors (react-native-worklets-core)
• I have read the Troubleshooting Guide
• I agree to follow this project's Code of Conduct
• I searched for similar issues in this repository and found none.

[maintenance-hans]

Guten Tag, Hans here 🍻.

Thanks for your detailed report! It looks like you've done some good debugging steps already. However, I noticed you mentioned you cannot reproduce this issue in the example app, which makes it tricky for mrousavy to investigate further.

Could you provide us with additional logs from Xcode (or adb logcat for Android) when this memory problem occurs? This information will help us better understand ze issue.

Also, if you find ze project useful, consider sponsoring mrousavy to support its ongoing development and maintenance. Thank you!

Note: If you think I made a mistake, please ping @mrousavy to take a look.

[xulihang]

Using runAtTargetFps works fine. My sample project: https://github.com/xulihang/react-native-zxing-mlkit-dynamsoft

[nmajumder12]

runAsync is known to problematic due to the a bug in the dependency RNWC.

Does the app crash after a while?

[xulihang]

Yes. It finally crashed after the memory usage exceeded 1.4GB.

[nmajumder12]

Might be related to these then, RNWC bug

#2820

margelo/react-native-worklets-core#173

[kuldip-simform]

I am getting issue with runAtTargetFps as well.
It is not removing memory even if I stop frame processor.
@xulihang Do you faced this issue.

const updateMetricsData = async (resized: MetricData) => {
    if (model === null) {
      return;
    }

    metrcsDataRef.current = [...metrcsDataRef.current, resized];
    // setMetricsDataList(prev => {
    //   // Check if the length of the array is 120 or more
    //   if (prev.length >= 40) {
    //     // Remove the first element and add the new one at the end
    //     return [...prev.slice(1), resized];
    //   }
    //   // Just add the new one
    //   return [...prev, resized];
    // });
  };

  const updateMetricsDataJS = Worklets.createRunOnJS(updateMetricsData);


const frameProcessor = useFrameProcessor(
    frame => {
      'worklet';
      if (!isScanning) {
        return;
      }
      runAtTargetFps(60, () => {
        const faces = detectFaces(frame);
        handleDetectedFaces(faces);
        if (faces.length > 0) {
          const firstFace = faces[0];

          const { yawAngle, pitchAngle, rollAngle, bounds } = firstFace;

          if (
            Math.abs(yawAngle) > TILTED_ANGLE ||
            Math.abs(pitchAngle) > TILTED_ANGLE ||
            Math.abs(rollAngle) > TILTED_ANGLE
          ) {
            handleFaceTitled(true);
          } else {
            handleFaceTitled(false);
            if (
              bounds.x > 10 &&
              bounds.y > 10 &&
              bounds.width + bounds.x < frame.height - 20 &&
              bounds.height + bounds.y < frame.width - 20
            ) {
              handleFaceIsInCenter(true);
              if (
                bounds.width > frame.height * 0.4 &&
                bounds.width < frame.height * 0.85
              ) {
                handleFaceIsFarNear(false);
                const resized = resize(frame, {
                  // crop: {x: 10, y: 10, width: 400, height: 400},
                  crop: {
                    x: bounds.y,
                    y: bounds.x,
                    width: bounds.height,
                    height: bounds.width,
                  },
                  scale: { width: FRAME_WIDTH, height: FRAME_HEIGHT },
                  pixelFormat: 'rgba',
                  dataType: 'float32',
                  // rotation: '90deg',
                });
                // runOnJS(updateFrameData)(resized);
                const reshapedArray: MetricData = [];
                let index = 0;
                for (let h = 0; h < FRAME_HEIGHT; h++) {
                  const row: number[][] = [];
                  for (let w = 0; w < FRAME_WIDTH; w++) {
                    // Extract RGB values for the current pixel
                    const pixel = Array.from(
                      resized.slice(index, index + CHANNEL),
                    );
                    row.push(pixel);
                    index += CHANNEL;
                  }
                  reshapedArray.push(row);
                }
                updateMetricsDataJS(reshapedArray);
              } else {
                handleFaceIsFarNear(true);
              }
            } else {
              handleFaceIsInCenter(false);
            }
          }
        }
      });
    },
    [isScanning],
  );

If run at lower fps it just delay app crashing but app will crash eventually.

[xulihang]

I don't have a problem with runAtTargetFps. The memory usage will go down.

[kuldip-simform]

@xulihang Can you spot anything unusual that I might be doing from my code ?