Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Use worklet(...) to automatically detect useFrameProcessor dependencies #2840

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/docs/guides/EXPOSURE.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -49,12 +49,12 @@ Just like [`zoom`](zooming), this property can be animated using Reanimated.
return interpolate(exposureSlider.value,
[-1, 0, 1],
[device.minExposure, 0, device.maxExposure])
}, [exposureSlider, device])
})

// 3. pass it as an animated prop
const animatedProps = useAnimatedProps(() => ({
exposure: exposureValue.value
}), [exposureValue])
}))

// 4. render Camera
return (
Expand Down
10 changes: 5 additions & 5 deletions docs/docs/guides/FRAME_PROCESSORS_CREATE_OVERVIEW.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ function App() {
// highlight-next-line
const faces = detectFaces(frame)
console.log(`Faces in Frame: ${faces}`)
}, [])
})

return (
<Camera frameProcessor={frameProcessor} {...cameraProps} />
Expand Down Expand Up @@ -91,7 +91,7 @@ const frameProcessor = useFrameProcessor((frame) => {
// by downscaling the frame, the `detectObjects` function runs faster.
const objects = detectObjects(resizedFrame)
console.log(objects)
}, [])
})
```

### Parameters
Expand All @@ -102,7 +102,7 @@ Frame Processors can also accept parameters, following the same type convention
const frameProcessor = useFrameProcessor((frame) => {
'worklet'
const faces = scanFaces(frame, { accuracy: 'fast' })
}, [])
})
```

### Exceptions
Expand Down Expand Up @@ -131,7 +131,7 @@ const frameProcessor = useFrameProcessor((frame) => {
} catch (e) {
console.log(`Error: ${e.message}`)
}
}, [])
})
```

## What's possible?
Expand Down Expand Up @@ -177,7 +177,7 @@ function App() {
const frameProcessor = useFrameProcessor((frame) => {
'worklet'
SomeAI.process(frame) // does not block frame processor, runs async
}, [])
})

useEffect(() => {
SomeAI.addListener((results) => {
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/guides/FRAME_PROCESSORS_CREATE_PLUGIN_CPP.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ const frameProcessor = useFrameProcessor((frame) => {
'worklet'
const result = global.myCppPlugin(frame)
console.log(`C++ result: ${result}`) // <-- 42
}, [])
})
```

To include VisionCamera's C++ library in your plugin's C++ code, you need to link it and include the headers.
Expand Down
10 changes: 5 additions & 5 deletions docs/docs/guides/FRAME_PROCESSORS_INTERACTING.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ const frameProcessor = useFrameProcessor((frame) => {
const objects = detectObjects(frame)
const bananas = objects.filter((o) => o.type === targetObject)
console.log(`Detected ${bananas} bananas!`)
}, [targetObject])
})
```

### Shared Values
Expand All @@ -45,7 +45,7 @@ const frameProcessor = useFrameProcessor((frame) => {
'worklet'
const objects = detectObjects(frame)
bananas.value = objects.filter((o) => o.type === 'banana')
}, [bananas])
})

// Draw bananas in a Skia Canvas
const onDraw = useDrawCallback((canvas) => {
Expand Down Expand Up @@ -76,7 +76,7 @@ const frameProcessor = useFrameProcessor((frame) => {
if (faces.length > 0) {
onFaceDetected(faces[0])
}
}, [onFaceDetected])
})
```

## Threading
Expand All @@ -101,7 +101,7 @@ const frameProcessor = useFrameProcessor((frame) => {
console.log("I'm running asynchronously, possibly at a lower FPS rate!")
const faces = detectFaces(frame)
})
}, [])
})
```

### Running at a throttled FPS rate
Expand All @@ -118,7 +118,7 @@ const frameProcessor = useFrameProcessor((frame) => {
console.log("I'm running synchronously at 2 FPS!")
const brightness = detectBrightness(frame)
})
}, [])
})
```

#### 🚀 Next section: [Zooming](/docs/guides/zooming) (or [creating a Frame Processor Plugin](/docs/guides/frame-processors-plugins-overview))
4 changes: 0 additions & 4 deletions docs/docs/guides/FRAME_PROCESSORS_TIPS.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -38,10 +38,6 @@ If you use native Frame Processor Plugins, make sure they are optimized for real
- Prefer plugins that support **GPU acceleration**. For Tensorflow, this might be the CoreML or Metal GPU delegates
- For operations such as resizing, **prefer GPU or CPU vector acceleration** (e.g. Accelerate/vImage) instead of just array loops

## ESLint react-hooks plugin

If you are using the [react-hooks ESLint plugin](https://www.npmjs.com/package/eslint-plugin-react-hooks), make sure to add `useFrameProcessor` and `useSkiaFrameProcessor` to `additionalHooks` inside your ESLint config so dependencies are detected properly. (See ["advanced configuration"](https://www.npmjs.com/package/eslint-plugin-react-hooks#advanced-configuration))

## Technical

### Frame Processors
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/guides/FRAME_PROCESSOR_CREATE_FINAL.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ function App() {
// highlight-next-line
const faces = scanFaces(frame)
console.log(`Faces in Frame: ${faces}`)
}, [])
})

return (
<Camera frameProcessor={frameProcessor} {...cameraProps} />
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/guides/PIXEL_FORMATS.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ function App() {
const frameProcessor = useFrameProcessor((frame) => {
'worklet'
console.log(frame.pixelFormat) // <-- "rgb"
}, [])
})

return (
<Camera
Expand Down
5 changes: 1 addition & 4 deletions docs/docs/guides/ZOOMING.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -92,10 +92,7 @@ export function App() {
})

// highlight-start
const animatedProps = useAnimatedProps<CameraProps>(
() => ({ zoom: zoom.value }),
[zoom]
)
const animatedProps = useAnimatedProps<CameraProps>(() => ({ zoom: zoom.value }))
// highlight-end

if (device == null) return <NoCameraDeviceError />
Expand Down
8 changes: 0 additions & 8 deletions package/.eslintrc.js
Original file line number Diff line number Diff line change
Expand Up @@ -74,14 +74,6 @@ module.exports = {
'@typescript-eslint/no-non-null-assertion': 'error',
'@typescript-eslint/no-unnecessary-condition': 'error',
'@typescript-eslint/consistent-type-imports': 'warn',

// react hooks
'react-hooks/exhaustive-deps': [
'error',
{
additionalHooks: '(useDerivedValue|useAnimatedStyle|useAnimatedProps|useWorkletCallback|useFrameProcessor|useSkiaFrameProcessor)',
},
],
},
env: {
node: true,
Expand Down
4 changes: 2 additions & 2 deletions package/example/ios/Podfile.lock
Original file line number Diff line number Diff line change
Expand Up @@ -337,7 +337,7 @@ PODS:
- react-native-video/Video (= 5.2.1)
- react-native-video/Video (5.2.1):
- React-Core
- react-native-worklets-core (1.2.0):
- react-native-worklets-core (1.3.2):
- React
- React-callinvoker
- React-Core
Expand Down Expand Up @@ -684,7 +684,7 @@ SPEC CHECKSUMS:
react-native-safe-area-context: 0ee144a6170530ccc37a0fd9388e28d06f516a89
react-native-skia: d368ae81d61c0253df36106ea76324b5a5519ddc
react-native-video: c26780b224543c62d5e1b2a7244a5cd1b50e8253
react-native-worklets-core: a316975bba20b73d47aa4d41bffb0ca984ba592e
react-native-worklets-core: ff4a4076ad86469e61ec8e39da2cb13c55da2385
React-NativeModulesApple: b6868ee904013a7923128892ee4a032498a1024a
React-perflogger: 31ea61077185eb1428baf60c0db6e2886f141a5a
React-RCTActionSheet: 392090a3abc8992eb269ef0eaa561750588fc39d
Expand Down
2 changes: 1 addition & 1 deletion package/example/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@
"react-native-static-safe-area-insets": "^2.2.0",
"react-native-vector-icons": "^10.0.0",
"react-native-video": "^5.2.1",
"react-native-worklets-core": "^1.2.0"
"react-native-worklets-core": "^1.3.2"
},
"devDependencies": {
"@babel/core": "^7.22.10",
Expand Down
4 changes: 2 additions & 2 deletions package/example/src/CameraPage.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@ export function CameraPage({ navigation }: Props): React.ReactElement {
return {
zoom: z,
}
}, [maxZoom, minZoom, zoom])
})
//#endregion

//#region Callbacks
Expand Down Expand Up @@ -187,7 +187,7 @@ export function CameraPage({ navigation }: Props): React.ReactElement {
examplePlugin(frame)
exampleKotlinSwiftPlugin(frame)
})
}, [])
})

const videoHdr = format?.supportsVideoHdr && enableHdr
const photoHdr = format?.supportsPhotoHdr && enableHdr && !videoHdr
Expand Down
27 changes: 12 additions & 15 deletions package/example/src/views/CaptureButton.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -196,20 +196,17 @@ const _CaptureButton: React.FC<Props> = ({
})
//#endregion

const shadowStyle = useAnimatedStyle(
() => ({
transform: [
{
scale: withSpring(isPressingButton.value ? 1 : 0, {
mass: 1,
damping: 35,
stiffness: 300,
}),
},
],
}),
[isPressingButton],
)
const shadowStyle = useAnimatedStyle(() => ({
transform: [
{
scale: withSpring(isPressingButton.value ? 1 : 0, {
mass: 1,
damping: 35,
stiffness: 300,
}),
},
],
}))
const buttonStyle = useAnimatedStyle(() => {
let scale: number
if (enabled) {
Expand Down Expand Up @@ -246,7 +243,7 @@ const _CaptureButton: React.FC<Props> = ({
},
],
}
}, [enabled, isPressingButton])
})

return (
<TapGestureHandler
Expand Down
8 changes: 4 additions & 4 deletions package/example/yarn.lock
Original file line number Diff line number Diff line change
Expand Up @@ -5233,10 +5233,10 @@ react-native-video@^5.2.1:
prop-types "^15.7.2"
shaka-player "^2.5.9"

react-native-worklets-core@^1.2.0:
version "1.2.0"
resolved "https://registry.yarnpkg.com/react-native-worklets-core/-/react-native-worklets-core-1.2.0.tgz#155040a48d1e3f66107544867ec579c2f5044271"
integrity sha512-Uuf1Hg4V1w4I8eYNinwnnf+h+2YdQgwkUDQszY2yEEXWPWcZBWec0xjYn079RFRcKmGWuJ/zOtokjwy/IjJ9Fw==
react-native-worklets-core@^1.3.2:
version "1.3.2"
resolved "https://registry.yarnpkg.com/react-native-worklets-core/-/react-native-worklets-core-1.3.2.tgz#0851ad7b524eae152d63adc20ad626eabecd9f26"
integrity sha512-wx2MdfiUP9gBluhtzrb5XeiAGxdnqVqosLCPIhP1sSF9EcgOKTBV4jxuRkYmZJ/DDcHvGPy0qwdqwyVtWAn3jQ==
dependencies:
string-hash-64 "^1.0.3"

Expand Down
6 changes: 3 additions & 3 deletions package/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -98,16 +98,16 @@
"react-native": "^0.72.3",
"react-native-builder-bob": "^0.21.3",
"react-native-reanimated": "^3.8.1",
"react-native-worklets-core": "^1.2.0",
"react-native-worklets-core": "^1.3.2",
"release-it": "^16.1.3",
"typescript": "^5.1.6"
},
"peerDependencies": {
"@shopify/react-native-skia": "*",
"@shopify/react-native-skia": ">=1.2.3",
"react": "*",
"react-native": "*",
"react-native-reanimated": "*",
"react-native-worklets-core": "*"
"react-native-worklets-core": ">=1.3.2"
},
"peerDependenciesMeta": {
"react-native-worklets-core": {
Expand Down
14 changes: 8 additions & 6 deletions package/src/hooks/useFrameProcessor.ts
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
import type { DependencyList } from 'react'
import { useMemo } from 'react'
import { withFrameRefCounting } from '../frame-processors/withFrameRefCounting'
import type { ReadonlyFrameProcessor } from '../types/CameraProps'
import type { Frame } from '../types/Frame'
import { WorkletsProxy } from '../dependencies/WorkletsProxy'

/**
* Create a new Frame Processor function which you can pass to the `<Camera>`.
Expand All @@ -28,18 +28,20 @@ export function createFrameProcessor(frameProcessor: (frame: Frame) => void): Re
*
* @worklet
* @param frameProcessor The Frame Processor
* @param dependencies The React dependencies which will be copied into the VisionCamera JS-Runtime.
* @returns The memoized Frame Processor.
* @example
* ```ts
* const frameProcessor = useFrameProcessor((frame) => {
* 'worklet'
* const faces = scanFaces(frame)
* console.log(`Faces: ${faces}`)
* }, [])
* })
* ```
*/
export function useFrameProcessor(frameProcessor: (frame: Frame) => void, dependencies: DependencyList): ReadonlyFrameProcessor {
// eslint-disable-next-line react-hooks/exhaustive-deps
return useMemo(() => createFrameProcessor(frameProcessor), dependencies)
export function useFrameProcessor(frameProcessor: (frame: Frame) => void): ReadonlyFrameProcessor {
return useMemo(
() => createFrameProcessor(frameProcessor),
// eslint-disable-next-line react-hooks/exhaustive-deps
WorkletsProxy.getWorkletDependencies(frameProcessor),
)
}
11 changes: 3 additions & 8 deletions package/src/skia/useSkiaFrameProcessor.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
import type { Frame, FrameInternal } from '../types/Frame'
import type { DependencyList } from 'react'
import { useEffect, useMemo } from 'react'
import type { Orientation } from '../types/Orientation'
import type { DrawableFrameProcessor } from '../types/CameraProps'
Expand Down Expand Up @@ -247,7 +246,6 @@ export function createSkiaFrameProcessor(
*
* @worklet
* @param frameProcessor The Frame Processor
* @param dependencies The React dependencies which will be copied into the VisionCamera JS-Runtime.
* @returns The memoized Skia Frame Processor.
* @example
* ```ts
Expand All @@ -260,13 +258,10 @@ export function createSkiaFrameProcessor(
* const rect = Skia.XYWHRect(face.x, face.y, face.width, face.height)
* frame.drawRect(rect)
* }
* }, [])
* })
* ```
*/
export function useSkiaFrameProcessor(
frameProcessor: (frame: DrawableFrame) => void,
dependencies: DependencyList,
): DrawableFrameProcessor {
export function useSkiaFrameProcessor(frameProcessor: (frame: DrawableFrame) => void): DrawableFrameProcessor {
const surface = WorkletsProxy.useSharedValue<SurfaceCache>({})
const offscreenTextures = WorkletsProxy.useSharedValue<SkImage[]>([])

Expand All @@ -292,6 +287,6 @@ export function useSkiaFrameProcessor(
return useMemo(
() => createSkiaFrameProcessor(frameProcessor, surface, offscreenTextures),
// eslint-disable-next-line react-hooks/exhaustive-deps
dependencies,
WorkletsProxy.getWorkletDependencies(frameProcessor),
)
}
2 changes: 1 addition & 1 deletion package/src/types/CameraProps.ts
Original file line number Diff line number Diff line change
Expand Up @@ -314,7 +314,7 @@ export interface CameraProps extends ViewProps {
* 'worklet'
* const faces = scanFaces(frame)
* console.log(`Faces: ${faces}`)
* }, [])
* })
*
* return <Camera {...cameraProps} frameProcessor={frameProcessor} />
* ```
Expand Down
4 changes: 2 additions & 2 deletions package/src/types/Frame.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ import type { PixelFormat } from './PixelFormat'
* const frameProcessor = useFrameProcessor((frame) => {
* 'worklet'
* console.log(`Frame: ${frame.width}x${frame.height} (${frame.pixelFormat})`)
* }, [])
* })
* ```
*/
export interface Frame {
Expand Down Expand Up @@ -73,7 +73,7 @@ export interface Frame {
* const data = new Uint8Array(buffer)
* console.log(`Pixel at 0,0: RGB(${data[0]}, ${data[1]}, ${data[2]})`)
* }
* }, [])
* })
* ```
*/
toArrayBuffer(): ArrayBuffer
Expand Down
Loading
Loading