🐛 Wrong format on ANDROID.

[siddharth-kt]

What were you trying to do?

I was trying to change format prop of camera(react-native-vision-camera) on android.

Reproduceable Code

...

 const formats = useMemo(() => {
    if (device?.formats == null) return [];
    return device.formats;
}, [device?.formats]);
const format = useMemo(() => {
    console.log(formats.length) // it returns 136
    return formats[80]; // its like 1280 * 960
}, [formats]);

...

return (
    <View 
        style={{
            flex: 1,
            justifyContent: 'flex-end',
            backgroundColor: 'black',
        }}>
        <View 
            style={{
                width: '100%',
                aspectRatio: 9 / 16,
                backgroundColor: 'black',
                borderRadius: 12,
                overflow: 'hidden',
            }}>
            <Camera
                ref={camera}
                style={[StyleSheet.absoluteFill, {backgroundColor: 'black'}]}
                device={device}
                isActive={isActive}
                orientation="portrait"
                photo={true}
                video={true}
                enableZoomGesture={true}
                audio={props.hasMicrophonePermission}
                onInitialized={onInitialized}
                onError={onError}
                format={format}
            />
        ...
        </View>
    </View>
)

What happened instead?

As per my device specifications react-native-vision-camera automatically picks {"height": 2160, "width": 3840} for photo/video capturing (ANDROID). But since the user of my app needs to post that video/photo to my backend server so i want user to send maximum {"height": 720, "width": 1280} file to my server. Accepting large files leads to various issues.
Then i tried to change format prop of camera(react-native-vision-camera) on android. But no matters whatever format i pick, camera always set it to {"height": 640, "width": 480}.
I think this is really BIG ISSUE. Kindly tell me if i am doing anything wrong.

Relevant log output

No response

Device

Vivo v15 pro

VisionCamera Version

2.12.0

Additional information

• I am using Expo
• I have read the Troubleshooting Guide
• I agree to follow this project's Code of Conduct
• I searched for similar issues in this repository and found none.

[mrousavy]

Can you try the following:

            <Camera
                ref={camera}
                style={[StyleSheet.absoluteFill, {backgroundColor: 'black'}]}
                device={device}
                isActive={isActive}
                orientation="portrait"
                photo={true}
                video={true}
                enableZoomGesture={true}
                audio={props.hasMicrophonePermission}
                onInitialized={onInitialized}
                onError={onError}
                format={{ ...format, width: format.height, height: format.width }}
            />

And see if that works?

[siddharth-kt]

Thanks @mrousavy ,

I tried it but my format actually returns :-
{"autoFocusSystem": "none", "colorSpaces": ["jpeg"], "fieldOfView": 68.38481019665146, "frameRateRanges": [{"maxFrameRate": 30, "minFrameRate": 1}, {"maxFrameRate": 10, "minFrameRate": 10}, {"maxFrameRate": 15, "minFrameRate": 15}, {"maxFrameRate": 28, "minFrameRate": 8}, {"maxFrameRate": 28, "minFrameRate": 28}, {"maxFrameRate": 30, "minFrameRate": 8}, {"maxFrameRate": 30, "minFrameRate": 30}], "isHighestPhotoQualitySupported": false, "maxISO": 800, "maxZoom": 10, "minISO": 50, "photoHeight": 960, "photoWidth": 1280, "pixelFormat": "420v", "supportsPhotoHDR": false, "supportsVideoHDR": false, "videoHeight": 960, "videoStabilizationModes": ["off", "auto", "standard"], "videoWidth": 1280}

So we can not get format.height or format.width. (these values are undefined)
Or am i doing anything wrong.

But i tried with
format={{
...format,
width: format.photoWidth,
height: format.photoHeight,
}}
but it also did'nt improved anything i am still getting image with {"height": 640, "width": 480}.

[siddharth-kt]

when setting format as explained in my question the camera view is like.
Its like ZOOMED ({"height": 640, "width": 480} image clicked output dimensions)

But when i don't set format to anything. Then it looks like.
It looks fine but it will take too much storage on my backend-storage ({"height": 2160, "width": 3840} image clicked output dimensions).

[mrousavy]

Huh this is really weird. I'm working on an alternative API approach for VisionCamera that should ideally solve this issue - stay tuned. I'll start working on it soon, got Covid rn so I'll be offline for a few days 😅

[siddharth-kt]

😮 @mrousavy please take care of yourself.

Untill you complete your work on new API could there be any workaround for this issue since its kind of major issue for me and my app is almost in release state.
and what do you think by when can we enjoy the new API 😎. Just asking.

Thanks for your efforts.

[mrousavy]

Thanks!

Phew, no idea. One to three months I guess? That assumes I have enough free time.

If you are interested in paid development/services, contact us at [email protected] :)

[siddharth-kt]

Thanks for replying,
If I felt to go with paid solution, then For sure I reach out to you.

[viljark]

Hey @siddharth-kt, I think your issue is duplicate of #281

As a workaround I've used this approach #281 (comment)
This is only needed if the camera is being initialized while phone orientation is portrait.

[siddharth-kt]

Thanks @viljark .But still we need a real solution.

[siddharth-kt]

@mrousavy kindly update us whenever this issue gets solved.

Thanks

[siddharth-kt]

@viljark are you facing this issue.
On recording video from front camera its mirrored.

[garmoshka-mo]

Photo & video sizes not configurable neither with presets nor with format props. (With format you can actually change size - but it will be random on android, not even close to width/height of specified format).

I had to hardcode desired resolution with patch: https://gist.github.com/garmoshka-mo/a6e5eca85537771082e7ca5ffb8131e1

[mrousavy]

Hey @garmoshka-mo! That patch on iOS is not safe and might crash on a few devices.

On Android; yes, I left a few comments in the codebase warning that this is not accurate on Android due to CameraX's design. I will rewrite the android part to Camera2 in the V3 PR, sponsor me on GitHub to support this effort. I need a lot of free time to do that.

[garmoshka-mo]

@mrousavy just sent you $$ 👍 I'm developer, and your lib is truly decent project - top in rn camera's niche, thanks! ⭐

while v3 is in progress,
how would you recommend to hotfix iOS resolutions, which quick changes can I hardcode?
In my case I just need hardcoded 768x1024 for photo and 480x640 for video with strong compression like SD (which would produce small enough videos).

[mrousavy]

@garmoshka-mo thanks man, I really appreciate it!! This goes in the pool for V3, I'll do a hackathon/week sprint to rewrite the Android part and make it more solid soon. ❤️

Regarding the hotpatch; on Android this looks somewhat safe to do as CameraX will internally use the format that best matches the given width/heights. so you're good there 👍
On iOS, it has to match the given width/height exactly, otherwise it crashes.

That's why on iOS I would write a function that just gets all the formats as it would right now, and then loops through them to find the one that best matches your target resolution (768x1024 or 480x640). This requires a bit of sorting logic. I have written a similar function here:

react-native-vision-camera/src/utils/FormatFilter.ts

Lines 45 to 71 in 8833ac1

	/**
	* Sort formats by resolution and aspect ratio difference (to the Screen size).
	*
	* > Note that this makes the `sort()` function descending, so the first element (`[0]`) is the "best" device.
	*/
	export const sortFormats = (left: CameraDeviceFormat, right: CameraDeviceFormat): number => {
	let leftPoints = 0,
	rightPoints = 0;
	// we downscale the points so much that we are in smaller number ranges for future calculations
	// e.g. for 4k (4096), this adds 8 points.
	leftPoints += Math.round(left.photoWidth / 500);
	rightPoints += Math.round(right.photoWidth / 500);
	// e.g. for 4k (4096), this adds 8 points.
	leftPoints += Math.round(left.videoWidth / 500);
	rightPoints += Math.round(right.videoWidth / 500);
	// we downscale the points here as well, so if left has 16:9 and right has 21:9, this roughly
	// adds 5 points. If the difference is smaller, e.g. 16:9 vs 17:9, this roughly adds a little
	// bit over 1 point, just enough to overrule the FPS below.
	const leftAspectRatioDiff = left.photoHeight / left.photoWidth - SCREEN_ASPECT_RATIO;
	const rightAspectRatioDiff = right.photoHeight / right.photoWidth - SCREEN_ASPECT_RATIO;
	leftPoints -= Math.abs(leftAspectRatioDiff) * 10;
	rightPoints -= Math.abs(rightAspectRatioDiff) * 10;
	return rightPoints - leftPoints;
	};

Just write your own sorting function and prefer formats that are closest to your target resolution (so e.g. when an iPhone doesn't have a 480x640 format, it would choose the next best format. That's why I have a sort function, so there's always a format)

lmk if you have any questions, I know this can be a bit tricky but that's just how cameras work, I've been thinking about a better API for this but it's hard to abstract this aways without loosing flexibility.

[garmoshka-mo]

@mrousavy i've tried first returned format with:
"videoHeight":144,"videoWidth":192,"photoWidth":4032,"photoHeight":3024
though both output photo & video size was 144x192. It looks like photos also use videoDimensions and photo dimensions ignored.

since formats don't work on android in any way, for consistency I've just patched CameraView+AVCaptureSession.swift to pick format by exact dimensions. In case if device doesn't have such exact resolution - then default simply will be used, but in my case 1024x768 & 640x480 are common dimensions so it should work always.

updated my patch to work on both ios & android: https://gist.github.com/garmoshka-mo/a6e5eca85537771082e7ca5ffb8131e1

Related question - could you advice how to adjust video bitrate? it looks like videos with low resolution still take decent space (2,8 Mbit/s pretty high for 640 × 480 video, 1Mbit would be enough)

[mrousavy]

Good point on Bitrate, I will research that right now! Maybe I can add it to V3...

[mrousavy]

For hotpatching it, you can do this in CameraView+RecordVideo.swift:

(my bad the code comment is wrong, that's 5Kbit/s, not 5Mbit/s)

[mrousavy]

Hey! I've rewritten the entire Android codebase of VisionCamera from CameraX to Camera2 in the efforts of ✨ VisionCamera V3.

I just now completed the Camera2 rewrite and I believe the core structure is running, but there might be some edge cases to iron out. Can you try and test the PR #1674 for me to see if you can still reproduce this issue here?

Here's an instruction on how you can test that: #1674 (comment)

If the issue cannot be reproduced with that version/PR anymore, then hoorayy, I fixed it! 🎉
Otherwise please let me know and I'll keep this issue open to keep track of it.

Thank you!