Quest 2 Crashes on Selecting NDI Source - Unity 2022 Build

[eeease]

Upon selecting an NDI source on Quest 2, the system crashes instead of rendering the source.

This was tested with many different variables, the constant one being Unity version:

• The sample Klak NDI scene that comes with the package (with a dropdown, receiver, sender, and canvas to render the video)
• Unity 2022.3.4f1
• Android API 30, 31, 32, 33
• URP
• No URP
• ARM64
• NDI Launcher Tools Test Pattern for Windows, same Wi-fi network
• NDI Launcher Tools VLC video that is confirmed to work in Editor and Standalone builds
• MSAA on
• MSAA off
• Color Space: Gamma
• Color Space: Linear
• Graphics API: OpenGLES3 (which works in our 2020 version of the project)

Notes:

1. NDI rendering works fine in Editor (2022) and in desktop standalone builds (Windows, Mac).
2. Our project includes the changes from PR191, which were needed in order to detect sources in API 31+.
3. Also includes changes from PR186.
4. The changes from PR191 and PR186 work when we make builds from Unity 2020, API 30, 31, 32.

Logcat logs from when I select an NDI source:

Any suggestions or help would be greatly appreciated. We want to update our project to Unity 2022, but we need NDI streaming to work on Quest 2 builds.

Thanks in advance!

[keijiro]

First of all, I don't have a Quest 2 device, so unfortunately, I can't investigate the issue on my side.

I haven't had a firm idea, but I guess the issue is Unity version-specific because it runs on Unity 2020.

How about updating the project to the latest version of Unity 2022.3? Unity 2022.3.4 is a bit too outdated.

[eeease]

Thanks for the reply! At your suggestion, I tried making a build with Unity 2022.3.20f1, which seems to be the latest LTS version of 2022.
The same crash occurs when I select an NDI source from the dropdown on Quest.

We know that Vulkan works but we currently have a dependency on OpenGLES3 and we're unsure why OpenGLES3 works in 2020 but not 2022.

[keijiro]

How about changing the resolution of the video stream?

I still don't have any idea about the issue, but I guess it's related to the memory alignment of something. I wonder if it works with "compute friendly" dimensions like 256x256 or 1024x1024.

[eeease]

Good suggestion!
I tried a 1024x1024 video (modified via Handbrake) and the issue still occurs.
It's worth noting that I also tried making a build from Unity 2021.3.35f1, same issue.

For NDI source, I've also been testing using the Test Patterns from the official NDI Tools Launcher via Windows. Everything - test patterns, videos from VLC, etc - works fine from Unity 2020.

[keijiro]

How about disabling the compute shaders? You can disable them by commenting out the kernel functions in Decoder.compute.

https://github.com/keijiro/KlakNDI/blob/main/jp.keijiro.klak.ndi/Runtime/Resource/Decoder.compute

You should comment out only the body of the kernel functions -- You shouldn't comment out the function declarations.

[eeease]

I commented out the body of the following functions:
DecodeUYVY_sRGB
DecodeUYVY_Linear
DecodeUYVA_sRGB
DecodeUYVA_Linear

I made an Android build and attempted to receive 1) NDI Test Pattern and 2) VLC video.

The performance did not get majorly impacted when I selected a source.
However, the visuals didn't appear on the Canvas.

Did I comment out too much? (I'm not too knowledgeable of shaders.)
Hopefully that helps point in the right direction.

[keijiro]

Thanks for testing it.

However, the visuals didn't appear on the Canvas.

That's the intended behavior. I just wanted to check if the compute shader is relevant.

You turned it off, and the crash disappeared. It indicates that the crash happens in the compute shader or how GLES3 handles it.

[keijiro]

How about adding alpha channels to the sending frames? It's relevant because there are two versions of compute shaders (UYVY - without alpha, UYVA - with alpha)

You should revert commenting-out before testing it.

[eeease]

Hey @keijiro , I'm happy to try that out but not 100% sure what you mean.
Do you want me to change something in the code? I'm not a shader programmer so I'm not sure exactly how to add alpha channels to the sending frames.

Or should I try streaming something that has alpha?

[keijiro]

Just turning on/off the "Keep Alpha" switch on the sender component is enough for testing it.

[eeease]

Got it. When I select a source with the Keep Alpha bool set to true, the same issue occurs. I will try to capture the issue without revealing any of the protected images in our app.

[eeease]

NDICanvasQuest2.mp4

Here's a video of what happens when I select a source. I tried to capture the screen-tearing that I see in the headset, but it seems every way I try to capture video, the Quest just stops processing when the NDI source is selected.

[keijiro]

By the way, I wonder if GLES3 on Quest supports unordered access views (UAV), which is vital for the compute shaders. Could you check if SystemInfo.SupportsRandomWriteOnRenderTextureFormat returns true on the device?

[eeease]

That function is returning true on the device; testing in NdiReceiver:

[keijiro]

Thanks for the clarification. So it satisfies the requirement.

[eeease]

Do you have any other suggestions for things to try? Anything else you'd like me to Log?
We are trying to switching our entire app to Vulkan to be able to use NDI streaming, but we're seeing much worse performance on Vulkan and running into other issues.

[keijiro]

Those kinds of driver-level issues are quite hard to investigate without an actual device. Sadly, logging can’t help a lot.

[keijiro]

I updated this package (2.1.1). While this update is not directly related to this issue, it may potentially affects it. Could you give it a try? PR191 and PR186 are no longer needed.

[eeease]

I'll give it a try, thanks for the update.

[eeease]

When I select an NDI source, I'm running into a NullReferenceException in FormatConverter.
var compute = _resources.decoderCompute;

Something's returning null there, presumably _resources. I have another task to focus on but when I get time I'll get back to this and try to debug it.
Any suggestions would be helpful.

[eeease]

I managed to pass _resources in before the Decoding. It works in Editor.
I made an OpenGLES3 build and I'm experiencing the same issue as initially reported -> Can see the NDI Source, can select it, but it does not render to the media surface and instead the app chugs to the point of being unusable.

[pfcDorn]

Hey, we have a simliar (or same) issue.
I found out that the computeBuffer.SetData() causes memory leaks. You can see that in the Profiler > Memory > Untracked Memory. Also some random Unity Editor crashes. And of course, higher video resolutions (like 4k) ends up really fast in x GBs memory consumption which also results in crashes at some point.
The interessting part is: When i copy data from ndi data pointer to a temp nativearray and put this data then to the computerbuffer.setData, the memory leaks are gone and the crashes also.

        var temp = new NativeArray<byte>(view.Length, Allocator.Temp);
        NativeArray<byte>.Copy(view, temp, view.Length);
        buffer.SetData(temp);
        temp.Dispose();

I'm still not sure why :/

[keijiro]

@pfcDorn Sorry but that's off topic. This issue reproduces only on Quest devices with OpenGL ES3.

[pfcDorn]

So...another test:
I put the ReceiveFrameTask in the Update loop and removed "mainThreadContext.Post(ProcessVideoFrame, video);" and call the ProcessVideoFrame directly. This also seams to work (of course, not ideal)

[pfcDorn]

sorry, but my guess was its releated. Because on the quest this issue starts earlier (lower memory capacity). We first observed this issues on our quest build... thats the reason :D

[keijiro]

@eeease I reproduced the issue on Quest Pro. The root cause is still unknown, but I found a workaround for it. Please try it with the following steps:

• Update KlakNDI to 2.1.2.
• Add KLAK_NDI_ISSUE200_WORKAROUND to Scripting Define Symbols in Player Settings.

It changes the receiver texture format to RGBAHalf.
https://github.com/keijiro/KlakNDI/blob/main/jp.keijiro.klak.ndi/Runtime/Internal/FormatConverter.cs#L150

It increases the memory requirement and bandwidth usage, so it's not an ideal solution, but I couldn't find other solutions.

FYI, I used the following project for testing. Now it works properly on Quest Pro. I haven't checked with Quest 2 because I don't have a device.
https://github.com/keijiro/QuestGLES3Test

[eeease]

Thank you! I was in the midst of wrapping up another project but I'll try this out as soon as possible and report back.