Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

visionOS / Vision Pro support #203

Open
gwangyu-lee opened this issue Mar 21, 2024 · 6 comments
Open

visionOS / Vision Pro support #203

gwangyu-lee opened this issue Mar 21, 2024 · 6 comments
Assignees
Labels
enhancement New feature or request

Comments

@gwangyu-lee
Copy link

I'm having trouble building klakNDI in visionOS, but it works fine in iOS.

Unity Version: 2022.3.21f1

I got this error.
"CS0246: The type or namespace name 'Klak' could not be found”

@keijiro keijiro self-assigned this Mar 21, 2024
@keijiro keijiro added the question Further information is requested label Mar 21, 2024
@keijiro
Copy link
Owner

keijiro commented Mar 21, 2024

Could you try adding the visionOS platform to Klak.Ndi.Runtime.asmdef?

@gwangyu-lee
Copy link
Author

Thank you for the reply!

I changed it and I can build it now. But when I ran it on Xcode(using a simulator), I got these errors.

Could not locate file 'default-binaryarchive.metallib' in bundle.
Built from '2022.3/staging' branch, Version '2022.3.21f1 (bf09ca542b87)', Build type 'Development', Scripting Backend 'il2cpp'
MemoryManager: Using 'Dynamic Heap' Allocator.
[UnityMemory] Configuration Parameters - Can be set up in boot.config
    "memorysetup-allocator-temp-initial-block-size-main=262144"
    "memorysetup-allocator-temp-initial-block-size-worker=262144"
    "memorysetup-bucket-allocator-granularity=16"
    "memorysetup-bucket-allocator-bucket-count=8"
    "memorysetup-bucket-allocator-block-size=4194304"
    "memorysetup-bucket-allocator-block-count=1"
    "memorysetup-main-allocator-block-size=16777216"
    "memorysetup-thread-allocator-block-size=16777216"
    "memorysetup-gfx-main-allocator-block-size=16777216"
    "memorysetup-gfx-thread-allocator-block-size=16777216"
    "memorysetup-cache-allocator-block-size=4194304"
    "memorysetup-typetree-allocator-block-size=2097152"
    "memorysetup-profiler-bucket-allocator-granularity=16"
    "memorysetup-profiler-bucket-allocator-bucket-count=8"
    "memorysetup-profiler-bucket-allocator-block-size=4194304"
    "memorysetup-profiler-bucket-allocator-block-count=1"
    "memorysetup-profiler-allocator-block-size=16777216"
    "memorysetup-profiler-editor-allocator-block-size=1048576"
    "memorysetup-temp-allocator-size-main=4194304"
    "memorysetup-job-temp-allocator-block-size=2097152"
    "memorysetup-job-temp-allocator-block-size-background=1048576"
    "memorysetup-job-temp-allocator-reduction-small-platforms=262144"
    "memorysetup-temp-allocator-size-background-worker=32768"
    "memorysetup-temp-allocator-size-job-worker=262144"
    "memorysetup-temp-allocator-size-preload-manager=262144"
    "memorysetup-temp-allocator-size-nav-mesh-worker=65536"
    "memorysetup-temp-allocator-size-audio-worker=65536"
    "memorysetup-temp-allocator-size-cloud-worker=32768"
    "memorysetup-temp-allocator-size-gfx=262144"
-> applicationDidFinishLaunching()
Found 1 interfaces on host : 0) 192.168.0.21
Player connection [4371890752] Multi-casting "[IP] 192.168.0.21 [Port] 55000 [Flags] 2 [Guid] 186663699 [EditorId] 0 [Version] 1048832 [Id] VisionOSPlayer(50,Gwangyus-MacBook-Pro.local):56000 [Debug] 0 [PackageName] VisionOSPlayer [ProjectName] <no name>" to [225.0.0.222:54997]...

[Physics::Module] Initialized MultithreadedJobDispatcher with 15 workers.
[Subsystems] Discovering subsystems at path /Users/gwangyulee/Library/Developer/CoreSimulator/Devices/FC7AB30E-CB4B-4B7A-8CD3-19F680E2D2C0/data/Containers/Bundle/Application/4E084D14-5DF5-4E8A-9EC1-6EE4746F2C3E/KlakNDIURP.app/Data/UnitySubsystems
[Subsystems] No descriptors matched for  examples in UnitySubsystems/UnityVisionOS/UnitySubsystemsManifest.json.
[Subsystems] 1 'inputs' descriptors matched in UnitySubsystems/UnityVisionOS/UnitySubsystemsManifest.json
[Subsystems] 1 'displays' descriptors matched in UnitySubsystems/UnityVisionOS/UnitySubsystemsManifest.json
[Subsystems] 1 'meshings' descriptors matched in UnitySubsystems/UnityVisionOS/UnitySubsystemsManifest.json
GfxDevice: creating device client; threaded=1; jobified=0
Initializing Metal device caps: Apple xrOS simulator GPU
Initialize engine version: 2022.3.21f1 (bf09ca542b87)
AddInstanceForFactory: No factory registered for id <CFUUID 0x600000230260> F8BB1C28-BAE8-11D6-9C31-00039315CD46
       MEMixerChannel.cpp:1006  MEMixerChannel::EnableProcessor: failed to open processor type 0x726f746d
[0x122164b80|InputElement #0|Initialize] Number of channels = 0 in AudioChannelLayout does not match number of channels = 2 in stream format.
The referenced script (Unknown) on this Behaviour is missing!
The referenced script on this Behaviour (Game Object '<null>') is missing!
WARNING: RGBA Compressed ASTC6X6 UNorm format is not supported, decompressing texture
XRGeneral Settings awakening...
UnityEngine.XR.Management.XRGeneralSettings:Awake()

[Subsystems] Loading plugin UnityVisionOS for subsystem VisionOS-Display...
[Subsystems] VisionOS XR Plugin successfully registered Provider for VisionOS-Display
[Subsystems] VisionOS XR Plugin successfully registered Provider for VisionOS-Input
[Subsystems] VisionOS XR Plugin successfully registered Provider for VisionOS-Display
[Subsystems] VisionOS XR Plugin successfully registered Provider for VisionOS-Input
[Subsystems] VisionOS XR Plugin successfully registered Provider for VisionOS-Display
[Subsystems] VisionOS XR Plugin successfully registered Provider for VisionOS-Meshing
[XR] VisionOSDisplayProvider::Initialize
[Subsystems] Loading plugin UnityVisionOS for subsystem VisionOS-Input...
[Subsystems] Loading plugin UnityVisionOS for subsystem VisionOS-Meshing...
New input system (experimental) initialized
[XR] Display Start
UnloadTime: 0.185666 ms
dyld[32781]: missing symbol called

But I can build and run it on devices and simulators.
Thank you for creating such a useful package!

@keijiro
Copy link
Owner

keijiro commented Mar 23, 2024

Thank you for confirming that. It's helpful since I don't have an actual device.

@gwangyu-lee
Copy link
Author

Here are a few updates.
I can build and run with real Vision Pro, using the iOS platform. Here is the screenshot.
IMG_0033

I can build but can't run with real Vision Pro, using the visionOS platform. Here is the log.
Screenshot 2024-03-24 at 5 11 25 AM

And I can build but can't run with a Vision Pro simulator, using the iOS and visionOS platforms.

@keijiro keijiro added enhancement New feature or request and removed question Further information is requested labels Jun 12, 2024
@keijiro keijiro changed the title Can't build to visionOS visionOS / Vision Pro support Jun 12, 2024
@gwangyu-lee
Copy link
Author

Hi! I fixed the error. There is no UnityEditor.visionOS.Xcode yet so I did some tricks.

  1. Check that libndi_visionos.a is in /Library/NDI SDK for Apple/lib/visionOS
Screenshot 2024-08-24 at 2 16 27 AM
  1. Use the UnityEditor.iOS.Xcode but edit the path manually.

PbxModifier.cs

// #if UNITY_IOS
using UnityEditor;
using UnityEditor.Callbacks;
using UnityEditor.iOS.Xcode;
using System.IO;

namespace Klak.Ndi.Editor
{

    public class PbxModifier
    {
        [PostProcessBuild]
        public static void OnPostprocessBuild(BuildTarget buildTarget, string path)
        {
            ModifyProjectFile(path);
            ModifyPlistFile(path);
        }

        static void ModifyProjectFile(string basePath)
        {
            // Set visionOS project file path
            var path = Path.Combine(basePath, "Unity-VisionOS.xcodeproj", "project.pbxproj");

            if (!File.Exists(path))
            {
                throw new FileNotFoundException($"Xcode project file not found at path: {path}");
            }

            var proj = new PBXProject();
            proj.ReadFromFile(path);

            var target = proj.GetUnityFrameworkTargetGuid();
            // Set visionOS lib file path
            var libPath = "/Library/NDI\\ SDK\\ for\\ Apple/lib/visionOS";
            proj.AddBuildProperty(target, "LIBRARY_SEARCH_PATHS", libPath);
            proj.AddFrameworkToProject(target, "Accelerate.framework", false);
            proj.AddFrameworkToProject(target, "VideoToolbox.framework", false);
            // Set the libndi_visionos.a file name
            proj.AddFrameworkToProject(target, "libndi_visionos.a", false);

            proj.WriteToFile(path);
        }

        static void ModifyPlistFile(string basePath)
        {
            var path = Path.Combine(basePath, "Info.plist");

            var plist = new PlistDocument();
            plist.ReadFromFile(path);

            var root = plist.root;

            // Bonjour service list
            {
                var key = "NSBonjourServices";
                if (root.values.ContainsKey(key))
                    root.values[key].AsArray().AddString("_ndi._tcp");
                else
                    root.CreateArray(key).AddString("_ndi._tcp");
            }

            // LAN usage description
            {
                var key = "NSLocalNetworkUsageDescription";
                var desc = "NDI requires device discovery capability " +
                           "on the networks you use.";
                if (!root.values.ContainsKey(key)) root.SetString(key, desc);
            }

            plist.WriteToFile(path);
        }
    }

} // namespace Klak.Ndi.Editor

// #endif

I just changed the project path like var path = Path.Combine(basePath, "Unity-VisionOS.xcodeproj", "project.pbxproj"); and libpath like var libPath = "/Library/NDI\\ SDK\\ for\\ Apple/lib/visionOS"; And this one. proj.AddFrameworkToProject(target, "libndi_visionos.a", false);

  1. Add visionOS platform to Klak.Ndi.Runtime.asmdef

Klak.Ndi.Runtime.asmdef

{
    "name": "Klak.Ndi.Runtime",
    "rootNamespace": "",
    "references": [
        "GUID:df380645f10b7bc4b97d4f5eb6303d95"
    ],
    "includePlatforms": [
        "Android",
        "Editor",
        "iOS",
        "visionOS",
        "LinuxStandalone64",
        "macOSStandalone",
        "WindowsStandalone64"
    ],
    "excludePlatforms": [],
    "allowUnsafeCode": true,
    "overrideReferences": false,
    "precompiledReferences": [],
    "autoReferenced": true,
    "defineConstraints": [],
    "versionDefines": [
        {
            "name": "com.unity.render-pipelines.core",
            "expression": "0.0.0",
            "define": "KLAK_NDI_HAS_SRP"
        }
    ],
    "noEngineReferences": false
}
  1. Now you can receive the NDI source but if you are using Mixed Reality app mode, the texture won't update. So you should make your texture dirty to update each frame. Please check this document PolySpatial Render Texture Support

So add this script and select your render texture.

SetRenderTextureDirty.cs

using UnityEngine;
using Unity.PolySpatial;

public class SetRenderTextureDirty : MonoBehaviour
{
    public RenderTexture texture;

    void Update()
    {
        Unity.PolySpatial.PolySpatialObjectUtils.MarkDirty(texture);
    }
}
Screenshot 2024-08-24 at 2 25 28 AM

This is the process I built and made the app.

p.s. I've made a Vision Pro app using this package. It is released on the Apple app store and you can download it for free until Aug 27. If you are interested, here is the link. Thank you.

@keijiro
Copy link
Owner

keijiro commented Aug 26, 2024

Thanks for giving the detailed information! I'll try it when the company device is available for testing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants