This contains a fully-featured input system, which allows you to handle various types of input and send them to any game object being currently gazed at, or any fallback object. It also includes a few example cursors, similar to the HoloLens shell cursor, that fully leverages the Unity's animation system. This input system uses Unity's default EventSystem and there's no need for a custom input module. The default Standalone Input Module works with this input system.
The input module is designed to be extensible: it could support various input mechanisms and various types of gazers.
Each input source (hands, gestures, others) implement a IInputSource interface. The interface defines various events that the input sources can trigger. The input sources register themselves with the InputManager, whose role it is to forward input to the appropriate game objects. Input sources can be dynamically enabled / disabled as necessary, and new input sources can be created to support different input devices.
Game objects that want to consume input events can implement one or many input interfaces, such as:
- IFocusable for focus enter and exit. The focus can be triggered by the user's gaze or any other gaze source.
- IHoldHandle for the Windows hold gesture.
- IInputHandler for source up and down. The source can be a hand that tapped, a clicker that was pressed, etc.
- IInputClickHandler for source clicked. The source can be a hand that tapped, a clicker that was pressed, etc.
- IManipulationHandler for the Windows manipulation gesture.
- INavigationHandler for the Windows navigation gesture.
- ISourceStateHandler for the source detected and source lost events.
- ISpeechHandler for voice commands.
- IDictationHandler for speech to text dictation.
- IGamePadHandler for generic gamepad events.
- IXboxControllerHandler for Xbox One Controller events.
The input manager listens to the various events coming from the input sources, and also takes into account the gaze. Currently, that gaze is always coming from the GazeManager class, but this could be extended to support multiple gaze sources if the need arises.
By default, input events are sent to the currently focused game object, if that object implements the appropriate interface. Modals input handlers can also be added to the input manager: these modal handlers will take priority over the currently focused object Fallback handlers can also be defined, so that the application can react to global inputs that aren't targeting a specific element. Any event sent by the input manager always bubbles up from the object to its ancestors.
In recap, the input manager forwards the various input sources events to the appropriate game object, using the following order:
- The registered modal input handlers, in LIFO (Last-In First-Out) order of registration
- The currently focused object
- The fallback input handlers, in LIFO order of registration
Prefabs related to the input features.
Torus shaped basic cursor that follows the user's gaze around.
Torus shaped CursorOnHolograms when user is gazing at holograms and point light CursorOffHolograms when user is gazing away from holograms.
Torus shaped cursor that follows the user's gaze and HandDetectedFeedback asset to give feedback to user when their hand is detected in the ready state.
3D animated cursor that follows the user's gaze and uses the Unity animation system to handle its various states. This cursor imitates the HoloLens Shell cursor.
Unity camera that has been customized for Holographic development.
- Camera.Transform set to 0,0,0
- 'Clear Flags' changed to 'Solid Color'
- Color set to R:0, G:0, B:0, A:0 as black renders transparent in HoloLens.
- Set the recommended near clipping plane.
- Allows manual movement of the camera when in editor
Input system that manages gaze and various input sources currently supported by HoloLens, such as hands and gestures.
This also includes a fake input source that allows you to simulate input when in the editor. By default, this can be done by holding Shift (left input source) or Space (right input source), moving the mouse to move the source and using the left mouse button to tap.
Camera capabale of rendering for HoloLens and occluded Windows Mixed Reality enabled devices. MixedRealityCameraManager.cs exposes some defaults for occluded aka opaque displays Vs HoloLens. You can either use the defaults that have been set or customize them to match your application requirements.
For HoloLens:
- 'Clear Flags' is set to 'Solid Color' and Color to 'Clear' as black renders transparent in HoloLens.
- Near clip plane is set to 0.85 per comfort recommendations.
- Quality Settings to be Fastest.
For occluded aka opaque devices:
- 'Clear Flags' is set to Skybox
- Near clip plane is set to 0.3 which is typical for VR applications.
- Quality Settings to be Fantastic as it uses the PC GPU to render content.
This prefab is used when you want to enable teleporting on mixed reality enabled occluded devices. In order to prevent the MainCamera position from being overwritten in the next update we use a parent GameObject.
This prefab is used when you want to enable teleporting on mixed reality enabled occluded devices, as well as motion controller visualization. In order to prevent the MainCamera position from being overwritten in the next update we use a parent GameObject.
Scripts related to the input features.
Use this to visualize a 6DoF controller in your application. Add this script to a GameObject as a child of the MainCamera, or use the MixedRealityCameraParentWithControllers prefab. Either specify a shader to use for the glTF model or add GameObject overrides to represent the controllers.
- leftControllerOverride [Optional] A prefab to spawn to represent the left controller. This will automatically move and reorient when the controller is moved.
- rightControllerOverride [Optional] A prefab to spawn to represent the right controller. This will automatically move and reorient when the controller is moved.
- touchpadTouchedOverride [Optional] A prefab to spawn to represent the user's touch location on the touchpad. This will automatically move when the user moves their touch location. Default is a sphere.
- GLTFShader [Optional, if using overrides] If using the controller's built-in glTF model, this will be the shader applied to the resulting GameObject.
This can be used to load a glTF file in the editor, as well as displaying input and source data from motion controllers on a text panel for debugging purposes on the device.
- LoadGLTFFile A boolean to specify if a glTF file should be loaded from StreamingAssets.
- GLTFName The name of the GLTF file to be loaded from StreamingAssets.
- touchpadTouchedOverride [Optional] A prefab to spawn on the touchpad of the glTF file specified. Default is a sphere.
- GLTFShader [Optional, if not loading a glTF model] If loading a glTF model, this will be the shader applied to the resulting GameObject.
- TextPanel This is the text display where controller state info will be logged, for on-device debugging purposes.
Add this to a GameObject to register it as a global listener on the InputManager. This means it will receive events from the InputManager even while not focused.
Animated cursor is a cursor driven using an animator to inject state information and animate accordingly.
Abstract class that a concrete class should implement to make it easy to create a custom cursor. This provides the basic logic to show a cursor at the location a user is gazing.
- Decides when to show the cursor.
- Positions the cursor at the gazed hit location.
- Rotates the cursor to match hologram normals.
CursorModifier is a component that can be added to any game object with a collider to modify how a cursor reacts when on that collider.
Cursor interface that any cursor must implement.
Interface that any cursor modifier must implement to provide basic overrides for cursor behaviour.
Cursor whose states are represented by one or many meshes.
Cursor whose states are represented by one or many game objects.
Cursor whose states are represented by colored sprites.
With Windows Mixed Reality enabled devices and motion controllers, you can have different ways of representing the user's attention aka pointing. You can use the conventional 'gaze and commit' style interactions. In these your cursor follows your head gaze. With pointing ray enabled motion controllers you can perform the 'point and commit' style interactions. In these cases, your cursor will follow the pointing ray from the motion controller. Focus classes are meant to act as a bridge between these mechanisms.
Currently, we recommend using gaze and commit style interactions.
FocusDetails struct contains information about which game object has the focus currently. Much like RaycastHit. Also contains information about the normal of that point.
Focus manager is the bridge that handles different types of pointing sources like gaze cursor or pointing ray enabled motion controllers. If you dont have pointing ray enabled controllers, it defaults to GazeManager.
Class implementing IPointingSource to demonstrate how to create a pointing source. This is consumed by SimpleSinglePointerSelector.
Implement this interface to register your pointer as a pointing source. This could be gaze based or motion controller based.
Script to register a Canvas component so it's capable of being focused at for 'point and commit' scenarios.
Script shows how to create your own 'point and commit' style pointer which can steal cursor focus using a pointing ray supported motion controller. This class uses the InputSourcePointer to define the rules of stealing focus when a pointing ray is detected with a motion controller that supports pointing.
Data class that carries the input data for the event handler.
Defines the controller mapping for the input source.
Controller axis and button types.
A base abstract class for a stabilizer that takes as input position and rotation, and performs operations on them to stabilize or smooth that data.
Singleton component in charge of managing the gaze vector. This is where you can define which layers are considered when gazing at objects. Optionally, the gaze manager can reference a ray stabilizer that will be used to stabilize the gaze of the user.
- MaxGazeCollisionDistance : the maximum distance to raycast. Any holograms beyond this value will not be raycasted to.
- RaycastLayers : the Unity layers to raycast against. If you have holograms that should not be raycasted against, like a cursor, do not include their layers in this mask.
- Stabilizer : stabilizer to use to stabilize the gaze. If not set, the gaze will not be stabilized.
- Gaze Transform : the transform to use as the source of the gaze. If not set, will default to the main camera.
Stabilize the user's gaze to account for head jitter.
- StoredStabilitySamples Number of samples that you want to iterate on. A larger number will be more stable.
This script teleports the user to the location being gazed at when Y was pressed on a Gamepad. You must have an Xbox gamepad attached to use this script. It also works in the Unity editor.
Base class for all input event data. An input event data is what is sent as a parameter to all input events.
Event data for an event coming from dictation.
Event data for an event coming from a generic gamepad.
Event data for an event coming from the hold gesture.
Event data for an event that represents a click, including the number of taps.
Event data for an event that represents an input interaction such as a tap / click.
Event data for an event that represents changes in the X-axis and Y-axis for a thumbstick or touchpad.
Event data for an event coming from the manipulation gesture.
Event data for an event coming from the navigation gesture.
Event data for an event that represents a pointer entering or exiting focus on an object.
Event data for an event that represents a change in rotation of an input source.
Event data for an event that represents a change in position of an input source.
Event data for an event that represents an input source being detected or lost.
Event data for an event that represents a recognition from a PhraseRecognizer.
Event data for an event that represents the amount of depression of a trigger.
Event data for an event coming from an Xbox controller source.
Interface that a game object can implement to react to a controller's trigger value changing or touchpad/thumbstick XY changing.
Interface that a game object can implement to react to a controller's touchpad being touched or released.
Interface that a game object can implement to react to dictations.
Interface that a game object can implement to react to focus enter/exit.
Interface that a game object can implement to react to gamepad events.
Interface that a game object can implement to react to hold gestures.
Interface that a game object can implement to react to taps / clicks.
Interface that a game object can implement to react to an input button being pressed or released.
Interface that a game object can implement to react to manipulation gestures.
Interface that a game object can implement to react to navigation gestures.
Interface that a game object can implement to react to a specific pointer's focus enter/exit.
Interface that a game object can implement to react to a source's position changing.
Interface that a game object can implement to react to a source's rotation changing.
Interface that a game object can implement to react to source state changes, such as when an input source is detected or lost.
Interface that a game object can implement to react to a keyword being recognized.
Interface that a game object can implement to react to Xbox Controller events.
Abstract base class for an input source that implements IInputSource. Defines the various abstract functions that any input source needs to implement, and provides some default implementations.
Singleton class that implements the DictationRecognizer to convert the user's speech to text. The DictationRecognizer exposes dictation functionality and supports registering and listening for hypothesis and phrase completed events.
IMPORTANT: Please make sure to add the Microphone capabilities in your app, in Unity under
Edit -> Project Settings -> Player -> Settings for Windows Store -> Publishing Settings -> Capabilities
or in your Visual Studio Package.appxmanifest capabilities.
- InitialSilenceTimeout : The time length in seconds before dictation recognizer session ends due to lack of audio input in case there was no audio heard in the current session.
- AutoSilenceTimeout : The time length in seconds before dictation recognizer session ends due to lack of audio input.
- RecordingTime : Length in seconds for the manager to listen.
Input source for in-Editor input source information, which can be used to simulate hands and controllers in the Unity editor.
Base class that all gamepad input sources should inherit from.
Input source for gestures information from the WSA APIs, which gives access to various system supported gestures.
Interface for an input source. An input source is any input mechanism that can be used as the source of user interactions.
Allows you to specify keywords and keyboard shortcuts in the Unity Inspector, instead of registering them explicitly in code. Keywords are handled by scripts that implement ISpeechHandler.cs. You can utilize keywords with the SpeechInputHandler component by assigning game objects and specifying a Unity Event trigger.
IMPORTANT: Please make sure to add the Microphone capabilities in your app, in Unity under
Edit -> Project Settings -> Player -> Settings for Windows Store -> Publishing Settings -> Capabilities
or in your Visual Studio Package.appxmanifest capabilities.
Persistent Keywords Keywords are persistent across all scenes. This Speech Input Source instance will not be destroyed when loading a new scene.
RecognizerStart Set this to determine whether the keyword recognizer will start immediately or if it should wait for your code to tell it to start.
KeywordsAndKeys Set the size as the number of keywords you'd like to listen for, then specify the keywords to complete the array.
RecognitionConfidenceLevel The confidence level for the keyword recognizer.
Enumeration of the supported input infor for Unity WSA APIs.
Allows you to specity EventSystem axis and button overrides and custom controller mappings.
Horizontal Axis Sets the horizontal axis override. Vertical Axis Sets the vertical axis override. Submit Button Sets the submit button override. Cancel Button Sets the cancel button override.
Use Custom Mapping Enables custom mapping for your controller. Strings should match the input mapping from Edit/Project Settings/Input.
Allows dragging an object in space with your hand on HoloLens. Just attach the script to a game object to make it movable.
Allows dragging an object in space with your hand on HoloLens. Just attach the script to a game object to make it movable.
Allows users to tap on an object to move its position.
Used to register the GameObject on the InputManager as a global listener.
Very simple class that implements basic logic for a trigger button.
IMPORTANT: Please make sure to add the Microphone capabilities in your app, in Unity under
Edit -> Project Settings -> Player -> Settings for Windows Store -> Publishing Settings -> Capabilities
or in your Visual Studio Package.appxmanifest capabilities.
Struct that facilitates the storage of keyword and keycode pairs.
-
KeywordsAndResponses Set the size as the number of keywords you'd like to listen for, then specify the keywords and method responses to complete the array.
-
RecognizerStart Set this to determine whether the keyword recognizer will start immediately or if it should wait for your code to tell it to start.
Used to assign a Unity Event to a keyword stored in the SpeechInputSource component.
Very simple class that implements basic logic for a trigger button.
Prefabs used in the various test scenes, which you can use as inspiration to build your own.
Keyword manager pre-wired to send messages to object being currently focused via FocusedObjectMessageSender component. You can simply drop this into your scene and be able to send arbitrary messages to currently focused object.
Keyword manager pre-wired to send messages to object being currently selected via SelectedObjectMessageSender component. You can simply drop this into your scene and be able to send arbitrary messages to currently selected object.
Sends Unity message to currently focused object. FocusedObjectMessageSender.SendMessageToFocusedObject needs to be registered as a response in KeywordManager to enable arbitrary messages to be sent to currently focused object.
Lets you access beam-formed microphone streams from the HoloLens to optimize voice and/or room captures, which is impossible to do with Unity's Microphone object. Takes the data and inserts it into Unity's AudioSource object for easy handling. Also lets you record indeterminate-length audio files from the Microphone to your device's Music Library, also using beam-forming.
Sends Unity message to currently selected object. SelectedObjectMessageSender.SendMessageToSelectedObject needs to be registered as a response in KeywordManager to enable arbitrary messages to be sent to currently selected object.
Example on how to handle messages send by SelectedObjectMessageSender. In this particular implementation, selected object color it toggled on selecting object and clearing selected object.
A grid of dynamic objects to illustrate sending messages to prefab instances created at runtime as opposed to only static objects that already exist in the scene.
This class implements IFocusable to respond to gaze changes. It highlights the object being gazed at.
This class implements IInputClickHandler to handle the tap gesture. It increases the scale of the object when tapped.
This class implements INavigationHandler to handle the navigation gesture. It rotates the object left or right based on X movement.
Tests related to the input features. To use the scene:
- Navigate to the Tests folder.
- Double click on the test scene you wish to explore.
- Either click "Play" in the unity editor or File -> Build Settings.
- Add Open Scenes, Platform -> Windows Store, SDK -> Universal 10, Build Type -> D3D, Check 'Unity C# Projects'.
- Click 'Build' and create an App folder. When compile is done, open the solution and deploy to device.
Shows the basic cursor following the user's gaze and hugging the test sphere in the scene.
Shows the cursor on holograms hugging the test sphere in the scene and cursor off holograms when not gazing at the sphere.
Shows the cursor hugging the test sphere in the scene and displays hand detected asset when hand is detected in ready state.
Example on how to send keyword messages to currently focused dynamically instantiated object. Gazing on an object and saying "Make Smaller" and "Make Bigger" will adjust object size.
Test scene shows you in a simple way, how to respond to user's gaze using the Input module. It also shows you how to respond to the user's tap gesture.
In this scene, under the InputManager prefab > GesturesInput, the script GamepadInput.cs helps map the Xbox gamepad buttons to gestures. Press A to air tap.
Test scene shows how to respond to user's navigation gesture by rotating the cube along Y axis.
In this scene, under the InputManager prefab > GesturesInput, the script GamepadInput.cs helps map the Xbox gamepad buttons to gestures. With A pressed rotate the left joystick to trigger the navigation gesture.
This scene has some objects in there which respond to the Xbox gamepad input. It also has the HoloToolkitCameraParent prefab which is useful for testing teleporting scenarios for occluded devices. Cube: Press A to air tap. Press A and left joystick to rotate the cube.
Quad: Gaze at the quad and press Y to teleport to that location. Press B to return to the original location.
Shows how to use the KeywordManager.cs script to add keywords to your scene.
- Select whether you want the recognizer to start automatically or when you manually start it.
- Specify the number of keywords you want.
- Type the word or phrase you'd like to register as the keyword and, if you want, set a key code to use in the Editor. You can also use an attached microphone with the Editor.
- Press the + to add a response. Then, drag a GameObject with the script you want to call into the "None (Object)" field.
- Select the script and method to call or variable to set from the "No Function" dropdown. Add any parameters, if necessary, into the field below the dropdown.
When you start the scene, your keywords will automatically be registered on a KeywordRecognizer, and the recognizer will be started (or not) based on your Recognizer Start setting.
This scene shows how to manually control the camera. The script is on the main camera of the scene. When preview mode in Unity is activated, the user can move around the scene using WASD and look around using right-mouse-button + mouse.
Example usage of MicStream.cs to select and record beam-formed audio from the hololens. In editor, the script lets you choose if you want to beam-form capture on voice or on the room. When running, press 'Q' to start the stream you selected, 'W' will stop the stream, 'A' starts recording a wav file, and 'S' stops the recording, saves it to your Music library, and prints the full path of the audio clip.
This scene shows how to render motion controllers in your app. It also contains a debug panel to help diagnose the state of a connected controller.
Example on how to send keyword messages to currently selected dynamically instantiated object. Gazing on an object and saying "Select Object" will persistently select that object for interaction with voice commands, after which the user can also adjust object size with "Make Smaller" and "Make Bigger" voice commands and finally clear currently selected object by saying "Clear Selection".
Test scene shows you in a simple way, how to route input to an object not being gazed/focused at. Useful for scenarios like placing head locked content or clicking around to create objects.
Shows how to use the SpeechInputSource.cs script to add keywords to your scene.
- Select whether you want the recognizer to start automatically or when you manually start it.
- Specify the number of keywords you want.
- Type the word or phrase you'd like to register as the keyword and, if you want, set a key code to use in the Editor. You can also use an attached microphone with the Editor.
- Attach a script that implements ISpeechHandler.cs to the object in the scene that will require the gaze focus to execute the command. You should register this script with the InputManager.cs as a global listener to handle keywords that don't require a focused object.
When you start the scene, your keywords will automatically be registered on a KeywordRecognizer, and the recognizer will be started (or not) based on your Recognizer Start setting.