This repository contains a implementation of Spatial Continuity. Spatial Continuity is a novel concept that proposes a combination of spatial computing and large language models (LLM) to provide seamless transitions between virtual and real-world contexts for making the world more accessible to users with low vision. Ultimately, this should lead to a truly continuous experience where the friction that accessible technologies introduce, e.g., by setup and positioning, is fully removed.
This implementation is part of our work on the "Spatial Continuity: Investigating Use Cases of Spatial Computing for Users with Low Vision" poster, which was accepted at the 27th International ACM SIGACCESS Conference on Computers and Accessibility.
The current prototype implementation of Spatial Continuity consists of two apps: the Spatial Continuity app on Vision Pro and the Spatial Continuity Camera app on iPhone.
The Spatial Continuity Camera app functions as a hand-held magnifying glass. After connecting, it streams the camera feed from the iPhone to the Spatial Continuity Camera app running on Vision Pro.
The Spatial Continuity Camera app can open screenshots from the camera livestream in a separate window for further inspection. If an OpenAI API key is provided, spoken image descriptions will be generated as well.
- Upon opening the Spatial Continuity app for the first time, you will be presented with the SpeziLLM onboarding window where you may enter your OpenAI API key. An OpenAI API key is required for generating spoken image descriptions.
- Ensure that both your iPhone and Vision Pro are on the same network. When opening the Spatial Continuity Camera app on iPhone, the connection should be established automatically.
You can find a list of contributors in the CONTRIBUTORS.md file.
As part of this work, we made the following open source contributions:
- apple/swift-openapi-runtime#115
- apple/swift-openapi-generator#625
- openai/openai-openapi#311 (comment)
- StanfordSpezi/SpeziViews#49
- StanfordSpezi/SpeziAccount#82
- StanfordSpezi/SpeziLLM#64
- StanfordSpezi/SpeziViews#43
- StanfordSpezi/SpeziLLM#61
Contributions to this project are welcome. Please make sure to read the contribution guidelines and the contributor covenant code of conduct first.
This project is licensed under the MIT License. See Licenses for more information.
For more information, check out our website at biodesigndigitalhealth.stanford.edu.
This project is the result of a collaboration between the Stanford Mussallem Center for Biodesign and the Technical University of Munich.
