seemore

An app to make the lives of visually impaired people a little more ordinary.

Introduction • Installation • Application Structure • Key Features • How To Use • Credits • Contributors • Future Work • References • License

Introduction

The development of tools and technology hasn't resulted in the development of applications that could aid those with visual impairments. With the development of Data Modelling techniques, which can be used to give even basic computers a bit of "intelligence," and the ease of accessibility, this "intelligence" can be extended to our smartphones to aid those who are blind in navigating their surroundings and going about their daily lives. By utilising the power of Deep Learning, which can be made accessible even on low-end devices with a clear User-Interface that would precisely allow them to better grasp the world around, our application seeks to close the gap between them and the visible world.

This app enables the community of blind and visually impaired people to correctly identify objects they come across in everyday life without the need for sighted assistance.

Installation

Install with pip:

$ pip install -r requirements.txt

Application Structure

.
seemore/
├─ Frontend/
├─ calibration/
│  ├─ Ref_image.png
├─ currency_model/
│  ├─ bovw_codebook_600.pickle
│  ├─ rfclassifier_600.sav
├─ yolo_v5/
│  ├─ yolov5s.onnx
│  ├─ classes.txt
├─ app.py
├─ currencydet.py
├─ objdet.py
├─ requirements.txt
├─ .gitignore
├─ Procfile
├─ LICENSE

Key Features

⭐️ Voice Commands - Activate and access all the features of the app through voice commmands.

⭐️ SOS - Quickly send alerts to your emergency contacts.

⭐️ Object detection - Detects the object in front of you and the distance you are from it.

⭐️ Currency detection - Detects currency denominations.

⭐️ Read text - Reads the text for you.

For more details, visit nilavan.github.io/projects/1-seemore.

How To Use

To clone and run this application, you'll need Git and Flutter.

From your command line:

# Clone this repository
$ git clone https://github.com/Nilavan/seemore

# Go into the repository
$ cd seemore

# Install dependencies
$ pip install -r requirements.txt

# Run the app
$ flask run

Credits

This software uses the following open source packages:

• Flutter
• Python
• Python Anywhere
• Twilio
• Flask

Contributors

• A Nilavan
  • Backend development
  • Object detection (v1 & v2)
  • Currency detection (v2)
• Ajith Manivannan
  • Frontend development
  • Speech-to-text & text-to-speech
• N Lirajkhanna
  • SOS feature
  • Currency detection (v1)
  • Backend deployment
• TM Vishnu Mukundan
  • Text detection (OCR)

Future Work

Although the features we set out to build have been successfully implemented, the following areas can be improved in future versions of the app.

• Accuracy of detection models can be improved. We can use better and more efficient models trained on a wide variety of data to make it more robust.
• Extend object detection to more classes. The current version of the app can detect up to 80 different everyday objects. Our goal is to extend this to most objects we come across.
• Implement object detection in real-time instead of capturing image. This can drastically improve the “independence” of the visually impaired. At present, we send an image to the API and it returns the result. Our next goal is to allow the user to simply have the camera open while our app informs the person about objects detected at any time.

References

1. Detecting Objects in Flutter
2. Find distance from camera to object/marker using Python and OpenCV
3. How to OCR with Tesseract, OpenCV and Python
4. How to Send an SMS With Python Using Twilio
5. Adding speech-to-text and text-to-speech support in a Flutter app

License

This project is licensed under the MIT License - see the LICENSE.md file for details

nilavan.github.io  ·  GitHub @Nilavan  ·