Walkie Talkie

Inspiration

We wanted to use our knowledge to build something that tries to solve a problem, is impactful, and has potential to be upscaled.

Our smart cane integrates two key features to assist users:

Object Detection: Using a camera, the cane detects objects in the user’s surroundings through YOLO (You Only Look Once) object detection model trained on a dataset including pedestrians, bicycles, cars, etc. It then provides real-time audio feedback, describing the surroundings using the detected objects.
Haptic Feedback: Proximity sensors trigger vibrations in the handle to alert users when they are approaching an obstacle, providing haptic feedback to guide them in real-time. The vibration intensity depends on distance to obstacle. These features work together to make navigation smoother, safer, and more intuitive for visually impaired individuals.

Camera & Object Detection: A lightweight camera mounted on the cane feeds an image stream into a YOLO object detection model. We fine-tuned the model to recognize common obstacles and provide real-time descriptions using text-to-speech (TTS) technology.
Proximity Sensors & Haptic Feedback: Ultrasonic sensors placed on the front of the cane detect objects within a certain distance. The data is sent to the Raspberry Pi, which triggers vibration motors embedded in the cane’s handle when objects get too close.
Raspberry Pi: We used an embedded system to handle sensor data processing and control the camera, TTS system, and haptic actuators. It also hosts the client which does the computer vision work, while streaming the video feed to the server which handles the YOLO object detection. This was achieved using python's websockets.

Real-Time Processing: One of the biggest challenges was ensuring that the object detection and haptic feedback systems worked in real-time. Processing camera data fast enough to provide immediate feedback was tough, especially when running YOLO on limited hardware.
Limited Hardware options: We faced difficulties with most of the Raspberry Pis where they would repeatedly hang or crash (which makes hosting the YOLO model on board impossible) and limited sensors and motors.
Integration of Multiple Systems: Bringing together the object detection model, sensors, and feedback systems was a complex task. Ensuring seamless communication between hardware and software without delays or glitches took a lot of troubleshooting.

We have created a relatively inexpensive and highly scalable solution that delivers practical value to visually impaired users for road safety and aided navigation in a real-world environment. Our solution lays a solid foundation for the idea while leaving a lot of room for improved features. More sensors can be added/replaced with relative ease, more software options can be deployed in addition to visual description, such as an interactive exploratory mode, or dangerous obstacles only mode. Creating a device that is lightweight, easy to use, and delivers practical value to visually impaired users in real-world environments.

Hardware-Software Integration: We gained experience in integrating sensors, cameras, and microcontrollers to create a cohesive system that can handle multiple inputs and outputs simultaneously.
User-Centered Design: Working with assistive technology made us realize how crucial it is to prioritize the user experience, from ease of use to accessibility and comfort.

Conducting deeper research to gather information on the specific needs of our target users.
In terms of hardware, we plan on adding more ultrasonic sensors for more detailed/fine grained haptic feedback. Additionally we would like to incorporate a GPS tracker onto the cane to leverage google maps API for a more detailed description of road and street networks, particularly crosswalks and junctions.
In terms of software, we would like to add features such as voice control and a conversation AI model for a more interactive navigation experience. We would also add specific software modes based on user requirements.