The goal was to create a supplemental sensory channel for the visually impaired. We envisioned an individual who already has a support cane to navigate around immediate obstacles in their path. Our system would give them an increased perception of the people and cars further in front of them.
The Project Vision / How it was built
The user would wear a device with a depth-sensing camera, that sends a constant stream of images to our system on the Google Cloud Platform Virtual Machine. In this implementation, our device was an iPhoneX running our proprietary app that is sending the stream of images to our LAMP server on the VM. The system then examines the images to detect objects such as people and cars in front of the user. This is accomplished through our adaptation of Darknet, an open source neural network framework written in C and CUDA. The resulting object identification data was meant to be integrated with the depth data from the iPhoneX's dual camera, however the depth data was unexpectedly inaccessible. Regardless, the information is analyzed, simplified, and sent back to the user's device. This data is to be used by a lightweight band worn under one's shirt running around their chest. This band would vibrate in corresponding locations and frequencies dependent on the proximity and type of object faced by the user. We did not plan to actually implement the vibrating band, but instead send the data to a DragonBoard connected to a programmable LED light strip that would visually simulate the differing locations on vibrating band. However, there were unexpected problems, and we instead just represent the data in a simple GUI.
Challenges We ran into
Team's total lack of hackathon experience. Lack of hardware. Most everything we had to self-teach on the spot. Lots of "duct taping" our processes together. The first DragonBoard was defective. Tedious process when installing our necessary dependencies on the GCP VM.
Accomplishments that we're proud of
Everything we did, no one had any solid prior experience with any of it! Successful implementation of Darknet! An all nighter!
What's next for Foresight
We're going to order the proper hardware and finish this project!