A few months ago, a member of the VIA group had an experience in the Philadelphia SEPTA subway where he saw a visually impaired individual nearly crashing into a pole until he was millimeters apart from it. Observing this dangerous and preventable scenario, we realized that simply knowing whether there is something in front of someone isn't enough. Environmental context is innate to each and everyone of us, except for the visually impaired. With VIA (Visual Impairment Assistant), we want to bring context back to those that don't have it.

What it does

VIA consists of a close and long range component for the visually impaired to gather feedback and re-gain context of their surrounding environment. Five ultrasonic sensors interface with the a 4 x 5 array of micro-servos to give feedback to the user for up to 7 feet for close-range context. A recurrent neural network NeuralTalk2 trained on Microsoft COCO datasets describes to the user in real-time the environment in front of them for long-range context.

Together, the haptic feedback and the real-time neural network gives a complete overview of the user's immediate environment, telling the visually impaired individual what is in front of them, how far away an object is away from them, and where it is in front of them.

How we built it

1 - Ideation on design thinking


On Friday night, using the Make Toast design thinking method, we came up with 4 - 5 different project ideas, from an electric snowboard (to pay homage to the Blizzard Jonas), to a machine learning electronics part sorter, and more. Our group had unanimous agreement on attempting a hack to assist the blind.

2 - Proposing and testing out different mechanical haptic solutions

One major part of VIA is the haptic feedback device. The team proposed multiple mechanical solutions for the haptic feedback:

  • pins that go up and down driven by electromagnets (a solution that was dismissed after the prototype experienced too much EM interference when used in an array of said electromagnets)
  • ferrofluids (too complex/crazy)
  • smart fabric driven by electro-current that expands/contracts (too complex as we need to take into account the speed of heating and cooling)

After many hours of continued prototyping, the final haptics solution consists of 4 x 5 micro-servos that takes in commands from a row of 5 multi-directional ultrasonic sensors controlled by an Arduino Mega. The final design also probably has enough wire to go from Drexel to Penn and back.

3 - Recurrent neural network real-time video description

Using the NeuralTalk2 library, the team quickly built up an efficient and functional real-time environment describor that uses a webcam as the input feed. The original NeuralTalk2 library only offers image descriptions in the form of text, so the program had to be hacked to add text-to-speech as well as hooking up an external webcam. Click here to see test #2 for the image describor.

4 - Rapid prototyping and modeling

As VIA consists of not only software components, but also heavy hardware and electronic components, rendering software such as SolidWorks and Keyshot were used for rapid prototyping, 3D printing, and laser cutting. All hardware components prior to assembling were designed on SolidWorks to ensure a 1:1 scale and maximum precision. VIA is a project that requires precision to ensure the haptic feedback device working correctly, so being able to rapidly prototype using 3D printing and laser cutting was an essential part of the success of the project.

Challenges we ran into

  • Installing all the necessary dependencies required for NeuralTalk2
  • Applying text-to-speech to NeuralTalk2 without simply using Bash piping
  • Rapidly designing and prototyping all hardware and electronic parts
  • Stripping long distances of wiring
  • Assembling all enclosures such as the 4 x 5 array of servos, the 5 ultrasonic sensors, and the webcam
  • Busted servos, not enough servos, confining servos
  • Limited hardware components
  • Faulty equipment, components, servos
  • Lots of soldering work
  • Time consuming
  • Outdated Arduino libraries
  • Supplying enough current to electronics
  • Initial design failed as there were too many custom hardware components
  • Power issues
  • OpenCV crashing exactly 100 seconds of program initialization - still haven't figured this one out
  • Neural network consuming over 4 GBs of memory and crashing frequently during demo time
  • Laser cutting and 3D printer not being available, taking too long, and not being precise enough
  • Sleep deprivation (all three members haven't slept in 48 hours)

Accomplishments that we're proud of

  • using a 15 year old Celeron floppy IDE cable to connect the Arduino Mega to our array of ultrasonic sensors and servos
  • after changing design 2 - 3 times, all the hardware/electronics problems, and all the issues mentioned before we were still able to get something working. The team succeeded by change direction in-time, thinking outside the box, staying calm when facing new problems, and encouraging each other.
  • incredible complexity of the multiple disciplines necessary to get VIA working (skills: electronics, hardware, prototyping, software, mechanical engineering design, working the 3DPrinters/LaserCutter, access to hardware things, machining.. and more)
  • Three people (two of which were first time hackers, and the other his first hardware + machine learning hack) with very different skillsets working so well together in creating such a complex project
  • VIA WORKS!!!!!

What we learned


Karpathy for writing the NeuralTalk2 recurrent neural network library.

Built With

+ 1 more
Share this project: