Smart See

Inspiration

I was researching about blind people in school for a project and I saw that 39 million people are blind. This is a big number and I thought why not with all the technology to make something that can tell what object is in front of you and help them identify household objects and navigate on the streets.

What it does

Smart See uses the rear camera to capture images of objects, run them through CoreML InceptionV3 Machine learning model. This makes a prediction on what kind of object it could be. We then used a speech synthesizer to speak out the most likely object.

How I built it

I researched the effectiveness of different models like InceptionV3 and Resnet50. I found them to be accurate 65% for most household objects. My theory is that this is pretty good for pre-trained models. Custom trained models may give up to 90% accurate results. I scavenged for sample code on the web and Apple's website. With a creative mishmash of cut, copy, and paste i was able to come up with a working prototype using the InceptionV3 model. The way it works is you can open the app using Siri. Once opened it starts predicting what object it sees and speaks it using speech synthesizer.

Challenges I ran into

After building the base I found that if you move it around even a little it would cycle through many possibilities. In order to combat that I added a timer where every 1.25 it would grab an image. Another problem was that it offered many predictions for one object. This made the result very confusing. To fix this we picked the most likely prediction and discarded others.

Accomplishments that I'm proud of

Proving the core concept out that can be extended for better results using a custom trained model.

What I learned

I learned more about how to use CoreML and a little bit more on how to use Xcode.

What's next for Smart See

• Extending object recognition using custom trained models
• Facial recognition to help people with Alzheimer's and the blind
• Combining object detection with landmark detection, geo-location and ability to read street signs to help with navigation on the sidewalk.

Built With

• xcode