Short Short Term Memory

• 

  Search the house

Inspiration

Frustrated when you can't find things you often misplace? Wish you had the power to search and find things/people sitting at your desk?

What it does

The application has multiple components ranging from smartphones to servers. Smartphones are the entry point of the system which supplies our server with real-time video streams that our server uses to perform object detection and facial recognition using Microsoft's Azure custom vision api. The result from this response is used to index images based on confidence. With this index, the user can perform a search to find objects that he may generally forget in his daily life.

How we built it

We used IP Cam to capture real-time video from smartphones and feed it to a python opencv server running to process the video feed. After extracting feeds using opencv, we used Microsoft Azure custom vision API to detect objects like keys, wallets even faces. We train the neural net with our custom images to perform personalized image searches. We used a simple and embedded database to persist indices which can be queried later. We used a javascript front end to display the location where the user misplaced some object.

Challenges we ran into

No hardware cameras, we used smartphones as webcams. Google cloud, Microsoft, AWS vision APIs failed to detect small images like keys. Usage of the cloud to store and retrieve images for the sake of feeding to APIs. This was decreasing the performance of our APP.

Accomplishments that we're proud of

Streaming images peer-to-peer in order to improve performance, thereby eliminating the middleman and not incurring a cost of cloud storage. Using multiple cameras to find objects. One line of code to add a new camera to monitor.

What we learned

Using opencv for real-time video processing. Streaming images to the APIs and clients instead of storing in the cloud and retrieving later.

What's next for Short Short Term Memory

IPO November, Christmas in December! Make it a platform for users to add a personalized image for the application to track and find.

Built With

• azure
• azure-cognitive
• deep-learning
• flask
• neural-networks
• opencv
• python
• streaming
• vision