Our team loves machine learning and AI technologies and we avidly study it. As a result, we wanted to marry our electrical engineering skills with some of the amazing emerging tech we were exposed to this weekend to create a smart home system.

What it does

We take short videos and stills from a front door feed that recognizes user faces and matches them with the residents' social media connections to identify friends. Furthermore our local server amasses histories of all visitors to intelligently decide if a visitor is a regular, welcomed visitor (family, friend, neighbor), an unknown, are definitely nefarious (returning thieves, a harassing ex-boyfriend, annoying solicitors) , and a sentiment analysis associated with them by the ever-learning Clarifai system to help determine appropriate measures. All of this only scratches the surface, as we have designed smart home responses to these various visit scenarios. We made a set of smart curtains from scratch, giving the user increased heating efficiency and privacy. Our light sensor detects sunrise and nightfall, calling for opening and closing of the curtains, respectively, leading to a natural awakening process that caters to our evolutionary biology's natural circadian rhythm for early momentum to a successful day, every day. The integrated temperature sensor automatically decides if the sunlight is welcomed (inside it's less than 70 degrees) and opens the blinds accordingly, saving on electric bills. Furthermore, Clarifai and our system's recognition program closes the blinds near the front door when unwanted visitors approach, and open them when welcomed visitors approach. This is a simple example of how we automate home security, privacy and environmental considerations all through Clarifai. We plan to implement simple voice controls with custom Alexa Lambda functions in the future.

How we built it

We used various micro-controllers and hacked together various circuits to run DC motors to open and close smart blinds for the home. Arduino was relied on heavily and we started with the Yun, considered other options before settling on an uno with a motor shield base shield from Groove with a light and temperature sensor. The Facebook API was explored with node.js to simulate a browser implementing Facebook's legendary friend face recognition protocol on visitors. This complemented our existing database of residence visitor history and tags from Clarifai to offer a weighted house response affecting the window systems (curtain opening/closing) and in the near future other feedback systems such as A/V readings and text messages.

Challenges we ran into

Primary challenges were time and a lack of preexisting work in the area. If people had published Facebook face recognition API work previously it would not have been so difficult to get running. We weren't able to finish this functionality due to a lack of time but we made promising strides and we're excited to have it running soon along with the voice recognition features. Another major challenge was dealing with fishing wire that loved to coil up as soon as it got a chance and had to be carefully wound around our spindles probably twenty times before achieving repeated flawless operation (the weakness of duct tape vs our final wall fixtures was another factor).

Accomplishments that we're proud of

We are extremely proud to have implemented smart home automation in truly innovative ways (building a smart window system from scratch that reacts to visitors as an example of how we can automate all kinds of smart actions such as audible and visual announcements of the visitor and their familiarity). We also are proud to have gotten extremely close to matching visitors with your social media friends, adding another layer to the tags and internal system matching functionality that guarantees UNSURPASSED security from harassment and privacy from intruders both visually through your window and audibly/physically due to alerts about visitors and a deterrant alarm system.

What we learned

We learned a ton about SDK's and API's, the state of home automation, how to deal with finicky duct tape and curly fishing line, quickly prototype alarm and motor driver circuits, program embedded systems and machine learning applications, identify visitors with sentiment analysis of unprecedented levels, and how to work as a team while having fun.

What's next for Smart Home with Clarifai

We plan to integrate Alexa and smart lights through IFTTT, as well as full Facebook friend searching for more accurate visitor information. Then we will sell our system, if possible!

Share this project: