We wanted to introduce ourselves to neural networks, machine learning, TensorFlow... But also maintaining our roots, as we have a background in computer engineering and embedded systems.

Our objective was doing something useful for us, because we have sometimes forgotten to take our keys, or we don't think about the weather before going outside.

What it does

It is a specialized assistant that tracks different actions when you enter or leave home.

It detects when the door is opened, which means that a family member is entering or leaving. If the user has come home but has forgotten to leave keys at its place, the Home Advisor will remind them to leave keys until they do it. Also, if the user is going outside and their keys are not taken, Home Advisor will remind them to take keys.

Also, when the user is leaving, Home Advisor recognizes if they are wearing a jacket or not. If the user is not wearing it and it's cold, Home Advisor will remind them to wear a jacket.

How we built it

We have three different parts: sensors, servers, computer vision.

We used an ultrasound sensor to detect when the door is closed and opened. An RFID is in charge to detect when a family member leaves or takes keys. We can know which member is thanks to de RFID uid, each member has a different uid and we can personalize the welcome messages. We have used a NodeMCU, an Arduino with WiFi integrated, to communicate differently states to the server with HTTP methods via WiFi. We also used an additional Arduino to be able to connect all the sensors.

We have developed two different servers. The first server is written in Go and it's the main server. It is in charge to handle the NodeMCU inputs and family members database. It coordinates the computer vision in the cloud and the microcontroller system. It is deployed in a Raspberry Pi 3B+, so we have connected it to a Logitech speaker via Bluetooth. This way, we can reproduce Home Advisor messages. The second server is written in Flask and it's in charge of computer vision. We have trained a Convolutional Neural Network (CNN) with TensorFlow to recognize when the user is wearing a jacket or not.

Challenges we ran into

  • We had a lot of problems with Bluetooth because it stops working sometimes.
  • TensorFlow was a challenge because we haven't ever designed a neural network, neither training it.
  • Finding a suitable data set for our approach.
  • Importing the trained model into a Raspberry Pi 3B+, as CNN was trained in a Power architecture and not in an Arm one.
  • Integrating the neural network in Flask server was tricky.
  • Coordinating all the sensors in the NodeMCU because it does not have enough GPIOs, so we had to add an Arduino as a slave to take care of a sensor.

Accomplishments that we're proud of

Although all the challenges we ran into, we were able to solve them.

What we learned

We learned a lot because we have never done a project with computer vision, neural networks, embedded communication via WiFi and Bluetooth.

We learned:

  • How to manage to take photographies via webcam from a server.
  • How to design and train a CNN.
  • How to export the model to a different architecture.
  • How to run neural networks inside a Flask server.
  • Flask
  • How to communicate embeddeds via WiFi.

What's next for Home advisor

We would love designing a printed circuit board (PCB) in order to integrate all embedded in a unique system. We would also like improving CNN to recognize more pieces of clothes, so depending on the weather would be able to suggest the user what should wear. For example, if it was raining, it would suggest you take an umbrella.

Built With

Share this project: