-
-
SmartRoomba Dashboard -- Top left: trajectory of Roomba's path, top right: some sensor data, bottom: bump and infrared sensor status
-
Jim controlling the SmartRoomba
-
Our dashboard live, while the Roomba is following Jim
-
Bumper sensor displays bumper status
What it does
Our SmartRoomba is a personal mess-cleaning buddy that follows you around using facial recognition/tracking. Using a webcam mounted on the Roomba, the SmartRoomba detects its user's face, and autonomously follows them. The SmartRoomba pushes all its data to a slick web dashboard, so you can monitor your Roomba's location, trajectory, and sensor status on one dashboard.
How I built it
We used Python to control the Roomba and perform facial recognition. Our facial recognition uses Microsoft Cognitive Services and OpenCV. Our dashboard is served using Flask, and prettified with plenty of D3 (for the trajectory map) and CSS.
Challenges I ran into
One of the biggest challenges is detecting and computing the turning angle of the Roomba. The Roomba's wheel encoders provided some very limited and noisy data, so we had to figure out how to accurately retrieve the Roomba's angle for our trajectory map. Another challenge was efficiently performing facial recognition and tracking -- we experimented with different ways of doing this. This is also our first time using D3, so we spent a lot of effort getting some the dashboard elements to look nice.
Accomplishments that I'm proud of
We're very proud of figuring out how to use both Microsoft Cognitive Services and OpenCV to perform facial recognition, and using this facial tracking to control the Roomba.
What I learned
We learned how to make the most of noisy data from sensors, to compute the Roomba's trajectory angle. We learned how to do facial recognition, and how to use Microsoft Cognitive Services. We used D3 for the first time.
What's next for SmartRoomba
We think the best feature to add next to SmartRoomba are voice commands. It'd be great for the SmartRoomba to not only follow you around, but also respond to your voice. "OK Roombert, don't forget to vacuum the living room at 3:30!"
Log in or sign up for Devpost to join the conversation.