Our Integrated Website
Live Stream of Images and Sensor Data
Our team was inspired to mimic Twitch Plays Pokemon, a game where hundreds of people are able to influence gameplay from the chat of a live stream. Our interpretation of this concept is to integrate an iRobot Roomba with Twilio in order to allow many people to remotely control the robot at once through SMS.
What it does
Our Roomba has the ability to drive around remotely when a text message with the proper syntax is sent to a Twilio phone number, which processes the message received into an action on the robot. The robot has the ability to move forward, backwards, turn clockwise and counterclockwise through the SMS remote control. Movement that is relayed to the robot is then run through an algorithm which graphs the motion of the robot itself– the data being presented on our website. Lastly, the Roomba has a Microsoft camera which takes pictures periodically and sends it through an assembly line of tasks. One of those tasks analyzes the image using Microsoft’s Cognitive APIs in order to distinguish the objects in front of the robot.
How we built it
The Roomba is modded to have a Raspberry Pi, webcam, and several portable batteries mounted to it. The Raspberry Pi runs a Python client script that listens for SMS commands from a remote Flask Python server. The Pi also sends a variety of Roomba sensor and image data to our website, located at https://twilio-plays.herokuapp.com. For example, the Pi integrates Microsoft Cognitive APIs to analyze the contents of an image and relay the results to our site. The website also tracks the motion of the Roomba in 2D space on a graph created using Chart.js.
Challenges we ran into
Some of the technical challenges we ran into involved integrating services with the Raspberry Pi. One area of difficulty was navigating the internet setup here and setting up communication between the Pi and the Flask server controlling Twilio. Separating the two components was helpful to maximize performance on the server while limiting the load on the Roomba’s hardware. The client on the Pi sends a request to our endpoint that contains the queue of commands sent from people using Twilio.
Accomplishments that we're proud of
Navigating the Raspberry Pi entirely through the command line and never using a GUI. Sending a video stream from the Roomba to our website. Mapping the robot’s movement accurately on top of streaming the motion on our website.
What we learned
Connecting to another local computer on a public Wifi network can be extremely complicated. Gained experience working with Microsoft Cognitive APIs. Learned to interact directly with the hardware of the Roomba to read sensor data.
What's next for TwilioPlaysRoomba
Smoother video streaming from the Roomba. Improve precision of the 2D motion tracking. Integrate text-to-speech into the Raspberry Pi.