Inspiration

A toaster doesn’t seem like a complex system -- bread goes in, bread goes out. However, we decided to infuse this simple machine with capabilities ranging from optical character recognition to network monitoring and the ability to drive on wheels. With the help of small Raspberry Pi computers that could communicate data from the components, we worked on making this possible.

What it does

Embedded internally with four separate raspberry pis, the toaster has a number of varying functions. Using machine learning-based optical character recognition, the camera attached to the toaster is able to look at a written representation of Python code (e.g. handwritten or typed on a sheet of paper) and compile it from the image that it obtains. We also worked on setting up a webpage to read the visual input received by the camera and the resulting output of the compiled code, to demonstrate what the toaster was doing. In addition, the toaster stands on wheels and can be physically maneuvered. We implemented the manual drive through Python-based keyboard controls, and also worked towards building an autonomous drive mode using an ultrasonic sensor. Several features we also worked on were the ability for the toaster to order pizza through the Dominos API and send tweets through the Tweepy API. The toaster monitored the WiFi networks and also captured data on airplanes passing overhead with a radius of one hundred miles, such as speed and altitude, through an antenna. And finally, the toaster still toasts!

How we built it

For the physical setup, the outer casing of the toaster was removed to allow for Raspberry Pi’s and wiring to be connected inside two of the toaster’s slots. The other two slots were saved for toasting bread, and the two halves were divided using ceramic tiles to ensure that the heat would be minimally transferred to the side with the other electronic components. The wheels were attached to the base of the toaster, along with a caster. Many of the parts needed for the physical setup were soldered before they could be integrated into the design. To make the toaster capable of processing written code through its camera, we used tools available through the Open Source Computer Vision Library (OpenCV), a machine learning software. The input and output of the written code recognition were stored into files to be accessed by the webpage. For the manual drive mode, we created a socket in Python through which the Raspberry Pi’s could receive keyboard commands. In building the autonomous mode, we used an ultrasonic sensor to gauge the distance between the toaster and nearby objects, leading the toaster to turn if it approached too close to another object.

Challenges we ran into

Throughout the design process, our implementation of initial ideas changed in response to unexpected difficulties as well as newly proposed approaches. In programming the motion of the toaster, we first planned to use Java, but partway through decided to switch to Python as we realized it was better suited for the task. Unclear readings from the camera sometimes made it difficult for the optical character recognition software to fully read the code shown, resulting in an inconsistent ability to compile. Segmentation faults also occurred repeatedly in the code for the Google Home API, impeding the coding process.

Accomplishments that we're proud of

Of the objectives we set out to accomplish, we were happy to have successfully implemented several of our main goals, and learned much more than we could have expected in the process of working through the challenges presented by other aspects of the design. We’re super excited that the toaster can drive around, and that the implementation of written code recognition worked impressively overall.

What we learned

Even though two of us were returning members to HackTJ, all of us had something to learn during the hacking time. Our two new members were not previously familiar with the hardware and software of a Raspberry Pi, and began with learning about its functionality. In addition, one of the new members was able to learn Python in the duration of the hacking time for the purpose of writing code to drive the toaster. One of the two returning members was able to became much more proficient with the Google Home API. In addition, we learned how to compile a library in C for OpenCV.

What's next for IoToaster

Some ideas which we hoped to implement today but were not able to completely finish as features of the toaster are ordering pizza and sending tweets. As future extensions, we would also love to be able to make the toaster enabled with speech recognition and be able to rate the quality of the toast that it made using machine learning software.