Rosie

Rosie is the retro-futuristic innovative homework helper designed to reduce the load parents undergo when they are helping their kids. But, Rosie provides fun and interactive experiences for all.

Comment 1

Inspiration

The inspiration for Rosie came from the Jetsons, where Rosie (formally Rosey) was a robot and maid for the Jetson family. Given the retro-like theme of Rosey in the show, and with this year's theme being Retro Future, Rosey was the perfect fit to bring into real life! We were determined to use AI in some form, but also made sure we integrated plenty of retro electronics and hardware, so this was an easy decision for us to make.

What it does

Rosie is a physical assistant that comes in the form of an oscilloscope that shows her face. Her primary job is to help you with any of your needs in terms of questions you could ask. However! She is specially trained to help your kids with their homework so that you can do other important tasks in your lives. Her calm demeanor and desire to find the best path to excellence make her a perfect fit in your kids' lives!

How we built it

Many different systems of software and hardware went into the creation of Rosie. First and foremost, we were determined to create Rosie in a retro format. Our first idea was to obtain a CRT monitor, but due to our lack of access to a CRT monitor, we decided to try something truly unique. We have utilized an Arduino Nano to function as a digital-to-analog converter that can be probed by an oscilloscope. This means that Rosie's facial expressions on the screen are generated by bit-maps and output by the Arduino Nano into analog waveforms that can be read by the oscilloscope. Rosie also comes with speech-to-text functionality on a 20x4 LCD for easy viewing of her responses. Furthermore, we wanted a way to take in any question a user could ask and formulate some response, and that is where we brought ChatGPT into the mix. ChatGPT allows us to take user-submitted questions and respond in real-time with mostly factual information. We of course also wanted Rosie to have a voice, and that is where we used default system text-to-speech integration to take that response and produce a cute robot voice!

Challenges we ran into

One of the first challenges we ran into was being able to do heavy computations like voice-to-text on a Raspberry Pi or an Intel NUC. We realized pretty quickly that our only solution in the budget and time frame we were constrained by was to use one of our personal computers on-site to handle the heavy lifting. This unfortunately also introduced another challenge, networking, where we realized we would need to communicate between our mini computer to the personal computer.

Another challenge we ran into was displaying Rosie's emotions and facial features on the oscilloscope. It took a lot of parameters being changed within the firmware, circuit-wise, and also within the oscilloscope itself.

Accomplishments that we're proud of

The team is very proud of utilizing a display that is both retro and non-typical for the main interface of the robot. It gives the device a unique feeling that can't be replicated anywhere else. In addition to this, our integration of both software and hardware is a marriage that can't be understated. Each system of this fantastic device works well together, which can be difficult the more parts and systems you add, both hardware and software. Finally, we are proud of the fact we were able to offload the computing to a remote desktop in favor of a smaller form factor PC to complete the package in a smaller form factor, reducing the device's overall footprint.

What we learned

As a team, we learned many important skills and problem-solving techniques throughout the 24 hours we worked on Rosie. One of those is networking, where we had to figure out how to network with an off-site machine that would handle our voice-to-text. Our solution to this problem was to use a VM we got from Digital Ocean that would act as a relay of sorts between our Intel NUC and our processing machine. It was critical that this communication not only be fast but also fairly error resistant as if the processing machine broke in any way we would have to have someone run to another room to fix it. The initial implementation of this took a lot of learning as we were unsure which protocols to use and whether or not to do any fancy techniques, but our final solution ended up being quite simple to fit into the time constraints of the hackathon.

The other is system integration. Just doing software or hardware by itself is difficult, but creating a full system with software and hardware requires a lot of communication between the two teams. With this project being so interdisciplinary, it required us to pay attention to the software/firmware being written to stay within spec to receive and send commands.

Finally, we learned that oscilloscopes can be used as displays! While not entirely practical, it is quite cool getting something like a picture to display on something that typically only measures 2D waveforms.

What's next for Rosie

As a team, we would love to keep Rosie intact and expand on her functionality. Things such as:

  • More facial expressions.
  • Deeper integration.
  • And, more interaction!

Built With

Share this project:

Updates