STEVE

Inspiration

JARVIS is the famous virtual assistant of Tony Stark, and it displays whatever Tony requests in a physically interactive hologram. While we are limited in that we do not have access to physically interactive holograms, we are able to replicate JARVIS's creativity, visualization and understanding with... STEVE!

What it does

STEVE takes in verbal inputs, full sentences in whatever phrasing, and interprets what you would like it to show. For example, you can say "STEVE, give me a pendulum", and STEVE draws out a pendulum. Then, you can give STEVE what angle the pendulum should swing from, or change the length of the string, or the mass of the ball, and STEVE will adjust the animation accordingly. You can also tell it to plot the relationship between any two variables in the simulation, and STEVE shows you a live graph of the relationship of the given variables.

How we built it

We used a Physics engine to simulate the physics of our objects, tweaked it accordingly to fit our desired format/look, and implemented it to display it as we wanted. We also used co:here's NLP, with a training dataset of 350 examples to make STEVE's understanding of inputs flexible, and voice-to-text to translate the verbal inputs to strings that STEVE can interpret.

Challenges we ran into

It was incredibly hard to create the examples for STEVE to learn from, specifically examples that would teach it to be able to understand all given sentences. The NLP would at times train STEVE to misunderstand us, and we had to change our dataset accordingly. The physics engine we used also required a Herculean effort to understand and implement, and also to pick apart to display it to fit our format. We then had to code quite a complex HTML page with CSS to create a user-friendly interface, that is fully interactive at every stage.

Accomplishments that we're proud of

We love how our front-end looks, going as far as to include an animated waveform as a visual indicator of detected audio input. The implementation of the physics engine also makes us incredibly proud of ourselves, because it was a vastly complicated system that we had to pick apart and understand, as well as be able to weave into our code. We also felt like proud parents when we saw STEVE get better and better at understanding us through the NLP.

What we learned

We learned that data abstraction is hard! The physics engine was difficult to understand, but through this experience we learned how physics engines work and how it simulated the movements of the objects in it. We also learned how precise our example dataset has to be for the NLP to learn adequately, and in a broader sense, the applications of NLP.

What's next for STEVE

Currently, we are focused on Physics models, but we are hoping to be able to implement all kinds of visualization, including drawings, images of any kind, or even simulations of experiments from other areas of science.