Multivac

Multivac is an interactive multimedia text console which leverages AI to bring any piece of text to life.

Comment

Inspiration

When I was 10, I played a text-adventure game related to simply traversing the world. It was small, there was no GUI, but even then, outside of Minecraft, it was one of the few games who captured my interest because of it's wide field and untapped ability for exploration. Recently I wanted to revisit that game, but I found that in the time a lot had changed. As such, we decided to re-explore that field ourselves using Generative AI to capture the non-linear storytelling, the exploration, and enjoy the books we read nowadays in a new refreshing light.

What it does

Multivac stands for Multimedia Visual Adventure Console, because it uses both text and images to turn any piece of text into an interactive adventure. Basically, you can upload any book, and turn it into an interactive fiction game.

Multivac processes and chunks the uploaded story storing it in a vector database. From this, it creates a list of major states with chronological timestamps from the story that are pivotal points in the book that -- unless monumental work is done on your part -- will occur again. This allows Multivac to know what the main plot points of the story are, so that it has something to work off of. In addition, it helps the list of states helps to tie you back to the major plot points of your favorite books, allowing you to relive those memories from a new and fresh perspective. From this, Multivac uses relevant info from the story vector database, chat history vector database, state list and current timestamp to generate responses to the user that move the story along in a cohesive manner. Alongside these responses, Multivac generates images with Stable Diffusion that enhance the story being told -- allowing you to truly relive and feel immersed in the story you're writing.

How we built it

For frontend, we used React and Typescript. For backend, we used Flask for server management. We used Langchain for querying Claude and Anthropic and we used Llamaindex to store story data and chat history in a Vector database and to do vector searches through Llamaindex's query engine. We also used Replicate AI's API to generate images with Stable Diffusion. For persistent database systems, we decided to create our own individual SQL-like system so we could avoid the additional overhead that came with SQL and it's cousins.

Challenges we ran into

There were 2 major challenges we ran into. The first one was related to building Multivac's response pipeline. It was our first time using vector databases and Llama-Index as a whole to search for relevant details within large bodies of text. As such, as you might imagine, there were quite a few bugs and unforeseen break points. The second was related to the actual database system we were using. As one would imagine, there's a reason SQL is so popular. It wasn't until 3:00 A.M. at night, when we were creating and debugging our own operations for getting, writing, etc... from this database system did we truly understand SQL's beauty.

Accomplishments that we're proud of

We are proud of creating a persistent database from scratch, as well as being able to build all the features we set out to create. We are also proud of completing the end to end pipeline using Llama-Index and LangChain (which was surprisingly difficult). We are especially proud of our UI as -- if we don't say so -- it looks pretty slick!

What we learned

Couple of things. Firstly, if your going to use a database system or your application requires persistent storage, use SQL or a founded database in the field. Don't make your own. Just because you can doesn't mean you should. Secondly, we learned how to use Llama-Index to process, chunk and use text end-to-end in a vector DB for LLM calls. Thirdly, we learned how tiring 1-day 2-night hackathons can be and how taxing they are on the human spirit. Lastly, we learned how cool Harvard and MIT looks.

What's next for Multivac

As of writing, we currently use a list of states to track where the user is in the story, but we want to expand Multivac to instead be a self-generative state machine with stochastic transitions. This would make the world feel more alive and grounded in reality (since the world doesn't always go your way) giving you further immersion to explore new paths in your favorite stories. This would also create more control over the story, allowing for more initial customization on the user's end regarding what journey they want to take.

Built With

Share this project:

Updates