Inspiration
We wanted to simulate Sherlock's amazing memory skills using a combination of VR an Natural Language Processing
What it does
A user selects either a speech or some notes to memorize. The content is then parsed using NLP techniques to extract useful information. Users can then enter Sherlock's Mind in virtual reality and select the content they want to memorize and choose a specific world to enter. Once they enter this world, they find colored cards with summaries and strange objects. By matching up these strange objects with the colored cards in various locations, they can successfully create useful associations in their brain.
How we built it
Our project has two major components: NLP backend and VR frontend.
NLP backend We first assemble the text into reasonably sized paragraphs. We then use various NLP techniques to extract information from these paragraphs like: Keyword Extraction: Microsoft Text Analytics, Sentiment Analysis: Microsoft Cognitive Services, Summarization: Gensim and Microsoft Linguistic Analysis API and served these from a flask server.
VR frontend We put the scene and different menus together using a combination of Steam VR and VRTK. Using the Vive Controller, we created User Interaction elements like grabbing, carrying, teleporting and object generation. Finally, we integrated language elements from the backend to create cards: eg coloring the cards depending on the sentiment expressed.
Challenges we ran into
Trying various frameworks like Newton VR and VRTK on the frontend to figure out which one works.
Accomplishments that we're proud of
Lugging around and setting up a Vive and 20kg PC setup in Pauley Pavillion.
What we learned
There is a bug in every square virtual inch of exploring space in VR.
What's next for Mind Palace VR
Adding a menu for random objects to be placed, accessing flashcards from anywhere in the world, better NLP
Log in or sign up for Devpost to join the conversation.