Sherlock's Palace

Sherlock's Palace
The Mind Palace is an Idea seen in the show "Sherlock". It is a method for storing information that uses association to improve memory
Utilizing Sherlock's Palace, we can learn to store and retrieve information more quickly and accurately through mental association.
The Brain best remembers by association, so the more emotions or images we can attach to a piece of information the easier it is to remember
The Brain remembers times of great emotion best, often with vivid detail. This is why no one ever forgets the day they win the lottery.
By having a 3D space to walk around in when memorizing something, We make emotional and visual connections, which improves our memory

Inspiration

The brain is a powerful thing. It has the ability to make complex calculations in microseconds, and to remember events in exquisite detail, but still we as humans forget things all the time. This is because we need to format the things we wish to remember in a way that our brain can remember easily. In the same way that emotions and images help people to remember events in their lives, Sherlock’s Palace will help you remember information through the power of mental association in a VR environment.

What it does

Sherlock’s Palace is a VR environment that is auto-generated based on content you provide. You give the palace a some text that you wish to memorize (maybe a speech or a poem), and Sherlock’s palace will auto generate a 3D environment that you can use to help remember your information. Based on the text provided, the palace constructs a series of Rooms that each contain unique images and shapes connected a specific part of the text you are trying to memorize. For example, if you wanted to memorize sir Patrick Henry’s “Give me Liberty, or Give me Death” speech, the palace might generate a room based off the first line “No man thinks more highly than I do of the patriotism...” that contained some images of skyscrapers , brains, or american flags (highly, think, patriotism). Whereas in the last line of the speech “Give me Liberty, or Give me death” you may encounter a picture of the statue of liberty accompanied by the grim reaper. These images, shapes, and the text that generated them found in each room will help you to be able to remember the speech or poem in its own separate parts. And put those parts together with a mental map of the route you walked and what you experienced inside the VR environment.

How we built it

The core of the project was build using Unity, along with scripts in Python and C#. The user would input a text file, and then a python script would read from the file, break the text into different parts, and then ping different APIs for information related to that line. We used Microsoft Cognitive Services APIs (Content Analysis, Sentiment Analysis, and Bing Image Search) to grab the relevant data and keywords from the phrase in question, and then grab the first few image results based on that passage. These images were passed over to a unity project that used the information (Number of phrases, images, etc. all stored in JSON format) to auto generate a series of rooms (number determined by number of phrases to memorize). We implemented the transport function into the Unity project using a series of wqeb The rooms were generated using a flexible entrance/exit vector computation algorithm that enables the project to take on more shapes than a straight line in future editions. Inside each of the rooms, we displayed the text and images for the phrase the room was based on. We also used a Google Text to Speech API to have the phrase be spoken to the user. The demo was done using the Unity VR interface inside the Unity Editor.

Challenges we ran into

This was the first time that anyone on our team had ever used Unity or worked with VR (one team member had minimal experience), so learning to navigate the new environment was definitely a challenge. Generating the rooms was a challenge, as we used a flexible algorithm that looked at the entrance and exit points for each room and rotated the new room based on vector calculations rather than just hard-coding the change in the location variable. This enables the project to be easily changed to allow for different paths in the rooms, such as rooms with 3 doors or hallways with corners rather than just the straight line of rooms seen in the demo. The vector calculations to make this happen were tricky. We also had some hardware problems that prevented us from accessing certain APIs, so we had to use different ones.

Accomplishments that we're proud of

We had a novel idea that inspired all of us and were able to pull off a prototype in 24 hours. This project is designed to help you redefine how you think and to help you understand how your brain works. It also causes you to consider our current education system and how it could be improved. In this aspect, the project is revolutionary, as we are not only out to push the boundaries of computer science, but also to push the human experience to its peak.

What we Learned

I (RShiner45) learned a lot about APIs, how to access them, along with python scripts. This is the first time I have ever used unity, and I learned a lot about 3D Development. I also acquired a greater appreciation for Game Development. I learned a great deal about Project Management and Workflow. Everyone on the team worked hard to keep each other focused and maximize each of our strengths and to learn from each other. Most importantly, I learned how to work with a team of people i had never met before to create a novel idea, and implement it in 24 hours. This was my 2nd Hackathon, and I learned more than I could ever imagine or list in one paragraph.

What's Next for Sherlock's Palace

Sherlock's Palace would benefit greatly from the user being able to make their own floor plan that they can walk through. This would help greatly with mental association and the overall intent of the project. We also meant to implement more use of the sentiment analysis API to edit the color of the room, but didn't have time for the scope of this project. Lastly, Sherlock's Palace could easily be adapted for all sorts of uses from disabled assistance to a new education style entirely. Even digital art museums could be created using Sherlock's Mind Palace as a template. Wherever we go, Sherlock's Palace will help us to remember where we started.