Everyone has moments where we need to recall long lists of information or ideas whether it is studying for a course, remembering concepts, or even the items on a long shopping list. The Ancient Greeks used Memory Palaces, an imaginary place in the mind that together with the surrounding environment acts a storage for knowledge.
The Memory Palace is a concept that allows a user to walk through a familiar place in mind (like your room) and recall different elements of the surroundings that you associate with an object. Along the way, the user can use these different elements to remember a list of concepts that are hard to recall. The idea is that people generally have a good memory for objects in a place that they know that they have for ideas or words.
What it does
We introduce a concept of learning with a space you live in. In our application, the user can become Sherlock Holmes and evolve the evolution of the brain training, removing the cognitive burden of creating this map in your head. We are using AR and Microsoft HoloLens _ to build this knowledge map. _
Using spatial mapping — one of the key features of the Microsoft Hololens — we let people use the surrounding physical environment as a memory palace (a living room, office etc.). Imagine your room as a whiteboard, where all the things you want to remember are augmented objects (3d models, voice recordings, text annotations, images) and you can learn them together within your personalised space.
Our application allows you to build a map using different objects from a library and attach them to the point where you want to associate them with the physical world. This allows the user to associate the location or object with the specific concept that the user is trying to remember.
To illustrate this idea, we are using the _ periodic table of elements. _ For example, when the user needs to remember the facts about potassium, he or she can select the object associated with that item, e.g. a banana, and place it in a chosen place such as on a pillow and so on. Once the object is placed, the user can also add annotations to these objects. At the end, you can complete a quiz and try to recall all the objects.
How I built it
We build it with Microsoft HoloLens, C#, HolotoolKit, Unity, VisualStudio, Maya and Adobe CC
Challenges I ran into
The Hololens is a challenging technology to use. Since it is expensive, it is hard to obtain one and use it for experimentations. Also, there were compatibility issues with the version of Visual Studio working with a specific version of Unity. Another challenge that we had was Hololens picking up movements of people on the space as objects which interfered with the interactions that we wanted to add to the objects in our application.
Accomplishments that I'm proud of
Getting something finally to work. It is hands-free real-world interaction! We made a good representation of the idea, but it is reasonably far away from what we have in our heads (we would want to add more interaction, more guidance for the user and definitely more interactivity)
What I learned
Good things take time. We learned a lot about integration, cooperation and collaboration. We deepen your understanding in spatial mapping, gaze and voice inputs, learned more about the workflow between Unity and Maya ( the structure is a key).
What's next for Within
We would continue to work on this project. Spatial mapping is a key thing for AR, and while waiting for an upcoming version of HoloLens, we want to have this app as a working tool. In future, this application can be used not only for memorising things and learning complex concepts but also as a mood board for collecting ideas, inspirations, learning languages (new Italian verbs are on the left wall and some of the new French adjectives near my staircase).
Team lead - Alla Vovk, firstname.lastname@example.org, +4477278778797 Members - Nathan Yu, Richard Hsu, Qinqyue Li, Elio Pajares Caterogy - Education, AR for Good Place - 3rd floor, table 43