Rosetta Kart

Rosetta Kart Gameplay!
Drive through the correct gate
A player using the Bose Frame

Inspiration

Learning a new language can be difficult, especially for character-based languages, where you have to build up a large number of associations between symbols and meaning. Rosetta Kart gameifies learning Mandarin using the Bose Frame wireless headset!

What it does

The Bose Frame has off-ear speakers, a gyroscope, and an accelerometer. The Rosetta Kart game has the player drive down a track while the Bose Frame reads out English words. The player then steers the Kart by tilting their head, which is picked up by the Frame. The goal is to steer the Kart towards the correct gate labeled with the Chinese character associated to the dictated English phrase. The game builds a direct association between English meaning and Chinese characters, reinforced by visuals of the object in the environment surrounding the road (for example, a dumpling shop appears on the side of the road when 饺子 is dictated!). This Demo showcases a lesson on Chinese foods such as dumplings and fried rice.

How we built it

We utilize the Bose AR SDK with the game-engine Unity to implement this project. The Bose Frame is a sleek tool that takes a different approach to AR – audio first. When learning a new language, it is incredibly useful to take advantage of sight as well as sound for deep and effective learning, a task that the Bose Frame is uniquely suited for!

Challenges we ran into

The Bose Frame is a fairly new device with very little online resources or notes from the community, so debugging was especially difficult.

Accomplishments that we're proud of

The applications online of the Bose Frame are mostly recreational, so we were excited to figure out a pedagogical application for the product. We feel that language learning is an application that really takes advantage of the device's features, and on top of that the project gameifies the learning process. Also, none of us are experts in Unity, and none of us have used a Bose Wearable device before, so it was exciting to be able to put this project together in just 24 hours.

What we learned

How to generally use the Unity game engine
The features of Bose Wearable devices
How to use the Bose AR SDK
Integration of the Bose Frame to Unity
How to make a real-time game in Unity
Coordinating a device with an gyroscope and accelerometer with the real world

What's next for Rosetta Kart

This game structure is very versatile. Any words may be dictated with the accompanying characters and even visuals on the side. For more well-rounded learning, the game can also be switched to dictate in Mandarin and have the gates labeled with English words, Pinyin, or even pictures. This game can be extended to other languages, or even other subjects one might wish to teach! We would like the build a back-end that allows us to quickly build other lessons.

Built With

bosearsdk
c#
unity

Submitted to

BostonHacks 2019
- Winner 1st Place Prize

Created by

Unity object design and C# programming, general coordination and polishing

Ajeet Gary
Enrolled at New York University Courant for an MS in Mathematics, Dual Degrees in Math and Computer Science from the University of Maryland.
I find the fbx files which used to decorate scene, and make the learning materials.

Chia Ching Liang
Came up with the idea of using Bose Frames as a game controller in a MarioKart-inspired racing game, and using obstacle blocks to turn the game into a language learning app.
Implemented the scoring system when players runs through a given obstacle block, as well as the logic for audio cues to be played on time in relation to the player's Z position to the block, and the block itself as a game object.
Implemented other Unity objects including visual cues of correct answers as image posters on the billboards, and other designs of Chinese buildings as game objects to complement the visual theme of our game.

Leslie Meng
I worked in Unity to design the track, incorporate physics of Kart motion, and calibrate the input signal from Bose wearables motion sensors

Anna Mun
General research on the Bose AR SDK and configuring Bose Frames to be used as a game controller in Unity. Implemented an AudioManager in C# that controls audio files and allow for easy configurations of audio spatialization to take advantage of the Bose Frame's spacial awareness capabilities. Overseeing merging of different Unity components. Managing GitHub Repo. Making demo video.

Miffy Chen