ToneSurfer Mandarin

Inspiration

80% of non native Chinese-speakers experience tonal fear, meaning that they are scared to speak due to fear of messing up tones. Personally, I have been scared and embarrassed to speak because of that very reason. We created ToneSurfer Mandarin to make this learning process more accessible, engaging, and fun for Mandarin learners, especially little kids starting out.

What it does

ToneSurfer Mandarin says a mandarin word out loud, allowing the user to hear the tone of the word. Then, it watches as the user traces the tone out in the air, giving audio feedback depending on where the person's hand is (if it's up the tone is higher, and if it's down the tone is lower). This allows users to hear the shape of tone that they are drawing. When the user traces the correct tone, ToneSurfer tells them that they are correct and tells them what that specific word means before moving on to the next word. It cycles through multiple words before moving on to the next level.

How we built it

Although we vibe-coded most of the app, we went through many iterations of slightly different code in order to get the features correct and debug. Also, we hand drew (online) our logo which we integrated into our app.

Challenges we ran into

We ran into a couple of issues with how to correctly detect what shapes the user has traced, as well as with the audio playback. We managed to fix these issues through trial-and-error as well as figuring out the specific root of each problem.

Accomplishments that we're proud of

To be honest, we are super new to hackathons, so we are very proud that we were able to get our app working. We are also new to using machine learning tools like mediapipe, and are proud that it works decently in our app.

What we learned

We learned a lot about how to smoothly integrate different frameworks into our own code and how to work with APIs. We also learned how to more effectively debug.