Nǐ Howdy

The Weekly Planner Page
The main dashboard of Nǐ Howdy
The AI Conversation Page
The Supplemental Materials page
The Speech to text YouTube Live Subtitler page.

Inspiration

The main inspiration that ignited our idea was Duo-lingo and similar applications. The typical language learning app is made to attract users through short-term learning of vocab words and keep its users on the app through a gamification technique and low time consumption. However, this usually is not effective enough to facilitate real learning, as many languages require active use in social situation and studying to refine the material for the user. We designed this app to try and fix those problems by provided a longer term study plan that can fit the users schedule, set study times assigned each day with assessments to evaluate user progress, and Artificial Intelligence to simulate having a conversation with a native speaker and helping the user learn from real time YouTube video translations.

What it does

This app utilizes several features to simulate a real learning environment. It assess the user, giving them a grade and allowing them to review material that they may have struggled on, or help them refine what they have learned. The site also provides downloadable PDFs to practice handwriting, especially for languages that are character-based. There is an AI chat bot that will talk to you in real time, testing you not only on your knowledge of what to say, but also your pronunciation of the words as well. The YouTube Video live translation feature is meant to help users by giving them real time subtitles in there preferred learning language, and allowing them to select words that were difficult so they can study them later.

How we built it

We created the app using a React Type Script front end framework and Next.js as the back end framework. We implemented auth0 as our authentication API to allow for account logins that also help track the users progress in the learning over multiple sessions. For the Artificial Intelligence we utilized the Google Gemini and ElevenLabs APIs, incorporating there text-to-speech and speech-to-text features, allowing for real time conversations and video translations.

Challenges we ran into

A lot of the major challenges we ran into were with integrated the A.I. APIs. We ran into problems with calling the API for Elevenlabs initially, and also getting the latency for the video transitions down. After a bit we implemented a different API (DeepGram) for the video translations as it was just more efficient, and then for the live conversations we kept ElevenLabs as it was more efficient at the voice recognition. This was the first time we have tried to implement A.I. into a project so it was a nice learning experience.

Accomplishments that we're proud of

We were stoked to get the A.I. conversation and video features of the project up and running. The ability to choose words from the video and save them into a "study later" bank, worked really well, and was satisfying to use. We were also happy with being able to implement new APIs such as ElevenLabs and Auth0, and have them work successfully!

What we learned

We learned the React framework and also how to efficiently do authentication, allowing us to easily track our webpages and keep persistent user data. Another big thing was learning how to implement the Gemini and ElevenLabs APIs, and get it to operate seamless on our web app without it looking too out of place.

What's next for Nǐ Howdy

We want to refine the app and allow for more language support since for the demo we mainly focused on English to Chinese. Another thing we would like to implement is a assessment generator that can create tests based on words that the user has saved in there word bank, or words that they have struggled with on previous assessments.