-
-
our learning flow vs. traditional
Inspiration
We started from a problem: People want to learn new languages but it takes a long time to learn the language in the traditional way of using textbooks. It’s hard to start communicating with others, which is the actual purpose of language.
The traditional pathway is to learn the knowledge then use it. We propose a learning-by-doing approach. It imitates how toddlers naturally learn their native language: driven by curiosity and to express, they try, ask, and get the right way to say something from their parents.
It’s made to be user-expression-focused, than curriculum-focused.
What it does
The main feature is an interaction interface where users can freely talk about anything (daily life, self-reflection, etc.) in imperfect language or even scattered words, and our Perceptron AI model will turn it into complete and accurate expressions and display it on the screen, with color-highlighted corrections.
Users can choose whether they want to see the feedback in real time or after recording stops. The former is designed for beginners who need more assistance and the latter is designed for advanced learners with fewer mistakes and are able to make complete expressions already.
Users can pause the recording and chat with a personal language tutor if you don’t know how to say something completely or want to ask questions. Users can ask in any language they are comfortable with and the tutor will give responses that match their language level.
How we built it
We first discussed as a team about system design and data flow. Then Olivia and Tanvee made prototypes of our web app on Figma. We then turned our design to code with React framework with Cursor and GitHub copilot to help with building.
At the same time, our other teammates worked on the backend. Soojeong set up the Perceptron API integration for speech and text conversion with the API documentation online and William started on building a database, though later we decided to not use a database to reduce complexity.
Then we added the chat feature, also integrating Perceptron API. Key frames of the video will also be passed to Perceptron AI for context.
Last, we deployed our website on Vercel.
Challenges we ran into
We encountered multiple API errors(401, 502). While trying to fix them ourselves, we also reached out to mentors and sponsors. At the end they were resolved.
Making grammar suggestions clear and easy to review. We spent a lot of time designing the UI to provide clean visuals for language learners with as little redundant information as possible. Through rounds of testing we managed to implement our design accurately.
William found it hard to save voice and video data in the database so we decided to not use a database for our MVP.
We also encountered multiple times of merge conflicts due to accidentally committing to wrong branches. We learned to solve that by switching other branches to main and manually fixing the discrepancies.
The scope of our product was too big that we couldn’t finish all the features but everyone got to sleep well! (big win in a hackathon!!)
We also had to find a balance between accuracy and speed for voice transcription. We ended up providing two options for the users: either real time transcription and analysis with the browser’s transcription feature or delayed analysis with Perceptron’s more powerful AI model.
Accomplishments that we're proud of
We built a tool that redefines language learning experiences and we can’t wait to use it ourselves!!
We were able to complete all key features we intended. We were to pivot flexibly for the database, resolving transcription delay, etc.
We provided constructive feedback to each other all the times and constantly polished our web app. We multithreaded our work and were very productive.
What we learned
We learned that making a prototype and discussing the overall systems design as a team will make the building much smoother.
We learned debugging skills and asking for help when stuck.
We learned to focus on the core feature first and start simple. First handling the text, then speech and last video.
What's next for Phrase
Add other optional features:
Review: At the end, AI will give you a review sheet of grammar mistakes you made as well as expressions you asked about. They will be saved and might pop up later for you to refresh until you master them!
It’s also a diary: Save those precious moments like a diary! We will build a database to save all your thoughts and videos.
Make a vlog easily: AI edits the videos and transcript together with music. It can be saved and shared on social media! Which might be a trend and go viral! People would love to show that they know many languages.
Robustness and accessibility:
Train AI models to recognize words spoken with accents which many beginners might have and in noisy environments.
Accuracy:
Train AI models to take in context of what the user said and the video to better interpret their intended expressions.
Built With
- chatgpt
- copilot
- cursor
- figma
- perceptron
