Inspiration
We have used multiple language learning apps and also studied languages in school. These things can only get you so far if you do not have an environment that encourages you to speak the language actively. Furthermore, trying to converse with native speakers is very intimidating and is a pain point that we have heard of from a lot of people. And combined with the love for cute miniature avatars, we came up with Lingo Bud.
What it does
Breaking through traditional boundaries of screen based learning, LingoBud gives you an opportunity to learn in XR within your own space and engage verbally in the language you are learning. Whether if you're learning grammar points of vocabulary and phrases, LingoBud’s voice and gesture-driven interface make your learning journey both immersive and interactive. Key Features:
- AI generated characters with unique personalities and names: Allows diverse possibilities of engagement in real-time scenarios. The characters will also remember past conversations with you. And some might even start to speak more openly and informally with you as time goes on. A focus on conversation and voice input: to give users the opportunities to speak in the language they are learning with real responses.
- Communicate your language learning objectives to the Mother Tree. She will curate a study plan for you and grow seeds that you can plant and spawn learning buddies with.
- There are 3 types of learning buddies: Vocab Buddies that teach you vocab, Grammar buddies that teach you Grammar, and Conversation Buddies that have unique characters assigned to them for specific scenarios (e.g. Waiter, Bus driver etc.). You can also freely chat with all of them.
- Real time Emotion and expressions feedback: Learn as if you’re talking to real friendly people. They will also remember all of your conversations.
- Fun way of Learning and tracking: Ensure you meet your learning goals by growing the plant. Spatial engagements: Learn with a world of language buddies in your own environment.
How we built it
I used Unity with mainly the Meta SDK, VoiceSDK, OpenAI Plug in, and Azure Speech Service. I used the Voice SDK to transcribe speech. All the generations for a language learning plan, seed and plant creations, avatar responses, and realistic expressions are achieved through meticulously crafting the OpenAI prompts. All Speech to Text is done through the Azure Speech Service.
Challenges we ran into
- We started brainstorming around December 17th which meant that we were almost 2 weeks behind schedule, so working under the pressure itself was quite challenging.
- Getting OpenAI to follow strict guidelines so that the generations of seeds go smoothly. Regex was used to unpack the messages which meant that Chat GPT needed to generate in a precise format.
Accomplishments that we're proud of
- Created a unique language app with a level of polish that is useful and fun in a short period of time.
What we learned
- How to effectively integrate an LLM into an app, and all the nuances that come with it.
- AI will actually play a huge role in language learning efficiency.
What's next for Lingo Bud
- Resize all avatars based on your room's space availabilities. And also have the ability to enter any new scanned environments and have your saved avatars be introduced to them.
- Integrate most languages into the app.
- Have a stats based progress tracker that shows how much you have improved. Make it so that you can even converse directly with the mother tree and she can verbally give you feedback.
- Each character right now is in its bare bones form. We want to integrate a large library of accessory customizations such as hair, hats, glasses, eyebrows etc. And the combinations will all be generated based on the character and personality of the avatars.
- Have a call system where a specific avatar will come find you by having you just call their name.
- Have avatars interact with each other and you are able to jump into a conversation that they are already having. Avatars will also chat about you if they have recently talked to you.
Log in or sign up for Devpost to join the conversation.