SpyLingo

• 

  The introduction agent generates missions for the user to complete in their native language

• 

  Talking about our favorite foods in Spanish to complete the mission

• 

  A Russian user learning Chinese

• 

  Entire messages can be translated into the user's native language

• 

  Users can select an individual word to understand it in their native language in context.

• 

  The system can catch mistakes in the user's foreign language message and explain how to correct it in their native language

About

Learning a foreign language can pose challenges, particularly without opportunities for conversational practice. Enter SpyLingo! Enhance your language proficiency by engaging in missions designed to extract specific information from targets. You select a conversation topic, and the spy agency devises a set of objectives for you to query the target about, thereby completing the mission. Users can choose their native language and the language they aim to learn. The website and all interfaces seamlessly translate into their native tongue, while missions are presented in the foreign language.

Features

• Choose a conversation topic provided by the spy agency and it will generate a designated target and a set of objectives to discuss.
• Engage the target in dialogue in the foreign language on any subject! As you achieve objectives, they'll be automatically marked off your mission list.
• Witness dynamically generated images of the target, reflecting the topics they discuss, after each response.
• Enhance listening skills with automatically generated audio of the target's response.
• Translate the entire message into your native language for comprehension checks.
• Instantly translate any selected word within the conversation context, providing additional examples of its usage in the foreign language, which can be bookmarked for future review.
• Access hints for formulating questions about the objectives list to guide interactions with the target.
• Your messages are automatically checked for grammar and spelling, with explanations in your native language for correcting foreign language errors.

How we built it

With the time constraint of the hackathon, this project was built entirely on the frontend of a web application. The TogetherAI API was used for all text and image generation and the ElevenLabs API was used for audio generation. The OpenAI API was used for detecting spelling and grammar mistakes.

Challenges we ran into

The largest challenge of this project was building something that can work seamlessly in 812 different native-foreign language combinations. There was a lot of time spent on polishing the user experience to work with different sized text, word parsing, different punctuation characters, etc.

Even more challenging was the prompt engineering required to ensure the AI would speak in the language it is supposed to. The chat models frequently would revert to English if the prompt was in English, even if the prompt specified the model should respond in a different language. As a result, there are over 800 prompts used, as each one has to be translated into every language supported during build time.

There was also a lot of challenges in reducing the latency of the API responses to make for a pleasant user experience. After many rounds of performance optimizations, the app now effectively generates the text, audio, and images in perceived real time.

Accomplishments that I'm proud of

The biggest challenges also yielded the biggest accomplishments in my eyes. Building a chatbot that can be interacted with in any language and operates in real time by myself in the time limit was certainly no small task.

I'm also exceptionally proud of the fact that I honestly think it's fun to play. I've had many projects that get dumped on a dusty shelf once completed, but the fact that I fully intend to keep using this after the hackathon to improve my language skills makes me very happy.

What we learned

I had never used these APIs before beginning this hackathon, so there was quite a bit of documentation that I had to read to understand for how to correctly stream the text & audio generation.

What's next for SpyLingo

There are still more features that I'd like to add, like different types of missions for the user. I also think the image prompting can use some more work since I'm not as familiar with image generation.

I would like to productionize this project and setup a proper backend & database for it. Maybe I'll set up a stripe integration and make it available for the public too!

Built With

• elevenlabs
• openai
• react
• together-ai
• typescript