Inspiration
Once an avid user of a renowned language-learning app, I thought I had my language skills well-honed. Armed with practiced phrases and vocabulary, I embarked on an exciting year-long adventure in France. The prospect of immersing myself in the language and culture thrilled me.
One day, I entered a French bakery, eager to try out my language skills in a real-world setting. However, as I tried to communicate my order, my carefully practiced phrases seemed insufficient. The French baker, speaking in rapid French, couldn't understand my attempts. With a polite smile, I exited the bakery, feeling defeated and disheartened.
The app I had relied on primarily focused on vocabulary and scripted dialogues, lacking the essence and unpredictability of real-life conversations.
What it does
Through engaging storylines, role-play AI assistants and progressive challenges, the platform allows one to test their conversational skills in GenAI-curated dialogues.
How we built it
Frontend (iOS App in Swift): Language: Swift (for iOS development) UI/UX Framework: UIKit for IOS17 Features: API access to backend, two way audio streaming, text translation and transcription, interactive ux with animations to capture attention
Backend (Google Cloud Compute Engine with Flask): Cloud Platform: Google Cloud Platform (GCP) Compute Engine: For hosting your backend server. Web Framework: Flask (Python) for creating API endpoints. API Endpoint Processing: Flask's route decorators for handling /connect, /start, and /storyline endpoints. Data Storage: Cloud Firestore to save user session data Authentication: Using session id and apple sign on
OpenAI Integration (for Storyline Generation): LLM Integration / Generative AI: OpenAI GPT-3.5 for generating storylines based on user input and json mode - prompt engineering Audio Processing: Using whisper-1 model to transcribe audio streamed from user to text and pass it to storyline engine for prompting Text-to-Speech: TTS-1 model to convert AI’s storyline response to user from text to speech - streamed to frontend from backend
Challenges we ran into
Integrating Frontend, Backend and OpenAI APIs
Accomplishments that we're proud of
The human-centric product we have laid the foundation for.
What we learned
New technologies as mentioned before and ways to integrate them; perform trial and error to see which tech works and which didn't
What's next for LangQuest
Customer Discovery and testing the market for the product