-
-
Street Talk English starts with practical scenarios that build confidence for real-life conversations.
-
Learners choose a coffee-shop roleplay or turn a real-world photo into a personalised mission.
-
A clear mission briefing shows the learner’s role, partner, objectives, and choice of voice or text practice.
-
During roleplay, an in-character barista responds naturally while Street Talk tracks objectives live.
-
After each mission, learners see completed objectives, practice needs, and progress saved for next time.
-
Upload a real-world image and context note to create a personalised mission based on a situation that matters.
About the project
Inspiration
Speaking English is not only about knowing vocabulary or grammar. Many learners understand English well enough, but still freeze when they need to speak to a barista, doctor, receptionist, interviewer, customer, or stranger in real life.
Street Talk English was inspired by that gap between learning English and actually using it. Not everyone feels comfortable practising aloud in front of a teacher or other people. I wanted to create a low-pressure space where learners can rehearse everyday conversations privately, make mistakes safely, and build confidence one mission at a time.
The idea evolved beyond a fixed coffee-shop dialogue. Real life is unpredictable, so learners can also upload an image such as a menu, sign, booking, or workplace situation and turn it into a personalised speaking mission.
What it does
Street Talk English is an AI-powered English roleplay coach for realistic, voice-first conversations.
Learners can:
- Practise a guided coffee-shop roleplay as a customer.
- Upload a real-world image and a short context note to generate a personalised mission.
- Reply by voice or text.
- Receive natural in-character responses instead of generic assistant answers.
- Track mission objectives as they complete them.
- Review completed objectives and identify what needs more practice.
- Generate a short private video recap after completing an eligible real-world-image mission. The recap helps learners revisit the situation, useful language, and the confidence they built during the roleplay.
- Create a private memory song after giving explicit consent. The song uses grounded lyrics based on verified completed objectives, turning useful phrases and successful talking points into something memorable.
- Use the video and song as recall tools before facing the real conversation. For example, a learner preparing to discuss adopting a kitten can remember the song’s chorus to recall their responsibilities, financial plan, and compromise.
The goal is not to replace human conversation. It is to help learners feel prepared enough to have it.
How I built it
The project uses a FastAPI backend with Google Agent Development Kit and Gemini through Vertex AI. The AI agent is instructed to remain in role, keep replies concise and learner-friendly, resist prompt-injection attempts, and guide the conversation without answering objectives for the learner.
The frontend provides a responsive roleplay experience with:
- Text and voice reply options.
- Image-based mission generation.
- Live objective progress.
- Evidence-based evaluation of learner responses.
- Mission results and personalised practice feedback.
- A reviewer access flow for the hackathon demo.
The application is deployed as a containerised service on Google Cloud Run, providing a scalable, managed environment for the React front end and FastAPI backend. Cloud Run handles incoming learner requests and runs the mission, roleplay, and objective-evaluation workflows.
Firestore provides persistent learner and mission state. It stores completed objectives, progress history, and practice needs so each new mission can adapt to what the learner has already achieved and where they still need support.
Gemini on Vertex AI powers the core learning experience. It analyses a learner’s uploaded real-world image and short context note to create a personalised mission with a clear goal, useful phrases, a conversation partner, and measurable objectives. During the roleplay, Gemini produces natural in-character replies, responds constructively to imperfect English, and helps evaluate whether the learner has completed the practical communication goals.
Google Cloud Speech-to-Text and Text-to-Speech support voice-based practice. For eligible completed missions, Vertex AI Veo can create a private visual recap, while Lyria can create a private memory song from verified mission evidence and explicit learner consent.
What I learned
This project taught me that a believable language coach needs more than a chat prompt. The experience depends on clear boundaries between the AI’s role, the learner’s objectives, session state, evaluation logic, and the user interface.
I also learned that objective tracking must be evidence-based. A model can sound confident while incorrectly assuming a learner has completed an objective. The evaluation flow therefore checks what the learner actually said, connects it to a specific objective, and avoids awarding progress without enough evidence.
On the product side, I learned how important the interface is for reducing speaking anxiety. Clear objectives, short replies, voice and text choice, visible progress, and a friendly visual style all make the experience feel more approachable.
Challenges
The hardest challenge was making personalised roleplays useful without making them too open-ended or unreliable.
Image-generated scenarios can vary widely, from a restaurant menu to a job interview document or a road sign. The system needs to create a safe, coherent roleplay with realistic roles, achievable objectives, and clear language for learners. It also has to avoid exposing internal instructions, handle imperfect English gracefully, and prevent the AI from drifting out of character.
Another challenge was balancing demo limits with a satisfying learning flow. Learners need enough turns to complete their objectives, but the system also needs predictable resource use. The experience now handles mission completion and incomplete objectives more gracefully instead of simply ending with a usage-limit error.
Street Talk English is an early but working step toward a larger vision: helping people practise the conversations that matter before they have to face them in the real world.
Log in or sign up for Devpost to join the conversation.