SnapLearn

logo v0
mascot v0
concept_2
concept_1

Inspiration

My daughter is two and a half years old. She already grabs my phone and demands to watch YouTube. Every time I hand it over, I feel a pang of guilt — she’s absorbing passive, low-value content when she should be exploring the world around her.

I’m not alone. Smartphones have become ubiquitous in children’s lives. Most kids use them to consume brainrot content on TikTok and YouTube — hours of passive scrolling that does nothing for their development. In Vietnam, the stakes are even higher: parents spend $50–200 per month on English tutoring, yet between classes, their children have zero engaging ways to practice. The result? Vietnam ranks 58th globally in English proficiency despite massive education spending.

We’re a team of four AI engineers who build enterprise AI systems professionally. For this hackathon, we wanted to challenge ourselves with a fundamentally different problem: building for a user who is six years old, can’t read, and needs an experience that is purely intuitive. We wanted to turn the device parents feel guilty about — the smartphone — into something they feel proud to hand over.

What it does

SnapLearn is Pokémon Go for vocabulary. A child points their phone camera at any real-world object — a water bottle, a cat, a bicycle — snaps a photo, and the AI identifies it in both Vietnamese and English. The app plays the pronunciation aloud, then challenges the child to say the word back. If the speech recognition understands them, they earn points and collect the word as a trophy card in their vault.

Every object has a rarity tier — Common items like chairs are worth 10 points, while Legendary finds like a calligraphy brush are worth 250. This motivates kids to explore beyond their living room: go to the market, the park, a museum. Rare words are out there waiting.

The entire experience is audio-first and icon-driven. A child who cannot read can use the app with ears and taps alone. Every text element is spoken aloud. Navigation uses icons, not labels. Error messages are delivered by a friendly mascot voice, never a text popup.

How we built it

Frontend: Next.js 14 PWA with Tailwind CSS. Camera access via browser MediaDevices API. Push-to-talk audio recording via MediaRecorder API.
Backend: FastAPI (Python) for AI orchestration with async processing across multiple AI services.
AI Pipeline: We use BAML (Boundary AI Markup Language) to define structured LLM functions with type-safe outputs and automatic fallback chains. Our vision pipeline falls through GPT → Gemini → Claude, ensuring our AI workflows are reliable and model-agnostic.
Pronunciation Scoring (ASR + LLM-as-Judge): A two-stage approach. First, OpenAI ASR transcribes the child's speech. Then an LLM-as-judge evaluates the transcription against the target word — handling accent variations, partial matches, and childish mispronunciations far better than rigid string matching. Low latency because ASR is fast and the judge call is lightweight. Our principle: if the speech model understands you, a native speaker would too.
Duplicate Detection (Hybrid Semantic-Keyword Search): When a child captures a new object, we check against their existing collection using the same RAG technique found in enterprise search — keyword matching for exact duplicates plus semantic similarity (OpenAI embeddings + ChromaDB) for near-duplicates like "thermos" ≈ "water bottle." Reciprocal Rank Fusion combines both signals.
TTS: ElevenLabs for natural-sounding Vietnamese + English pronunciation playback.
Database: Supabase (PostgreSQL) for user profiles, vocabulary collections, and leaderboard rankings.

Challenges we ran into

We're four engineers who build enterprise AI systems for a living. This hackathon reminded us that consumer products - especially for children - are a completely different game.

Time pressure and coordination. We worked in shifts — someone always coding, someone always sleeping. Handoffs were rough. We learned to communicate through docs and code rather than conversations.

Model decisions by vibes, not experiments. In a perfect world, we'd A/B test every model choice. With 36 hours, we went with prior experience and gut feel. GPT for vision because it's been reliable. ElevenLabs because we knew it had high-quality Vietnamese voices. OpenAI ASR because it is fast and can handle accent well. Some choices we don't regret, others we switch mid-demo. The lesson: shipped > optimal.

Old habits die hard. We burned hours building for a full authentication and onboarding flow. Muscle memory from enterprise projects. Then we realized: hackathon judges don't care about login. They care about the core loop. We ripped it out. Replaced it with non-intrusive dummy.

The discipline to not over-engineer. We had to fight ourselves. Vision embeddings for semantic retrieval? Tempting but unnecessary. Multi-language support from day one? The backend already returns trilingual data. But every "nice to have" steals time from the core loop. We had to self-discipline: if it doesn't show up in the 3-minute demo, it doesn't get built. Chinese language? Cut. Missions system? Cut. Parent dashboard? Cut. The discipline to say no was harder than the coding.

Sleep deprivation makes teams grumpy. 3/4 way in, running on coffee, bug-related humors, and AI-blaming, we also got snappy with each other. Small disagreements felt bigger. We learned to call timeouts — take 10 minutes, relax, then back at it. AI turns out to be an easy scapegoat for bug blaming.

Accomplishments that we're proud of

The core workflow works end-to-end. Snap a photo of any real-world object. The AI identifies it in Vietnamese and English. The card reveals with a rarity glow. TTS plays both pronunciations. The child says it back. ASR transcribes, the LLM-as-judge scores, points are awarded, the card is collected. Every step is real AI — not mocked, not hard-coded. A six-year-old could use this with ears and taps alone.

BAML orchestration with zero glue code. Our vision pipeline falls through GPT → Gemini → Claude automatically. BAML handles the failover chain declaratively with type-safe structured outputs. No if/else spaghetti, no retry logic scattered across the codebase. When one model is down, the next picks up seamlessly.

ASR + LLM-as-judge for pronunciation scoring. Instead of rigid string matching, we use a two-stage approach. OpenAI ASR transcribes the child's speech. Then an LLM evaluates the transcription against the target word — handling accent variations, partial matches, and childish mispronunciations gracefully. The principle: if the speech model understands you, a native speaker would too.

Hybrid semantic-keyword search for duplicate detection. When a child captures a new object, we check against their existing collection using both keyword matching and semantic similarity (OpenAI embeddings + ChromaDB). So "thermos" is recognized as similar to "water bottle" — the same RAG technique used in enterprise search, applied to a kids' app.

Audio-first UX is genuinely possible. Every text is spoken. Navigation uses icons, not labels. Error messages come from a friendly mascot voice, never a text popup. We proved that a pre-literate child can navigate an app with ears and taps alone.

The moment the TTS sounded authentic. When ElevenLabs returned Vietnamese pronunciation that didn't sound robotic - when it sounded warm, natural, something a child would actually want to imitate. That's when we knew this could feel magical, not mechanical.

36 hours, 4 engineers, 1 dad. We built this in a weekend because one dad wanted to hand his daughter a phone without guilt. Every technical decision was filtered through that lens. And we proved that enterprise AI engineers can build for consumers — even when the consumer is six years old and can't read.

What we learned

Children are the hardest users we've ever designed for. You can't rely on text, tooltips, or documentation. Every interaction must be intuitive. If a six-year-old can't intuitively figure it out in three seconds, it's no good.

Audio-first feels cool even for adults. We built this for pre-literate children, but found ourselves using the app without reading anything. Just listening, tapping, speaking. It's faster. It's more delightful. Accessibility made the product better for everyone.

The "shared visual context" insight. When a child snaps a photo, they don't think they're "prompting an AI." They think it's a game step/mechanism. But the camera snap IS the prompt — and because the AI now sees the same object the child sees, they share a visual context. This enables conversational AI: the AI sees the same cat your child does and asks "Is that your cat? What's its name?" That's a platform, not just a vocabulary app.

Gamification for learning is intuitive when the game format matches the learning mode. Pokémon Go proved that exploration + collection + rarity + competition = obsession. We channeled that exact psychology into vocabulary. Kids don't feel like they're studying. They feel like they're playing. But the learning is real — they're speaking words aloud, building a collection, competing with friends. The game isn't a wrapper around the learning. The game IS the learning.

Personal motivation matters. Building for my daughter gave us clarity on every decision. "Would a 6-year-old understand this?" "Would she find this delightful?" That question cut through debates faster than any framework. Projects with personal stakes feel different. You care more. You ship better.

Ruthless scoping is a superpower. We cut Chinese language, missions, parent dashboard, spaced repetition, AI storytelling. Every cut hurt. But each one protected the core loop. The discipline to say no — to good ideas, to easy ideas, to ideas we already built code for — was the difference between a finished demo and a graveyard of half-features.

What's next for SnapLearn

The vocabulary camera is just the starting point. The platform expands naturally along four axes:

Content depth: The child's captured words become sentences ("The ___ is sleeping"), then personalized stories the AI generates and reads aloud ("One sunny day, a cat climbed a tree and watched a bicycle go by..."). Every word they capture enriches their story world.

Activities: Beyond capture and pronunciation, we add spaced repetition (resurface weak words at increasing intervals), category completion badges ("Animals Master!"), and themed scavenger hunts ("Find 5 things in the kitchen").

Distribution: Today it's parent-to-child (B2C). Next, classroom mode — a teacher creates a class, assigns missions, sees per-student progress on a dashboard. One teacher equals thirty kids. This is the B2B play that scales.

Interaction model: The camera creates shared visual context between child and AI. Today the AI names what the child sees. Tomorrow it starts a conversation: the AI sees the same cat your child does and asks "Is that your cat? What's its name?" That's a conversational AI interaction triggered by a physical moment, not a text prompt. The AI becomes a companion, not a tool.

Chinese language is weeks away — the backend already returns trilingual data (English, Vietnamese, Mandarin with pinyin). The same core loop works for any language, any subject, any age.

We started this project because one dad wanted to hand his daughter a phone without guilt. We're continuing because every parent deserves that feeling.