Hear See Do AI

• 

  It is all coming together one click at a time

• 

  infographic of what I am trying to build

• 

  based on the Hear See Do Method

• 

• 

• 

  admin panel

HSD OS AI Platform — Project Story

"You're not bad at English. You were just never taught to speak it."

What Inspired This Project

Japan produces more English test-takers than almost any nation on Earth. Walk into a high school exam hall and you will find students parsing complex grammatical structures, translating dense prose, and identifying the correct preposition in sentences constructed expressly to defeat them. These students have studied English for six years, often longer.

Step outside and ask one of them a question in English. The words don't come.

This is not a failure of intelligence or effort. It is the predictable outcome of a system built almost entirely around input — reading, grammar, translation — with almost no room for output. Speaking English in Japan is treated as a rare performance event: something you do in a special class, once a week, in sentences you've already rehearsed. The gap between comprehension and production is never closed because the system never asks students to close it.

Years spent around English education in Japan surfaced the same pattern again and again: learners who had passed every written exam froze the moment spontaneous speech was required — not because they couldn't form a sentence, but because the system had never given them room to practise imperfect English without consequence.

The question that launched HSD OS was not "how do we teach English? " It was: what if every Japanese learner had a patient, always-available speaking partner who never judges, never tires, and always shows up?** AI made that economically possible — a tireless, warm, non-judgmental speaking partner is no longer expensive or rare, it's a software problem. HSD OS was built to solve that software problem.

What We Learned

Voice is not optional. Jona's spoken replies run through a real ElevenLabs TTS pipeline, not a decorative add-on. When the AI sounded encouraging rather than merely outputting encouraging words, the interaction stopped feeling like a quiz and started feeling like a conversation.

Specificity in praise is not optional either. Generic encouragement ("Good job!") doesn't reinforce the behavior that produced it. Every coaching prompt is written to name a real strength before coaching what to improve — Jona is instructed never to say "wrong."

Kids and adults are not the same design problem. Monkey Yoga Phonics needs kinetic feedback and immediate reward signals; university and adult learners need to feel the stakes are real and the feedback carries authority. Building one platform that genuinely serves both, under one identity and subscription, forced real discipline about what each cohort actually needs.

Sustainability has to be a constraint from day one, and it's cheaper to verify than to assume. Every AI conversation has a real infrastructure cost behind it, so we treated per-session cost the way a hardware engineer treats battery life. Auditing our own real vendor receipts against the running platform (rather than estimating) surfaced something we didn't expect: total infrastructure spend for the hackathon period, sourced entirely from vendor payment confirmations, comes to

$$ \text{Total COGS} = \underbrace{287.98}{\text{Netlify}} + \underbrace{88.00}{\text{Replit}} + \underbrace{26.40}{\text{ElevenLabs}} + \underbrace{22.00}{\text{Anthropic}} + \underbrace{13.30}{\text{Firebase/GCP}} + \underbrace{12.00}{\text{GitHub}} = \$449.68 $$

with the Gemini API itself costing $0, still comfortably inside Google's free tier, confirmed directly against Google AI Studio's own usage dashboard rather than assumed.

How We Built It

HSD OS is a unified platform — one account, one subscription, many apps, one AI coach (Jona) present everywhere.

Technology stack

Layer	Technology	Notes
Frontend	React 18 + Vite	SPA deployed on Netlify
Auth + Data	Firebase	Auth + Firestore
AI Conversation	Gemini 2.5 Flash (Google)	Serverless Netlify Functions; Claude as automatic fallback
Voice	ElevenLabs	Text-to-speech synthesis
Payments	Stripe	Subscriptions + webhooks
Speech Input	Web Speech API	Native browser recording

Applications

• Kids — Monkey Yoga Phonics™ (phonics through movement, SSO sub-app), Eiken AI Monkey (adaptive EIKEN prep, Grades 5–Pre-1), Wondercamp (adventure storytelling)

• Family — HSD Family (shared dashboard, per-member profiles), Monkeys Unlock™: Monkey Escape (speaking-gated puzzle game)

• Adult — Speak & Sweat, Sip & Switch™, Sip, Speak,Learn, The Inner Key Blueprint™ (mindset and confidence work)

• University — Career Ready, Global Ready, Speak Ready (interview, study-abroad, and speaking-confidence practice)

The placement engine

EIKEN placement is a deliberately simple, explainable adaptive staircase, not a black-box model: a learner climbs a grade after two correct answers in a row, and eases back a grade after two misses in a row, drawing questions live from the real content banks at whatever level they're currently on. Speaking items have no "correct" answer, so client-timed hesitation does the same job a wrong answer does on multiple-choice items. Across listening, reading, and speaking, a learner reaches a placement in under ten questions — deterministic and auditable, not a probabilistic guess.

Proof it's actually running

Rather than asserting the AI is "live in production," we checked. Our own Firestore-logged usage record shows 12 real Gemini calls, 9,451 tokens, across 4 separate days, and — independently — Google's own AI Studio usage dashboard shows a matching request/token pattern over the same window, with a 100% success rate and zero errors. Two independently sourced datasets agreeing with each other is stronger evidence than either alone.

Challenges We Faced

Trust in an AI speaking partner. Japanese learners — adults especially — carry real anxiety about speaking English imperfectly. Every design decision, from character-led kids' apps to the "never say wrong" instruction to the warm voice, exists to lower that threshold. You cannot force someone to speak by improving the technology alone; you can only make the space feel safe enough that they choose to.

A real production bug, caught and fixed during this process. While gathering usage evidence for this submission, we found the admin dashboard's aggregate "AI Messages" counter was stuck at zero — not because no one was using Jona, but because a Firestore security rule was silently rejecting the counter's increment write (allow write: if false on the analytics collection, versus the create/update: if true carve-out used correctly elsewhere in the same rules file for other server-side counters). The chat itself worked fine for users the whole time; only the metric was broken. We traced it to root cause and shipped the fix live — a small but honest example of actively operating and debugging a system in production, not just launching it and walking away.

Subscription architecture for a multi-app world. One payment unlocking multiple apps, different plans unlocking different subsets, webhooks propagating reliably to Firestore, current and legacy plans coexisting without conflict — this took real iteration, and real users subscribe, cancel, and change plans in ways the first design never fully anticipated.

Speaking on mobile. The Web Speech API behaves inconsistently across browsers and operating systems. Getting a reliable speaking-response recording across iOS Safari, Android Chrome, and desktop remains one of the hardest, least-glamorous parts of the stack — and the part that most reminds us how far browser standards still have to go before speaking practice is as reliable as reading.

None of the hardest problems were the ones we expected going in. The technical architecture came together faster than the human insight about why a learner needs to feel safe before they'll speak — and even the financial and usage reporting for this submission surfaced a real bug worth fixing. That's still the core of how we build: verify, don't assume. The AI is just the tool we built to solve the original problem.

HSD OS AI Platform · app.hsdos.ai · Hear See Do™

Built With

• and-technologies-used**-javascript
• anthropic-claude-api-(claude-sonnet-4-6)
• apis
• claude-code
• cloud-firestore
• cloud-services
• databases
• elevenlabs-api
• firebase-authentication
• frameworks
• github
• google-ai-studio
• google-gemini-api-(gemini-2.5-flash)
• manus-ai
• netlify
• netlify-functions
• node.js
• platforms
• postmessage
• react-18
• react-router-v6
• replit
• stripe
• vite
• web-audio-api