Miscue Lens

• 

  thumbnail

• 

  result

Inspiration

Teachers working with English language learners have to make this call constantly and informally: is a misread word an accent/language-transfer thing, or something else worth flagging, with no tool support and no time to think it through in the moment.

This isn't a Duolingo problem in disguise. Duolingo's speech recognition scores a learner's own pronunciation attempt against a target for practice, and the Duolingo English Test is a standardized proficiency exam. Neither one has a teacher listening to a specific student read a specific passage and getting an explanation of why a word was misread. Automating that explanation is closer to an open research problem than a shipped classroom product.

What it does

Miscue Lens listens to a student reading an English passage aloud, transcribes it, aligns the transcript against the original passage word-by-word, and classifies each misread word as either an interference_pattern (explainable by a known difference between English and the student's native language) or an inconsistent_pattern (doesn't fit a known interference explanation, worth a closer teacher look). It then generates a plain-language, teacher-facing summary and a concrete practice tip, translated into the student's native language when relevant. It is a descriptive observation tool, not a diagnostic one. This is enforced in code, not just in the prompt.

How we built it

Node/Express + TypeScript backend, single-file vanilla HTML/CSS/JS frontend, OpenAI SDK. Audio is transcribed with gpt-4o-transcribe, aligned against the source passage with a Levenshtein-style word diff, then each substitution is classified and explained by GPT-5.6 via Structured Outputs. The initial build (recording UI, passage picker, server scaffolding, transcription integration, alignment algorithm) was built iteratively in a Claude-assisted session. Codex was then used to execute the core product pivot: from a fixed two-population model (native English / Chinese-background) to a general framework that works for any native language. It rewrote the classification and explanation logic, added the language registry and API endpoint, and updated the frontend accordingly.

Challenges we ran into

Keeping the tool strictly descriptive (no risk score, probability, or IEP content) required enforcing the boundary in code (ensureDisclaimer()) as well as in the model instructions, since prompt instructions alone aren't a hard guarantee. We also scoped supported native languages to English and Chinese specifically, rather than claiming broader language support we couldn't personally verify for translation and linguistic-reasoning accuracy. We also learned that detection happens at the word level, not the pronunciation level: a genuine substitution to a different, real word survives into the transcript and gets classified, but a same-word pronunciation shift alone can get smoothed over by the transcription model, since ASR is trained to output fluent, expected text rather than a phonetic rendering of what was actually said.

Accomplishments that we're proud of

A working end-to-end pipeline: record, transcribe, align, classify, explain, bilingual toggle, with a hard-coded safety boundary that survives regardless of what the model returns.

What we learned

We learned that our early instinct to hardcode two native-language populations felt like the simpler path, but it boxed the whole design in. The rework showed that the real complexity lived in the language data model, not the UI; once languages.ts became data-driven, classification and explanation generalized almost for free. We also learned that a not-a-diagnosis boundary can't rely on prompt instructions alone: a model can drift under enough edge cases, so it had to be enforced in code too. And handing Codex one well-scoped, full-stack rework, spanning the backend model, the API endpoint, and the frontend wiring, worked far better than piecemeal file-by-file prompting.

What's next for Miscue Lens

Expand the native-language registry beyond English/Chinese once translation and linguistic-reasoning accuracy can be verified for more languages. Add a running per-student history so a teacher can see patterns across multiple readings, not just one. Add a supportive, encouraging moment for the student in the moment, not just the teacher-facing summary afterward. Today's pipeline only listens back after a full clip is recorded, so a live version would mean moving from single-clip batch analysis toward streaming audio, a real architectural step, not a small addition.

Built With

• codex
• css
• express.js
• gpt-4o-transcrive
• gpt-5.6
• html
• javascript
• node.js
• typescript