Inspiration

As a language learner myself, I find it tough to find lessons tailored to situations I may encounter, and instead of spending hours searching online for relevant content or copying AI responses, I wanted to create a cozy, interactive learning environment where you create the content you want to learn and can find content that other people create. So I created SceneSpeak to help with that!

What it does

  • Create a lesson from a sentence. Type a situation in and SceneSpeak checks the request privately on your device, then writes a short role-play (goals, lines, translations) and builds a 3D scene with an AI image backdrop for it from a catalog of rooms, props and characters.
  • Play it in 3D. A chibi character speaks in a speech bubble over a low-poly set. You reply by typing, or switch to Speaking mode and push to talk.
  • Instant, private feedback. Every typed reply is graded by Gemini Nano on your device: a score, a friendlier way to say it, and a hint. Exact answers are graded instantly without a model call at all.
  • Grammar breakdown. Tap 🧩 on any line to see each word's role and how it compares with English, also device-first.
  • Share, upvote, report. Lessons you create can be shared to the library. Others can play and upvote them, and three reports hide a lesson until it's reviewed.
  • Progress without tracking. Stars, points, streaks and phrases to review are stored only in your browser. A weekly recap from an on-device coach reads your stats without them ever leaving the device. An opt-in weekly board shares only a first name and days practised.
  • See where the AI runs. The Model Monitor lists every AI call: on-device or cloud, latency, routing reason, and how many cloud calls were saved.

How we built it

Hybrid inference everywhere, through Firebase AI Logic. Every model call goes through firebase/ai with hybrid inference. A small routing policy picks the mode for each call from ordered rules: privacy, capability, language support, connectivity, input size and the state of the on-device model. Each decision carries a human-readable reason that shows up in the Model Monitor.

Job Where Why
Grade typed replies Device first, cloud fallback Every reply: should be private, instant and free
Grammar breakdown Device first Short, frequent, personal
Check a lesson request (safety, topic, skills) Device first The request may be personal
"Is there already a lesson for this?" Device first Cheap judgement that can save a cloud write
Weekly coach recap Device only Practice stats never leave the browser
Write a new lesson (scenario + scene recipe) Cloud Long, creative structured output
Grade a voice reply (audio in) Cloud, opt-in The device model doesn't take audio; recordings are never stored
Paint the scene backdrop Cloud [only if shipped] Image generation

The lesson pipeline is built as small, testable steps:

  1. Remove emails, phone numbers and long numbers from the request, before any model sees it.
  2. Gemini Nano screens it for family-friendliness and extracts a topic and skills. A flaky device check falls back gracefully instead of blocking; an unsafe request stops before any cloud call.
  3. A word-overlap shortlist plus a Nano judgement looks for an existing library lesson.
  4. Cloud Gemini writes the scenario and a scene recipe using structured output. The JSON schema restricts every room, character and prop to IDs from our asset catalog, so the model never invents geometry, and it requires a full-length lesson.

Challenges we ran into

  • A small model's summary can drift. Gemini Nano once summarised "how to exchange money at the bank" as asking for directions to the bank, and the cloud writer faithfully wrote a directions lesson. The fix: the cloud writer now always gets the learner's own (redacted, escaped) words, and the device summary is only a hint.
  • Structured output has limits. Requiring 6–12 lines in the JSON schema made Gemini reject every request ("too many states for serving"). A minimum works; a maximum on complex objects doesn't. We cap in Zod instead.
  • On-device calls ignored the system prompt. The SDK only sends systemInstruction to the cloud, so Nano was grading without its rubric. We now pass it as an initial system prompt on device too.
  • Lenient grading. "Hola." passed a "Hola, buenos días" turn with 90/100. Short replies now still pass if they reach the goal, but can't score as "natural and complete".
  • Shared content in a family app. Votes and reports are enforced in security rules (one per person, counts can only move by one, three reports hide a lesson), with the rules covered by emulator tests.

Accomplishments that we're proud of

  • The hybrid split is real and visible: the Model Monitor shows calls landing on device and the cloud calls saved.
  • Generated lessons are safe by construction: catalog IDs only, schema-enforced, validated, repaired once, never crashing the scene.
  • Privacy as a feature, not a footnote: redaction before any model, device-only coach, recordings never stored, plain-language Privacy page with a one-click "delete my data".
  • It's genuinely cute to play.

What we learned

  • Treat model output like user input: parse, validate, clamp, and fall back.
  • Routing is a product decision as much as a technical one. Writing each rule's reason down made trade-offs (privacy vs quality vs cost) explicit.
  • Small models are great judges and screeners when the decision is narrow and the options are constrained.
  • [Eval lesson: fill in from the eval report, e.g. on-device vs cloud grading agreement and latency.]

What's next for SceneSpeak

  • More languages (Nano's on-device support currently covers English, Spanish and Japanese).
  • More Quality
  • A moderator dashboard for reported lessons.
  • Real-time spoken conversation for off-script moments.
  • More 3D interactive scenes/ lessons

Built With

  • chrome-built-in-ai
  • cloud-firestore
  • firebase-ai-logic
  • firebase-app-check
  • firebase-auth
  • firebase-hosting
  • gemini
  • gemini-nano
  • github-actions
  • react
  • three.js
  • typescript
  • vite
  • vitest
  • zod
Share this project:

Updates

Submission history