Project Rosetta - A Fire TV app that levels up the family.

A Fire TV app where homework, chores and daily habits earn TV time and real pocket money, checked by an AI tutor on AWS.

Comment

Inspiration

Every evening, in millions of homes, the same two questions are asked in front of the same screen: "Did you finish your homework?" and "Can I watch TV now?" Parents nag, kids bargain, and the TV, the biggest shared screen in the home, is only ever the prize that gets switched off.

We asked a different question: what if the TV itself could help? What if it could translate what a parent asks for ("do your homework", "tidy your room", "read 10 pages") into something a child can see, do and feel proud of, and then reward it fairly? That is the Rosetta Stone idea behind the name: a translator between parents and kids.

What it does

Rosetta is a native Fire TV app for the whole family, with an AI agent crew running serverless on AWS. Parents set the day from their phone, kids check in from theirs, and the TV shows everyone how the day is going. TV time unlocks only when today's must-do tasks are done.

  • A living home, full of quests. Home is a 3D-feeling dollhouse; the camera glides to each child's room as you browse homework, chores and daily habits. A TV Time portal shows "6 must-do to go", then bursts open when the work is done or a parent unlocks it from their phone.
  • ✦ AI Tutor for homework. A child photographs a worksheet with any phone. Seconds later the TV shows every answer marked on the child's own handwriting, a score ring, and a hint for each mistake: never the answer. Arithmetic verdicts are double-checked with exact maths in code (the model reads, the code decides). It works for coding class too: the Tutor traces little programs line by line.
  • Practice that makes them think. A short practice round on exactly the skills that slipped, written as easy reasoning questions for the child's grade, not "1 + 1".
  • Honest photo checks. Send the wrong photo and Rosetta says what it actually sees ("That's a lovely photo of your room, but I need your Fractions worksheet"). Tidy-ups turn each out-of-place item into a collectible Lost Thing, and the after photo is checked again: "5 of 6 back home".
  • Daily habits with voice notes. Make your bed, read 10 pages, prayer time: kids check in with a tick, a note, a photo or a voice note. The AI listens, summarises it and keeps it as a memory parents can replay. Every habit has its own streak.
  • Rewards for every child, and real pocket money. XP, coins, stamps and a star for the Family Goal (nine stars unlock Movie Night). Parents decide what each task pays in real pocket money; kids save toward a wish list, ask from their phone when they have enough, a parent taps yes, and the TV updates the balance live.
  • Family Quiz Night in 3D. A co-op quiz in a Unity 6 scene, with reasoning questions written for each child's class, subjects and upcoming tests: fractions for one sibling, code tracing for the other.
  • Progress and ✦ Coach's notes. Per-child dashboards (habit calendar, homework accuracy, pocket money, stars) and notes the AI writes from real activity only. No sample data anywhere.
  • Parents in control, from any phone. No app to install: scan the QR on the TV. Push, email or SMS alerts when something needs them; one-tap approvals; daily tasks, learning profiles, pocket money, wish lists and TV time.

How we built it

Fire TV app. Kotlin + Jetpack Compose for TV on Fire OS, D-pad first, designed on a 1440 × 810 canvas scaled to the panel so 720p, 1080p and 4K Fire TVs look identical. Real-time 3D (Filament via SceneView) for the coins, chest, padlock, trophy and stars; a Unity 6 companion APK for Quiz Night, launched with the quiz as an intent extra; a looping animated home scene played with Media3.

Serverless on AWS, end to end (one AWS CDK stack):

Layer AWS service What it does in Rosetta
Edge Amazon CloudFront + Amazon S3 Serves the kid and parent phone pages (PWA, Web Push)
API Amazon API Gateway (HTTP API) One API for the TV and the phones
Compute AWS Lambda The API, the agent crew and scheduled jobs
Agents Strands Agents SDK Quest Master agent decides which specialist to run for each photo or voice note and handles edge cases
Models Amazon Bedrock (Converse API, Amazon Nova 2 Lite) Every AI call: marking, practice, quiz, room checks, notes. Each prompt is a Bedrock Converse tool spec; light reasoning only for marking, code tracing and before/after checks
Voice Amazon Polly · Amazon Transcribe The Keeper's voice on the TV (Polly generative voice); kids' voice notes transcribed, then summarised by Nova
State Amazon DynamoDB One household document: kids, quests, habits, rewards, wishes, history
Media Amazon S3 Presigned uploads straight from the phone → S3 event → agent Lambda; photos deleted after 1 day, memories kept 90 days
Schedules Amazon EventBridge Daily streak nudges and the weekly family digest
Alerts Amazon SES · Amazon SNS Email and SMS for parents (plus Web Push)
Delivery AWS CDK cdk deploy creates everything; pay-per-use, scales to zero

The agent crew. AI Tutor (marking), Practice, Quizmaster, Scout (finds out-of-place objects with bounding boxes), Judge (before/after), Listener (voice notes), Proof check, Keeper (the friendly voice) and Coach (notes). Each is a single multimodal call with a forced JSON schema that must also say what it actually saw or heard, which is what makes the honest replies possible.

Trust layer. mathcheck.py computes exact answers with Python fractions and overrides the model's verdict on arithmetic; it also recognises classic slips ("you added the bottom numbers too") for precise hints. On our test sheets the model alone agreed with the true marking on 10 of 11 answers; with the check, 11 of 11.

Challenges we ran into

  • AI that marks a child's homework has to be right. Models read handwriting very well but can slip on "which is bigger, 2/3 or 3/5?". So the code decides arithmetic, and the model does what it's great at.
  • Honesty over magic. Early versions answered a wrong photo with a generic "try again". Every check now returns what it saw, and every number on screen comes from the server's history.
  • Fire TV can't let an app lock other apps. Earned TV time is honoured inside Rosetta (it opens Prime Video or YouTube and counts the minutes) and parents use Fire TV's own parental controls for the rest. This is our top feature request.
  • A real Fire TV is 960 × 540 dp. Screens that looked right on a phone image at TV resolution overflowed on a true TV image; scaling one fixed canvas fixed every screen at once.
  • No Fire TV hardware and no Vega SDK on Windows, so we built for Fire OS 16 on the Android TV emulator and tested on a 4K Android TV.
  • Packaging Strands for Lambda from Windows silently installed an old version until we built the bundle with uv for the Lambda platform.

Accomplishments that we're proud of

  • A live pipeline: phone photo → S3 → Strands agent on Lambda → marked homework and practice on the TV in about fifteen seconds.
  • An AI tutor that never gives the answer away, with maths it can't get wrong.
  • A family loop that is real end to end: a parent adds a task, a child sends a voice note, the AI checks it, pocket money lands, the wish list fills, a parent approves, and TV time unlocks.
  • A co-op family quiz in a real game engine, built from what the kids studied that week.
  • 45 offline tests covering the API, habits, voice notes, TV time, wish list, agent policy and exact maths.

What we learned

  • Multimodal models are excellent readers of handwriting; pairing them with small exact checks makes them trustworthy enough for children.
  • The TV is the right place for the shared family moment, and the phone is the right camera.
  • Keep the agent layer model-agnostic: our prompts and schemas are Bedrock Converse tool specs, so the crew isn't tied to one model.

What's next

  • An Amazon Kids "earn time" API, so earned TV time is enforced, not honoured.
  • Alexa voice answers during Quiz Night, reading-aloud fluency checks, more subjects, and a Vega OS build.

Built With

Share this project:

Updates

Submission history