Encore

A virtual 3D audience on your Fire TV that laughs only when your timing earns it, with your phone as the mic and an AI MC that debriefs you from what it measured.

Comment

Inspiration

Anyone who performs rehearses alone, into silence: an open-mic comic, a slam poet, a best man with a toast, someone giving their first conference talk. A mirror can't laugh. A recording can't tell you where the room would have gone cold. The first time you hear a room react is the night it counts.

The TV is the biggest screen in the house, and it faces the spot where you'd stand. We wanted it to be the room: a crowd that leans in, laughs on the beat you leave, and goes quiet in dead air. We also wanted it to be honest. A crowd that laughs at everything is easy to build and useless for practice. So every reaction had to come from something the app actually measured, and the score had to be the same in a forgiving open mic as in a tough club.

What it does

Put your phone on the coffee table, pick an act, a venue and an MC on the TV with the remote, and press OK.

  • A virtual 3D audience that reacts to your timing. It reacts to what a live room reacts to first: the beat you leave after a line, a short punch after a longer setup, emphasis at the end of a line, a clean line without fillers. Every reaction carries its measured reason, and the TV's director captions it on screen, for example "LAUGH · 0.7 s after your line · cut to the floor". A flat line gets silence, and the scorecard counts it as a miss.
  • Your words, if you allow them. With words on, the room also counts the patterns a stage writer would name (a callback, a list of three, wordplay, a rhyme), and a content judge on Amazon Bedrock adds one labelled estimate per line. Content can only add to a line's score, and it is capped, so a line still needs its timing.
  • Four acts, four venues, one honest score. Stand-up gets laughs, poetry snaps, a speech applause, singing cheers. A tougher venue is a tougher crowd: a higher bar before it reacts, less patience with dead air, a heckle after fewer dead bits. The scorecard counts every line against one fixed house standard everywhere, so a kind room never flatters you and a hard one never punishes you.
  • An MC, not a chatbot. Choose warm Meera, sarcastic Vik, strict Mr. Rao or deadpan Dev. The MC introduces you, throws the cards between bits, debriefs you out loud from the measured numbers, and answers questions in the green room. A model on Amazon Bedrock writes every word, an honesty guard checks every number, and Amazon Polly speaks it. If the model is late or the guard rejects it, the MC stays silent and the screen says why. There are no canned lines.
  • The cut sheet. After the set you see it as an edit timeline: gold cuts where the room gave you something, striped tiles where the dead air ran past the room's patience, and the measured ledger beneath it.
  • Your audio never leaves your phone. The phone (a web page, no install, joined by QR code) sends only events: voice on and off, phrase timings and levels, pitch for singing, and words only if you switch them on.

About the demo video

The video was filmed on the Android TV emulator at 720p, which the hackathon FAQ allows; the short Vega clip is the Vega Virtual Device. The act is highlights of one scripted 3:39 stand-up set, recorded in one take and cut only between lines. No person speaks: a test script sends the set's words and timing through the same phone-to-relay path a phone uses, and the comic's voice is an AI voice (ElevenLabs, elevenlabs.io) placed inside each line's recorded window. Every laugh, silence, camera cut, caption, score and MC line was computed live by the app. No laugh was added, moved or boosted. The MC's voice is Amazon Polly, exactly as the app played it.

How we built it

One pure engine at the centre. packages/crowd is pure TypeScript with no clock, no randomness and no React, so every decision the room makes is reproducible and unit-tested. The phone streams events, the TV feeds them plus a tick ten times a second into one reducer, and the reducer decides the reactions, the camera cuts, the title cards, the heckles and the scorecard. A line's landing score adds up timing features; the room reacts when the score clears the venue's threshold, but the scorecard counts a line as landed only at one house standard (0.55) everywhere.

The TV apps (Fire OS and Vega OS). One shared React Native UI package renders the whole show for an Expo / react-native-tvos build (Fire OS / Android TV) and a React Native for Vega build (Vega SDK 0.24). It's a remote-first screen machine (bill → live → cut sheet ⇄ green room), readable from 3 m, and every state is reachable with OK, Back and the D-pad.

The 3D audience. A Three.js page runs in the TV's own WebView, fed ten times a second with the engine's state: attention, restlessness, warmth, each reaction's envelope, the heckle and the finale. The page adds only seeded per-person variation (who laughs first, how far a laugh travels), so the room is never louder or warmer than the engine measured.

  • Built for cheap TV hardware. We profiled on a 32-bit Google TV with a Mali-G31 GPU, the same GPU class as the Fire TV Stick Lite. Front rows are skinned Microsoft Rocketbox people with motion-captured idles and laughs; the rows behind are instanced and lit per vertex. A club frame is about 77k triangles in 44 draw calls, with textures under 30 MB per venue.
  • Bodies that don't break. An anatomy validator checks every avatar against its body volumes, joint limits, neighbours and furniture; our sweeps for limbs through bodies and chairs, hands and bind poses read zero. While making the video we caught claw-like resting hands and contorted poses in the bill's preview, traced both to the pose code, and fixed them.

The MC (services/mc-agent, Python). A Strands agent with structured output into the TV's contract, one fresh agent per request, for the debrief, a green-room reply, a moment line and the content judge.

  • The model: GLM-5 on Amazon Bedrock, through Bedrock's OpenAI-compatible endpoint with SigV4 signing.
  • The honesty guard: it rejects any answer that quotes a number the measurements can't produce, in digits or in words, hedges the wrong way ("just over fifteen" for 14.7), drops a required "not measured" or puts a clock time on a card. The model gets one retry; after that the MC is silent and the screen says "rejected".
  • Written for the ear: numbers in words with honest hedges, moments in words ("about three minutes in"), exact figures on screen.
  • The voice: Amazon Polly gives each persona its own voice (Kajal, Brian, Stephen, Matthew) on the generative engine, with neural as a fallback. The service returns a presigned Polly URL, signed locally in about a millisecond, and the TV streams it, so speech adds nothing to the model's budget.
  • AgentCore-ready: the MC is served by the AgentCore SDK's BedrockAgentCoreApp and packaged for AgentCore Runtime as a direct-code zip with a keyless forwarder. It runs the same code on EC2 because our account's AgentCore runtime quota is 0 (support case open).

The content judge, chosen by measurement. We set a rule before testing: the best rank agreement with human ratings among judges that answer within 500 ms (p90) on the real path, and never a fast one that is no better than chance.

  • On 1,000 held-out rated jokes (rJokes), the fast LLMs alone (Ministral 3 8B and 14B, Qwen3 32B, GLM-4.7 Flash) ranked jokes no better than chance (Spearman −0.02 to 0.05). But they told a punchline from the same idea said flat in 19½ of our 20 stand-up pairs.
  • A CPU humour model (RoBERTa-large fine-tuned on rJokes, MIT, int8 ONNX) ranks jokes (ρ 0.37) but can't tell a punch from a flat line (8 of 20).
  • What ships for stand-up is a 0.7/0.3 mix of the two: ρ 0.34 on the held-out jokes and 17 of 20 pairs, at p90 under 500 ms on the TV. The demo take ran the fast model alone, with the verdict on the TV in 183 ms (p50).

AWS end to end. Amazon Bedrock (GLM-5 for the MC, Ministral 3 8B for the judge), Strands Agents, the Bedrock AgentCore SDK, Amazon Polly, Amazon S3 (act packs as JSON and the Vega simulator's media), Amazon EC2 (build, render and emulator boxes), IAM (a box role limited to Bedrock and Polly), and AWS Budgets and CloudWatch for spend alarms.

Built with Claude Code. Parallel agents built the engine, the phone listener, the TV screens, the MC, the 3D audience and the Vega port, each against fixed TypeScript contracts and their tests, with heavy jobs on EC2. The Amazon Devices Builder Tools (MCP server and Vega skills) answered Vega questions and profiled the simulator.

Challenges we ran into

  • Honesty costs features. Every reaction needs a measured reason, and every number on screen must be one the phone can actually measure; with words off, pace and fillers read "not measured", never 0. We removed things that looked good but couldn't be justified, and we kept the misses in the demo.
  • A believable crowd on a TV-stick GPU. On the Mali-G31 Google TV the open mic, club and theatre ran at about 20–24 fps. Our target is a steady 30, and we're not there yet on the cheapest hardware. On the 720p Android TV emulator, our first demo take stuttered to 4–10 fps during the laughs, exactly when the whole crowd moves, because the build box was busy rendering video at the same time. Recorded on an idle box, the final take holds a 26 fps average and never drops below 18.
  • The Vega simulator has no WebView on Linux. The Vega build runs on the Vega Virtual Device (the bill, a live set, the cut sheet, the MC's voice), but the Linux simulator ships without the WebView our 3D room lives in. A native three.js attempt on @amazon-devices/expo-gl ran but had no depth buffer. So on Vega the room falls back to licensed video plates. Neither path has been tested on a Fire TV device yet.
  • Model access and AgentCore. On a new AWS account Bedrock's Converse API refused every model, so the MC runs open-weight GLM-5 on the OpenAI-compatible endpoint. Creating the AgentCore runtime fails on a zero quota; the case is open.
  • Latency you can feel. Our first voice sent base64 audio inside the answer, which added synthesis time to every reply and couldn't play on either TV player. Presigned Polly URLs fixed both. The content judge was once late on 23 of 37 lines; the cause was a scorecard function that grew quadratically with the set and stalled the TV's JS thread, and the fix took it to 0 late.

Accomplishments that we're proud of

  • A rehearsal room that reacts in real time without sending audio anywhere.
  • An MC whose every word is model-written, checked against the measurements and spoken aloud, and which stays silent rather than make something up.
  • A content judge we can defend with numbers, including the finding that fast LLMs alone were no better than chance at ranking jokes.
  • A 3D audience in the TV's own WebView, built and measured for the cheapest class of TV hardware, with its limits published.
  • One shared UI on Fire OS and Vega OS, with the simulator's gaps documented in a friction log of 28+ entries, each with steps, a workaround and a suggestion.

What we learned

A room's reaction is mostly timing. The beat you leave after a line matters more than the line itself, and that's exactly what a microphone can measure honestly. Judging whether a line is funny is the hard part: a small, fast LLM that sounds confident was no better than a coin flip at ranking jokes, and we only found that out because we measured it. On a TV, every millisecond of JavaScript per frame is shared with the GPU, so profile on the weakest device first.

What's next

  • A steady 30 fps on a Fire TV Stick Lite-class GPU, then testing the WebView room on a Fire TV device.
  • Shipping on the Fire TV Appstore; the act packs are already JSON on S3, so new acts and venues can arrive without an app update.
  • Moving the MC onto AgentCore Runtime once the quota opens; the zip, the role and the forwarder are ready.
  • A more expressive room (shouts, whistles, a groan on a dead bit in a tough room), more acts (improv, karaoke), and household play: several phones on one couch as the heckling front row.

Built With

Share this project:

Updates

Submission history