Megastructures TV

Interactive engineering films for Fire TV. Ask the remote "what if?" and the AI changes the live simulation in front of your family.

Comment

Is this a new project?

Existing, significantly updated during the submission period.

What existed before: three interactive physics scenes as separate desktop browser pages with mouse controls: a tunnel boring machine boring under a street, Starship above the clouds, and a tsunami modelled on Tohoku 2011, plus their shared Three.js material and post processing code.

What is new, built during the hackathon:

  1. A Fire OS app: an Android Studio project in Kotlin with the Leanback launcher, a full screen WebView with WebGL, a remote key bridge, Android SpeechRecognizer and TextToSpeech.
  2. A 10 foot D-pad UI: a home row, a spatial focus engine, chapters on Left and Right, a settings drawer that reaches every control with five buttons, a 5% safe area and no text under 28 px.
  3. Voice to simulation: an Ask overlay plus an answer service that sends the question and the live on screen state to Claude on Amazon Bedrock with three tools (jumpToChapter, setParam, explain). The tool calls are checked against the real ranges and then animate the live simulation. The answer is spoken and captioned.
  4. A TV performance mode: dynamic resolution, no MSAA, smaller shadow maps, a fixed light budget, lower ray march steps and terrain detail, and a frame rate overlay. Measured at 1.6x to 3.2x the frame rate of desktop quality.
  5. An open sourced engine: megastructures-engine (MIT), split out of the scenes, with a TV input kit, docs and an example.
  6. The AWS path as code: a Lambda handler, an AWS SAM template (API Gateway, Lambda, S3, IAM) and an Amazon Polly adapter. Written, not deployed.

The video shows each of these, and ends with a "Built during the hackathon" card.

Inspiration

I make engineering simulations: a tunnel boring machine you can cut in half, Starship you can take apart, a tsunami you can try to stop. People watched them hundreds of thousands of times, almost always alone on a phone. The questions in the replies were always the same kind: what if it went faster, what if the wall were taller, what is inside. Those are living room questions. A family watching together on the big screen asks them out loud. So the TV should be able to answer by showing, not by reading out a paragraph.

What it does

Megastructures TV is a Fire TV app of interactive engineering films. Pick a machine from the home row. Each film plays like a documentary, and Left and Right on the D-pad move between chapters. Press Menu on the remote and ask anything: "What happens if the screw turns faster?" Claude, running on Amazon Bedrock, jumps to the street above the tunnel and speeds up the screw conveyor. The chamber pressure drops and the street really sinks, with houses cracking, while the TV explains why out loud. Ask the tsunami "Can a wall stop it?" and the sea wall rises to 45 m and the city stays dry. Ask Starship "When does the top part come off?" and it plays hot staging. Every setting is also in a drawer you drive with five buttons.

How I built it

  • Scenes: Three.js and TypeScript, built with Vite. One page per film, so only one WebGL context is alive at a time on a TV stick with 1 to 2 GB of memory.
  • 10 foot shell: a small TV layer around each scene. A spatial focus engine moves between anything marked data-focus. A remote module turns keyboard keys, Fire TV web key codes and native bridge calls into one key stream. The shell has a settings drawer, an Ask overlay and an answer card. Everything is sized in 1080p units, so it looks the same on a 720p, 1080p or 4K panel.
  • Voice loop: the page sends the question with a snapshot of the screen: the film, the current chapter, every control with its live range and value, and the readings. A Node service calls the Bedrock Converse API with three tools and toolChoice: any. Claude returns all its tool calls in one turn. The server drops anything that does not exist and clamps numbers into range. The page applies the chapter jump first, then animates each slider from the value the viewer last saw. The answer is spoken (speechSynthesis in the page, TextToSpeech in the Android shell, or Amazon Polly through an adapter) and captioned with the tool calls.
  • Fire OS shell: Kotlin, a full screen WebView with the web build bundled in assets and served through WebViewAssetLoader. D-pad and Select reach the page as DOM key events. Back, Menu and the media keys go through window.MegaTV.key(). Voice goes through SpeechRecognizer and window.MegaTV.onVoice().
  • TV performance mode: a resolution governor aiming for 30 fps, MSAA off, shadow maps down to 1024, a fixed light count per light kind, half the ray march steps for clouds and engine plumes, and a quarter of the triangles for the tsunami terrain.
  • AWS: Bedrock is used for real. The same request handler is wrapped for AWS Lambda behind API Gateway, with a SAM template that adds S3 and IAM for Bedrock and Polly.

Challenges I ran into

  • The microphone button belongs to Alexa. Amazon documents that the voice button on a Fire TV remote cannot be intercepted by apps. My whole idea was "press the mic and ask". I made Menu the Ask button, forwarded it from the native shell, and wrote it on screen so nobody has to guess.
  • Speech to text on Fire OS is not guaranteed. I could not find whether third party apps get a speech recogniser. So the Ask overlay always offers three suggested questions you can pick with the D-pad, plus a typed field.
  • A phone GPU driving a TV. The scenes were tuned for laptops. Starship went from 22 to 70 fps on my machine with TV mode, mostly from dropping MSAA and halving the ray march steps.
  • No hardware. I have no Fire TV device and no Android toolchain on the build machine, so the Android project is written carefully but untested. Everything else is tested in a browser with the same keys a remote sends.
  • Making an AI change a simulation safely. The model sometimes asks for values outside a range or names a chapter by its title. Sending the live ranges with every question, and validating every tool call on the server, made the changes reliable.

Accomplishments that I'm proud of

  • The AI does not chat about the film. It changes the film. You see the street sink because of your question.
  • Five of five recorded questions came back from Bedrock as valid tool calls in one round trip, in 3.1 to 3.6 s.
  • Every action in the app is reachable with Up, Down, Left, Right and Select.
  • TV mode measured at 1.6x (tunnel boring machine), 3.2x (Starship) and 1.6x (tsunami) the frame rate at 1080p.

What I learned

  • On a TV, the input model is the product. Five buttons force every feature into a clear shape.
  • Tool use is a good interface between a language model and a simulation. The simulation owns the physics and the ranges, the model only picks which knob to turn and explains it.
  • Sending the screen state with each question removes most of what makes voice assistants feel lost.

What's next

  • Run on a Fire TV Stick 4K Max and tune the render budget on the device.
  • Deploy the Lambda and API Gateway stack and switch speech to Amazon Polly.
  • Stream tool calls (ConverseStream) so the scene starts moving before the explanation is finished.
  • Port Fusion Power Plant, Particle Collider and Metal 3D Printer.
  • A Vega OS build with the Vega WebView, once WebGL support there is confirmed.

Built during the hackathon (required for existing projects)

Fire OS shell (Kotlin, Leanback, WebView, key bridge, SpeechRecognizer), 10 foot D-pad UI, voice to simulation through Amazon Bedrock tool use, TV performance mode, and the open source megastructures-engine package. The physics scenes existed before. See "Is this a new project or an existing one?" above for the full list.

Limitations (honest)

  • Not yet run on a Fire TV device or emulator. The Android project was written without a local Android toolchain and has not been compiled. Performance numbers are from a laptop GPU.
  • Lambda, API Gateway, S3 and Polly are written as code and IaC and unit tested locally, but not deployed. Bedrock is the only AWS service called for real.
  • The video replays real Bedrock answers that were recorded through the app the same day, so the film is exact. Live mode calls Bedrock for every question.
  • Three of six films are live. The other three are "coming soon" tiles.
  • The simulations are simplified models built on public figures, for explaining ideas.

Built With

Share this project:

Updates

Submission history