Inspiration

We came up with the idea based on Neptune North's challenge: Systems working in a distributed fashion, even when individual components are disconnected. We also considered Waterstons' challenge: connecting things which usually aren't connected.

What it does

A device can set up a "room". Any phone can scan a QR code and connect to a relay network, creating a "large screen" spread across each of the smaller phone screens. You can then screenshare from a device, or play a video. You can reposition phones by pinching between them, or manually reposition them on a laptop.

How we built it

The frontend is written in Typescript and Vite without a framework, using the Canvas API. The backend uses a tiny node server which serves the page and relays messages across devices in a room over WebSocket. It also keeps track of the clock to prevent drift. Each device broadcasts its state about once a second, if there isn't a heartbeat in 2 seconds, the device is disconnected. Casting uses WebRTC. We also use the rotation sensor and accelerometer in phones for "lie-flat-detection" and pickup detection.

Challenges we ran into

We struggled a lot to sync video between devices. Because different phones were receiving the stream at different points, video could lag behind on some devices and ruin the illusion of a full screen. To fix this, we embedded a strip of 32 black/white blocks into the stream, encoding its timestamp. When a phone receives this, it processes the timestamp and crops the strip out, delaying the stream to that timestamp, which puts each video in sync! We found that sync drift could also occur when relying on device clocks. To solve this, we made the clock server-authoritative and stopped trusting the clientside clocks entirely. Also, we ran into a bunch of random little issues, like the browser not reporting phone screen size, and iPhones not having a vibrations API.

Accomplishments that we're proud of

We're proud that the project "just works"! By doing it entirely with typescript in the browser, we ensure that compatibility works by default on most devices. It was important to us that a user didn't have to install anything or undergo any friction to get this running, so we planned this from the start.

We did lots of hacky things to make things in this project work.

  • Safari doesn't allow vibrations, so to trigger haptics on iPhones we toggle a hidden "switch" checkbox to trigger a vibration.
  • To determine screen size, we literally just get the phone model and do a lookup to get its resolution. We couldn't set this up for bezel sizes on time, so we just have hardcoded values for those.
  • To work out video length, we seek to 10^101 seconds. For some reason the browser was often reporting the length as infinite, so we forced it to display length by doing this.

What we learned

  • NEVER EVER EVER TRUST THE CLIENTSIDE CLOCKS
  • We learned a lot about synchronising video between devices on a network

What's next for Teleport

  • We may open source the project or make it publicly useable because it's pretty cool!!!
Share this project:

Updates

Submission history