Touch TV

Inspiration Every house has the same problem. The TV remote goes missing, or you spend five minutes typing a search with arrow keys. Meanwhile, the phone in your hand has a great touchscreen. We wondered: what if your phone could just be the TV's touchscreen? That idea became Touch TV. We wanted to mirror the TV on the phone and let touches on the phone control the TV, and to build it with solid DevOps practices so it is reliable and deploys automatically.

What it does Touch TV runs on the Fire Stick and streams the TV screen live to your phone. When you tap, swipe or scroll on your phone, those actions happen on the TV in real time, so your phone acts like a real touchscreen for the TV. Pairing is simple: scan a QR code or enter a PIN shown on the TV.

How we built it On the Fire Stick, we capture the screen with MediaProjection and encode it as H.264 using MediaCodec. The video goes to the phone over WebRTC for low latency. A small signaling server on AWS helps the two devices find each other. Touch events travel back over a data channel and are injected on the Fire Stick using ADB over the network.

Because the phone and TV have different screen sizes, we scale every touch point: x_tv = (x_phone / W_phone) * W_tv and y_tv = (y_phone / H_phone) * H_tv.

For the DevOps side, every commit triggers a CI/CD pipeline that builds, tests and packages the APK. The backend is created with infrastructure as code (AWS CDK), and Amazon CloudWatch tracks latency, connection failures and frame rate.

Challenges we ran into Latency was our biggest problem. The first version felt laggy, so we tuned the bitrate, resolution and keyframe interval until it felt smooth. Fire TV has no touchscreen, so turning phone touches into valid TV input took a lot of trial and error. Apps like Netflix and Prime Video show a black screen when captured because of DRM, so we focused on the Fire TV interface and apps without that restriction. Different screen sizes needed careful coordinate scaling, and Wi-Fi drops forced us to add automatic reconnection.

Accomplishments that we're proud of We built a working end-to-end prototype where a phone really behaves like the TV's touchscreen: you see the TV on your phone and control it by touch. We also set up a fully automated pipeline from commit to deployment, with monitoring in place, and got the delay low enough that using it feels natural instead of laggy.

What we learned We learned how Android screen capture and video encoding work, how WebRTC signaling and data channels work, and that in a real-time app, latency matters as much as any feature. We also learned how to apply CI/CD, infrastructure as code and monitoring to a project that runs on a physical device.

What's next for Touch TV Next for Touch TV, we want to make it work over the internet instead of only on the same Wi-Fi, add the phone keyboard and voice typing for faster search, support multi-touch gestures, and extend it to other Android TV devices.

Built With

• adb
• amazon-api-gateway
• amazon-cloudwatch
• amazon-fire-tv
• amazon-web-services
• android
• aws-cdk
• aws-lambda
• ci-cd
• fire-os
• github-actions
• h264
• infrastructure-as-code
• kotlin
• mdns
• mediacodec
• mediaprojection
• webrtc
• websocket