ContROLL

Our Car
The Badge

Inspiration

We wanted the drone-scouting feeling from Rainbow Six Siege: send something around a corner, see what it sees, drive it from that view alone. The HTN x Solana badge has a screen and a D-pad, which is most of a controller already. We wanted to find out if it could be the whole thing.

What it does

ContROLL turns the badge into a complete remote-vehicle controller.

The D-pad drives a car we built
The badge screen shows live video from the car's camera at 18.8 fps
A person-detection overlay draws boxes over that feed

You drive entirely from the badge. No laptop in the loop.

How we built it

Three hops, each chosen to keep work off the slowest part of the chain.

Camera to Pi

An OAK-1 on the car encodes JPEG on its own Myriad X hardware encoder. The Pi 5 never touches pixels, it accepts a connection and forwards bytes. A Pi 5 has no hardware JPEG encoder, so compressing there would burn CPU for nothing.

Pi to badge

Frames go over UDP at the panel's native 320×240, chunked by hand so every datagram fits inside an MTU. TCP was the wrong choice on purpose: a lost packet stalls the whole stream until retransmit, and for video a frame that arrives late is worth less than the one behind it. Over UDP, one lost packet costs one frame.

Badge to panel

The badge decodes JPEG one MCU block at a time straight onto an ST7789T3 TFT over SPI. It never holds a whole frame, because it can't. More on that below.

Control path

Buttons go back to the Pi over UDP at 20 Hz, into a driver that maps them to two L298N channels.

Challenges we ran into

The vehicle changed halfway. We started with a drone, then learned drones weren't allowed. Rebuilding around a car meant new motor control, new chassis, and a control mapping that no longer had a flight controller underneath it.

Nothing had ever driven the badge's screen. The repo docs called it an OLED with an unidentifiable controller. It's an HS20HS072RX, a 320×240 colour TFT on an ST7789T3. We traced the pins out of the KiCad board file to bring it up.

The frame doesn't fit in RAM. A 320×240 RGB565 frame is 153,600 bytes and the ESP32-C3 has 327,680 total. Holding one frame while receiving the next isn't possible. Decoding block-at-a-time directly to the panel is what makes the stream work at all, and it cut what crosses the radio by roughly an order of magnitude.

Detection is 25× slower than the video. A round trip to the inference endpoint measured 480 to 560 ms. Inline, that caps the stream near 2 fps against the 18.8 fps the badge manages alone. So detection runs out of band. It samples occasionally and publishes boxes, and the badge draws slightly stale boxes over smooth video. Stale boxes on fluid video beat fresh boxes on a slideshow.

The badge had to run on batteries. It only associated on USB at first. We dropped transmit power from 19.5 to 11 dBm, halved the CPU clock, and added a 400 ms settle delay so the radio doesn't try to associate into boot inrush.

The motors have no encoders. The car can't know how far it has turned, and a full spin depends on surface, speed, and pack charge, which drifts as the pack drains. Turns are timed, with a calibration routine to re-measure on whatever floor we're demoing on.

Accomplishments that we're proud of

Brought up a display nobody had driven, on hardware we didn't design, from the board file
Got live video onto a microcontroller with less RAM than a single frame
Kept the project alive through a pivot that invalidated the vehicle it was built around

The failsafe is the part we'd point at, though. The badge sends button state 20 times a second while anything is held. Silence means released or out of range, and both want the same answer: no packet for the timeout window and the motors cut. That isn't a fallback bolted on afterward, it's the primary way the car stops.

What we learned

Getting control signals to a vehicle is easy. Getting useful video back is the whole problem, and every constraint is somewhere you didn't look first: RAM on the receiver, the wrong transport protocol quietly stalling, an inference call fast enough in isolation that destroys throughput in-line.

The recurring lesson was that the right fix is usually to move work off the weakest component rather than to optimise it where it sits.

What's next for ContROLL

Reduce glass-to-glass latency
Tune the turn calibration per surface
Build a small course people can drive from the screen alone, which is the only real test of whether the video is good enough to navigate by