67 Racer

Inspiration

The meme 67. It’s probably not going away soon, so we decided to lean into it and use that same hand motion to steer the car. We’ve also watched ML videos about reinforcement learning, and getting cars to drive autonomously or getting stick figures to learn to run (If you’ve watched Code Bullet before, this is exactly what he does). We decided to use the DeepRacers because when we were browsing around in the AET computer science closet, and saw the AWS logo on a robot, and we wanted to check it out.

What it does

67 Racer lets you teach a real car to drive in under a minute, just by waving your hand in the 67 meme format :D. While you drive, we record what the car sees alongside what your hand is doing. Then, a neural net trained on those few laps takes over. The car keeps driving, by itself, as your clone. Do it again with a second person and a second car, and you can race two human-trained clones head-to-head with nobody touching anything. Eventually, the AI can perform better than you ever could. There’s a virtual simulation that the AI uses to learn and then to test. You place the DeepRacer on the physical track and hit play. Try to race against it using manual controls, or get another bot trained on another person’s racing style. The DeepRacer clones your racing style and replicates your driving strategy so you don’t have to do the heavy lifting anymore.

How we built it

Hand tracking: Leap Motion → Python, mapped to normalized (steer, throttle) with a deadzone and smoothing Car bridge: lightweight Python service on each DeepRacer — subscribes to commands, publishes camera frames over MQTT. MQTT (aka, mosquitto) is a real-time websocket-inspired architecture, where you subscribe to topics and then get broadcast information in real-time over LAN. Cloning model: PyTorch CNN trained on the recorded laps, served back to the car at inference time Race infrastructure: Mosquitto broker tying the laptop, both cars, the overhead DeepLens, and an ESP32 race-light tower together

Challenges we ran into

Almost everything. It took us over 6 hours to even get connected to the DeepRacers because the school's Wi-Fi kept blocking all of our connections. All the hardware was running programs from 10 years ago, and we had to figure out how any of it could work together. The entire AWS end of the machines was hidden behind a website that wasn't supported by Amazon anymore, and we had to reverse engineer the entire API. We reverse-engineered their robot's operating systems and flashed a custom Ubuntu image onto the DeepLens and DeepRacer that let us control the ROS2 controls ourselves and drive manually. Furthermore, the robot's USB-based setup method didn't work, so we had to physically connect it to a monitor and completely reset the firmware manually. Also, LCPS's wifi permabanned our robots out of nowhere, so we had to go home and reset everything prematurely. Also, the battery packs were broken, so we had to use one from Harris Teeter. Also, the Python package for a library that we needed only worked on 3.12, which was really annoying since Linux kept forcing 3.14. Nothing worked on Windows for a solid 2 hours.

Accomplishments that we're proud of

Getting everything to work together. We could get into the DeepRacer eventually, but it took a few more hours to get it to move. The web frontend worked, but we had to figure out how to connect it to the Python backend. The simulation worked, but it wouldn't connect to the physical DeepRacer. The reinforcement learning also kept getting stuck on a corner. To fix this, we rewarded the AI for not hitting the wall instead of rewarding it for following the player.

What we learned

So so much about debugging random hardware. We had to deal with Python version conflicts, figuring out how to get camera drivers up and running, writing out a driver for the Deepracer camera itself, and hard-resetting the camera's operating system. "It works on my end" was a very real problem we kept running into constantly.

What's next for Ghost Racer

Improving the learning loop: Currently, the AI learns from a few laps of your driving. We would like to make it so that the AI actively improves based on its own real-time input. Actual racing: Currently, driving and training happen in isolation. We would like to see if we could train AIs to intelligently race against each other. They could come up with strategies to defend their position or overtake the other car. More reliable setup: It took a lot of effort to get everything connected. We would like to simplify that process and automate configuration so that users can get straight into driving as easily as possible, without hours of debugging. We're planning on open-sourcing our backdoor into the Deeplens and Deepracer. Interpretability: Right now, we don’t have a good idea of what the AI is thinking. Using things like racing lines could help us understand the AI’s strategies.

AI Usage: A significant amount of the boilerplate and integration code was generated by Claude Code.