Spot the Rover

Spot

Inspiration

We were taken by the idea for an automated guide dog for the blind, using visual slam and a pathfinding algorithm to guide a visually impaired user through an obstacle-filled, changing environment, but given our limited experience with ML, and the idea of a group-controlled robot that came up earlier during brainstorming (inspired by twitch plays pokemon), we decided to build MassCommand(WIP).

What it does

MassCommand leverages the wisdom of the crowd, controlling a LeKiwi mobile robot with the most popular inputs given in real-time by a group of twitch viewers. While choosing one person to control a robot could have a wide range of outcomes, a group choosing the inputs averages the skill range, hypothetically leading to pretty alright robot driving. In our demo, a group of 5 chatters lead a blindfolded Anthony around the venue solely with twitch chatting.

How we built it

After setting up the robot and our local machine correctly, we picked apart the lerobot library to find the code we needed to command the robot the way we wanted to. We pulled from an existing twitchPlays library to get the most popular messages from a twitch chat. We came up with different commands that chatters could use: “forward/backwards” (moves the robot a certain distance), “left/right” (turns the robot 30 degrees), “up/down” (moves the arm up and down), “open/close” (opens and closes the claw). We translated these commands to velocity and position information, given to the robot every second. We then figured out how to stream the robot’s camera output through udp to our laptop using ffmpeg, enabling us to start a twitch stream with it, giving the twitch chat visual information about the robot’s location and surroundings.

Challenges we ran into

Setup was a big pain, even with the help of multiple experts from bitrobot it took us the whole first day and then some to get the robot moving. Turns out the real world is a lot harder to work with than just software lol.

Accomplishments that we're proud of

Getting something working at a hackathon is always a joy, we popped off when we saw our twitch chats move the robot for the first time. We’re super proud to have learned such a complex system fast enough to make something cool that works.

What we learned

We learned a ton about robot-computer communication, how finicky hardware can be, and the potential that robotics have to make a difference in the world. Our debugging skills specifically improved a ton over the course of the weekend as we traced the way inputs and outputs move across the lerobot library.

What's next for Untitled

A group-controlled robot has the potential to be used in many different applications. For the content-creation space, we were thinking we could expand upon our code base to create a real-time scavenger hunt, where the chat would find different qr codes in an area with the robot. A robot accessible on the open internet by anyone could be a powerful source of inspiration for children to take up a STEM field.