Catbot

Point Cloud and Position Graph
Robot Front
Robot Side
Robot (without decorations)
Component Summary

Inspiration

Originally an idea to become a teleoperated camera robot to be used to navigate the peer's environment in video calls, a fully autonomous robot got us thinking. What if the robot could become the peer?

We've all faced the scenario where we've locked ourselves out of the house, are too lazy to grab something, or can't reach something off a high shelf. What if as soon as you entered a zone, the robot could autonomously perform the action for you? For example, if you walked close to your house, the robot could open the door for you and greet you at the door. If you sat on the couch, it could automatically bring you the TV remote. If you went into the kitchen and opened a cabinet, it could automatically follow you and offer to open the door.

While we do admit that Catbot doesn't have all that functionality yet, it does have the backing to reach its full potential.

What it does

Catbot uses sensors in a phone to perceive the environment and its own movement. Our kit robot does not come with wheel encoders; to work around this we integrate the phone’s gyro, accelerometer and camera. Instead of relying on the robot’s open-loop behavior, we fuse a heuristic model of the robot’s open-loop dynamics with the phone’s gyro and accelerometer readings via a Kalman filter to achieve tighter dead reckoning accuracy. Camera data is fed into an off-the-shelf SLAM library (ORB-SLAM) to produce slower, but more robust pose estimates. The phone on the robot pairs with another bluetooth device to provide proximity detection for Catbot’s owner, letting it “see” beyond its natural vision range and even through walls. All of Catbot’s internal data pipelines are accessible through servers on the robot, making it very easy to extend or build off of. Catbot's demonstrated action trigger is currently based on Bluetooth proximity between human and robot device.

How we built it

Catbot integrates many existing pieces of software to work its magic, including the following: Phyphox (Stream phone sensor data) IP Camera app (Stream phone camera data) ORB-SLAM (sparse Monocular SLAM, pose estimates) Flask (RESTful API - Python-based REST API to store robot state) Android (Google API - Android app development for Bluetooth distance between phones)

Catbot’s software is written in many languages, mainly Python but also C++, and Kotlin.

Challenges we ran into

The processor onboard a Raspberry Pi is quite weak, as showcased by the Pi dying multiple times trying to build a library, and the quantity and types of sensing that we were given were not very accurate or useful. We played a clever game of balancing storage space with the swap file in order to give it the processing power it needed to finish building essential libraries (Debian alone is quite hostile, Raspbian is even worse). In order to combat the lackluster sensors, we included an onboard phone which acts as an IMU and camera.

Accomplishments that we're proud of

Catbot is also quite robust. Beyond the added robustness given by state estimation, Catbot’s software is built in a modular way, so that any individual component crashing does not bring down the entire system. For example, when the phone camera drops frames or loses connection altogether, the server it talks to stays alive and can continue to serve other state information, such as bluetooth signal strength or pose estimates.

What we learned

We learned a lot about building an end-to-end project, starting from the hardware design all the way up to android app development.

We also learned more about how (not) to schedule work at a quick hackathon pace :’)

What's next for Catbot

Catbot could use additional hardware and peripherals to accomplish more complex tasks. Rather than just driving to places, Catbot could then help with other monotonous tasks around the house.

Cat ears would also be nice :)