Seek-Bot

The user inputs whatever they want the bot to find, and using the OpenAI API for NLP and YOLOV8 computer vision, this talented robot is able to turn and find any item and move to it, "seeking" it out.

Comment

Inspiration

We were inspired by the concept of ConceptGraphs, an area in CV research combining 3d-point clouds and LLMs for people to select objects in scenes and control robots

What it does

Our project uses the RPI on our car to stream video over Wi-Fi to a laptop running a YoloV8 Object Detector and an instance of the OpenAI GPT 3.5 API. While it isn't powerful enough to use 3d point clouds, it does demonstrate rudimentary functionality!

How we built it

We tried different models and configurations for our hardware. We also had some issues configuring our image processing pipeline.

Challenges we ran into

We learnt a lot about deploying various models, especially on embedded systems, and learning how to use networking to create capable distributed systems. There were also a lot of tool setup related hassles we ran into.

Accomplishments that we're proud of

We are proud of achieving our major objectives, and creating a low cost and low scale version of the ConceptGraphs research!

What we learned

What's next for Seek-Bot

We want to refine our object detection by using better modes, and we would want to experiment with more sophisticated control algorithms for our motors.

Built With

Share this project:

Updates