ketchup

Inspiration

Since everyone is being propelled into an AI filled world, everyone seems worried about having their job being gone. What if we change that and have AI work for us instead of taking our work?

What it does

Ketchup is a prototype for a robotic system that has a LLM embedded within it to help orchestrate tasks without training. The LLM is provided with full visual information and context, allowing it to break down tasks into small steps. Then we provide the agent with the ability to act on those steps to complete the tasks through arm movements, rotations and different attachments. With adequate adjustments, this can fully accomplish household tasks with ease.

How we built it

The agent was built on Python, where we used the Lerobot library to repurpose a SO-101 arm for this purpose. Object and spatial recognition was done using a YOLO model, and the orchestrator agent was powered by Gemini. Using Solidworks CAD software, we created a custom modular attachment joint and attachments using 3D printing.

Challenges we ran into

Keeping track of the current position of the arm was a massive challenge, and required many different attempts from our end including using other libraries, looking through deprecated code, trying to learn the math behind forward kinematics, etc.

Accomplishments that we're proud of

We are very proud of the fact that the core feature of the project works. Despite the short time-span, we were able to build out a good proof-of-concept that works well in some situations.

What we learned

We learned that working with hardware and circuits was really hard! Fried a lot of servos.

What's next for ketchup

We feel that ketchup has a lot of potential beyond our own projects. There has not really been a mass-consumed robot produced before, and we feel that with the rise of AI this is the perfect time to explore its possiblities in the real world.

Built With

• python
• solidworks