Inspiration
With recent advances in LLM we have been working on ways to integrate prompts to control robots for complicated tasks that don't need custom programs.
What it does
We are focused on robotics problems related to the 100 percent automation of organic farming. A typical example in agriculture is having a robot arm pick tomatoes and sort them by grade. We have been working on the LLM integration in a simulator/digital twin environment we developed. For the hackathon, we wanted to pick something a little more fun, and actually get a real robot arm working under LLM control outside of the simulator. Playing checkers seemed a worthy challenge, so let the fun begin.
How we built it
We are using a 100% Java-based robotics framework we developed as the API presented to ChatGPT/Gemini/Claude, with some mythical prompt storytelling to convince the LLM to write some code that would control a robot. Most of our efforts have been developing robots that work and stay in the field doing agriculture-related tasks 24/7. Last month, we built a 6-axis open source/hardware robotics arm from Annin Robotics but had not allocated time to writing the Java-based drivers to control the arm and to set up the digital twin that would run in the simulator. With the hackathon, our time was spent getting the robot arm working in the simulator and on the table, and trying to convince ChatGPT that it was okay to play a game of checkers in the real world.
Challenges we ran into
Balancing the time needed to integrate many different hardware and software elements together is always a challenge. In particular, setting up a digital twin for the Intel RealSense camera attached at the end of the simulated robot arm for depth and location perception of the checkers without providing that information in advance is challenging. Having that same code work in the real world is what makes this project worth doing.
Accomplishments that we're proud of
Writing an enterprise robotics control system in Java, that has an API that allows LLMs to do amazing things is just the beginning of great things to come.
What we learned
We will continue to underestimate how long it takes to do things that have never been done before and why caffeine is so important to the process.
What's next for OneTouch Robotics
With confidence, we can get ChatGPT to play checkers in the real world, we will transition back to problems related to fully automating growing organic vegetables that can scale with robotics.
Log in or sign up for Devpost to join the conversation.