Inspiration
Knowing the details of a robot functioning and with new frontier models that have come into the market drew my interests with Idea of doing a simulation with physics of AI using vision.Nebius token factory provided a route to integrate with the vision models gave the approach to go in that direction
What it does
Leveraging a reasoner model to predict the next move by looking at the image ,predicting the next action and making the move appropriately.A robo arm moves the path to that physical dimension and aims to lift the geometric cylinder placed on a table ,assuming the geometric cylinder to be a bowl and be able to lift it on reaching the object.The reasoner model predicts the next move and I used a simulation robot arm to bring it into action.
How we built it
Nebius Token Factory to create a dedicated endpoint Integrating Nvidia's Cosmos reasoner model to make the prediction of movements by vision. Python to create a custom script to send visual images to the reasoner model. Pybullet to provide the simulation based on the inference provided by the reasoner model.
Challenges we ran into
Lack of credits and running dedicated endpoints without enough room for trying all possibilities was the biggest challenge. Sometimes the prediction by the model went into hallucination,to rule out all risks i had to incorporate strict system prompt to the model
Accomplishments that we're proud of
Integrating the inference to a physical robot arm simulating the prediction
What we learned
Physical robot arm simulation and more understanding of the models prediction.Concept of dedicated endpoints ,spinning a vm and using of gpus.
What's next for RoboTasks
With just an experimental simulation presented here I have plans to do more see more of physics of AI in to action with vision models.
Log in or sign up for Devpost to join the conversation.