scaledrive.ai - LLM for scalable robotics

scaledrive.ai
autonomous vehicle avoids the obstacle (ball on the road)
autonomous vehicle avoids the obstacle (ball on the road) without LLM maintaining small safety distance to the ball
AV avoids the ball with openAI API input keeping greater safety distance to the ball taking into account humans that might pick up the ball
open AI chatGPT reasons about the context helping the autonomous vehicle to take appropriate actions

Inspiration

One of the major challenges in self-driving and robotics, in general, is the ability of robots to deal with rare edge cases often caused by hard-to-predict human behavior or context that the robots don't understand. This challenge limits the scaleability and safety of today's systems. Example

What it does

We integrated generative AI with the autonomous driving system to help the robots better reason about such rare events and the context of the situation the robot is currently in, making them more generalizable and safe.

How we built it

We demonstrated the ability of our system using simulation of an uncommon road scenario in which our system successfully reasoned about possible human actions that can lead to a collision with an autonomous vehicle. With the help of openAI API, our system behaves safely and avoids collision. The scenario features a ball lying on the road. Based on the description of the scene, chatGPT provided an understanding of the context that a state-of-the-art autonomous stack would not be able to do. It reasoned that "other actors such as humans or animals might pick up the ball, ... and the scene would change accordingly". Based on this understanding chatGPT recommended to increase the safety distance to the ball making it safer in case a human would pick up the ball.

Challenges we ran into

The data transformation between robotics software and LLM represents a minor challenge that we solved during the hackathon.
The outputs of the LLM are not consistent. Therefore the LLM functionality has to be deployed under an "safety envelope" that controls it's outputs.

Accomplishments that we're proud of

Our autonomous vehicle successfully avoids possible collision with humans reasoning about complex context with help of chatGPT.

What we learned

LLMs can significantly increase the safety and scalability of autonomous driving technology and robotics in general.

What's next for scaledrive.ai - LLM for scalable robotics

We will integrate LLM with our autonomous driving stack to enhance its capabilities and commercialize this technology with existing and potential new partners and customers.