Sherpa

Networking with autonomous vehicles and navigating you through traffic congestion. Optimizing traffic flow with real-time data insights to reduce delays.

Comment

Inspiration

Imagine wasting a year of your life stuck in traffic. This is the reality for drivers in metropolitan cities worldwide. Our project addresses this problem at its core. On Halloweekend, we saw a Waymo pulled over after it had crashed into the sidewalk. Despite being controlled by specialized data, autonomous vehicles like Waymo sometimes fail in unexpected situations.

This got us thinking! What if a network of Waymo cars could collaborate, sharing data in real-time to learn from each other and the unpredictable nature of human drivers? And what if these vehicles could work together to detect traffic patterns, advising drivers on optimal speeds to avoid congestion and minimize stops at red lights?

As engineers, our goal is to make the world more efficient in all aspects. This is our grassroots inspiration for the project.

What It Does

Our platform serves as a central hub to monitor and analyze networks of autonomous cars. For example, when analyzing a congested area in Austin, our platform simulates potential collaboration and reasoning between autonomous vehicles using large language models (LLMs). To achieve this, we use a framework called CoMAL, which divides the tasks of the LLM into smaller sub-tasks for efficiency.

The platform’s collaboration module allows vehicles to communicate with each other. In our initial case, we tested this with three vehicles, but as more vehicles join, we envision adding further modules for reasoning, execution, environment perception, and shared memory among all vehicles.

Beyond providing real-time traffic flow information, our platform offers actionable data insights after the cars communicate and reason with each other. These insights include the potential time savings from driving at ideal speeds, reductions in fuel consumption, and the prevention of CO₂ emissions. We think it’s exciting to envision cars collaborating and reasoning with each other—whether for exploratory purposes or to reduce congestion and collisions.

How We Built It

We used TomTom Maps and Traffic API to visualize traffic flow. Leveraging real-time data on vehicle dynamics and environmental conditions, we vectorized this information and performed searches using IRIS Vector Data Search to analyze data and generate meaningful insights.

To further demonstrate our solution's real-world potential, we trained a small object detection model, YOLOv11, on images captured over the past 24 hours from Austin’s streets. We propose that car manufacturers and cities develop open-source datasets with detailed information about vehicles (such as frontal area, engine power, and driveline efficiency) and roads (such as speed limits, grades, and distances). Additionally, a dataset for traffic lights could provide cycle times and phase changes.

We also utilized BoundaryML's BAML to convert LLM responses into type-safe objects, ensuring that data is structured and consistent for downstream processing.

With these datasets in place, our platform could match detected vehicles to the dataset, extract relevant information, and offer optimized driving speeds while showing drivers how their actions impact the environment.

Learnings

Through this project, we learned about the complexities of creating a collaborative network of autonomous vehicles, especially in terms of data sharing and real-time decision-making. Working with object detection and real-time vectorized data search taught us about handling large datasets efficiently and the challenges of scaling our model to a full fleet of vehicles. Additionally, implementing reasoning and task management using CoMAL deepened our understanding of task-specific LLM workflows.

What's Next

Next, we aim to:

  • Expand our object detection model to analyze additional environments and scenarios in real-time.
  • Develop a standardized, open-source framework for vehicle and road datasets to support better data interoperability.
  • Integrate advanced environmental sensors to improve reasoning in adverse conditions (e.g., rain or fog).
  • Incorporate feedback from autonomous vehicle manufacturers to refine our collaboration, execution, and environment perception modules.

Ultimately, we aim to establish Sherpa as a city-scale platform that can optimize traffic flow, reduce emissions, and make autonomous driving more reliable through shared learning and collaboration.

Sherpa is the future. And together, we’re paving the way for smarter, greener, and more efficient cities.

Built With

Share this project:

Updates