Geopilot

Inspiration

We started with a simple question:

What if you could talk to a real-world traffic scene the same way you talk to an AI assistant?

Traffic cameras contain an enormous amount of information, but that information is usually locked inside hours of video. Understanding a scene can require manually watching footage, counting vehicles, estimating speeds, identifying traffic patterns, and switching between different maps and tools.

We wanted to make that interaction more natural.

Instead of asking someone to manually inspect a traffic camera, we imagined a system where they could point to a real camera, understand where that camera is in the world, and simply ask questions about what is happening.

That became GeoPilot — a conversational geospatial intelligence co-pilot that combines CCTV footage, satellite context, computer vision, and a digital twin view so users can explore and reason about a real traffic scene through natural conversation.

Our starting point was the existing TrafficLab 3D engine, which we extended into a broader system for discovering cameras, analyzing traffic, and interacting with the resulting geospatial data conversationally.

What it does

GeoPilot connects the physical traffic scene to a conversational interface.

Users can explore more than 229,000 CCTV cameras through a global map, filter cameras by location, select a feed, and view the available live stream.

Once a traffic scene is selected, GeoPilot can use a calibration workflow to establish the relationship between the CCTV camera and its corresponding satellite view. Computer vision can then detect and track vehicles in the footage and project their positions into a 3D digital twin.

The user can then interact with the scene using natural language.

For example, a user can ask:

"Show me cameras in Tokyo."
"What's the traffic like?"
"How many vehicles are there?"
"What speed are they going?"
"When was traffic busiest?"
"Generate a traffic analysis report."

GeoPilot combines the conversational response with the underlying visualization, allowing users to not only receive an answer but also see the evidence represented in the scene.

How we built it

GeoPilot is composed of a React + TypeScript frontend and a Python backend built around the TrafficLab 3D engine.

The frontend uses React, Vite, MapLibre, deck.gl, hls.js, and Framer Motion to provide the interactive geospatial interface. It includes a dark world map with more than 229,000 camera locations, a 3D globe view, camera filtering, live feed playback, and the conversational interface.

The Python backend brings together several pieces of the system.

TrafficLab provides the calibration, inference, tracking, projection, motion processing, replay, visualization, and analytics layers. YOLO is used for object detection and tracking, while vehicle positions, speeds, and headings are projected from the CCTV scene into the satellite-based digital twin.

For visual reasoning, we integrated Qwen-VL so the system can analyze captured traffic frames.

The conversational layer uses AssemblyAI for voice interaction. The voice agent can interact with our backend tools to search cameras and retrieve traffic analytics rather than relying entirely on generated guesses. Responses can then be returned to the user through the voice interface.

The overall system can be thought of as:

Camera discovery → CCTV feed → Calibration → Computer vision → Geospatial projection → Traffic analytics → Conversational AI → Visual + spoken answer

Challenges we ran into

One of our biggest challenges was connecting different representations of the same physical scene.

A CCTV camera gives us a 2D perspective of the world, while the digital twin needs to understand where objects are located within a geospatial environment. Establishing that relationship requires calibration between the camera footage and the corresponding satellite representation.

We also had to deal with the limitations of computer vision data. Detection and tracking are not perfect: occlusion can cause vehicles to disappear temporarily, tracking can become noisy, and projected 3D vehicle representations do not yet behave like physically accurate vehicle models.

Another challenge was bringing conversational AI into a system that contains real, structured data. We wanted GeoPilot to answer questions using actual camera and traffic information rather than simply producing plausible-sounding responses. This required connecting the voice agent to backend tools and analytics services.

Finally, we wanted the project to work at two different levels: a global camera discovery experience through Argus, and a detailed traffic intelligence experience for an individual calibrated scene. Connecting those experiences into one coherent product was an important part of the build.

Accomplishments that we're proud of

We are proud that we were able to bring together several technologies that normally exist as separate systems into one experience.

GeoPilot can take a user from a global map containing more than 229,000 cameras to an individual traffic scene, connect that scene to satellite context, run computer vision over the footage, and expose the resulting information through natural conversation.

We are especially proud of the digital twin component. Vehicle detections are not simply displayed as bounding boxes over the original video; their positions, speeds, and headings can be represented within the corresponding geospatial scene.

We are also proud of the conversational layer. A user does not need to know which API or analytics function contains the information they need. They can simply ask a question in natural language and let GeoPilot determine how to retrieve and present the relevant information.

Most importantly, we have a working prototype of what we believe could become a much larger interface for interacting with real-world geospatial intelligence.

What we learned

We learned that building a useful AI system around real-world data requires much more than adding a language or voice model.

The AI is only one part of the system. Reliable answers depend on the quality of the underlying detection, tracking, calibration, projection, and analytics pipelines.

We also learned how difficult it is to connect computer vision outputs to a meaningful geospatial representation. Small errors in detection, tracking, or calibration can become much more noticeable once objects are projected into a digital twin.

Another major lesson was the importance of grounding conversational AI in tools and real data. Instead of allowing the agent to invent traffic statistics, we designed the system so it can retrieve information from the underlying analytics and camera services.

Finally, we learned that the most compelling AI interfaces may not look like traditional chat applications. With GeoPilot, the conversation is only one part of the interface. The map, camera feed, digital twin, and visual responses all contribute to the answer.