-
Tagline
-
Agentic AI Architechture
-
Demo rendering
Inspiration
Our story begins in the vast, turbulent expanse of the ocean. The quest to deploy fully autonomous surface vessels (USVs) faces a persistent, paralyzing obstacle: the Perception Bottleneck.
In marine environments, data acquisition is chaotic. Sensors (LiDAR and Cameras) battle dynamic wave movements and severe weather conditions. When training perception models, the available high-quality data points ($D_{\text{avail}}$) are vastly dwarfed by the required set for safe autonomy ($D_{\text{req}}$):
$$D_{\text{avail}} \ll D_{\text{req}}$$
Currently, the industry relies on human annotators to manually correct tracking drift and calibrate 3D bounding boxes. This manual loop is agonizingly slow, prone to human error, and subject to an exponentially scaling cost function:
$$\text{Cost}(x) = C \cdot e^{x}$$
We realized that until we break this broken, manual loop, rapid algorithm iteration for next-generation marine robotics is impossible. We were inspired to replace this human bottleneck with an intelligent, autonomous pipeline.
What it does
AquaAgent is an Agentic AI pipeline designed to automate 3D bounding box calibration and trajectory tracking for multi-sensor marine data.
Instead of humans spending hours scrubbing through noisy Lidar points, AquaAgent acts as an automated data engineer. It scans sequential tracking data, identifies spatial anomalies (like a bounding box suddenly drifting off a ship due to wave-induced sensor noise), and autonomously triggers physical reasoning algorithms to correct the trajectory. It transforms chaotic raw data into benchmark-ready, high-quality perception labels in a fraction of the time.
How we built it
We architected AquaAgent using a closed-loop Agentic workflow, seamlessly integrating deep learning perception with Large Language Model (LLM) reasoning:
- Initial Perception (The Baseline): We process raw multi-sensor data through a CenterPoint 3D object detection model. This generates a preliminary JSON sequence containing raw, often noisy, 3D bounding boxes ($P_{\text{noisy}}$).
- LLM-Driven Diagnostic Reasoning: We feed this sequential data to an LLM. Through carefully engineered prompts, we inject strict physical priors into the agent's context, such as rigid-body dimensions ($L, W, H$) and heading angle continuity ($\Delta \theta$).
- Autonomous Tool Execution: When the LLM detects a physical impossibility (e.g., a boat "teleporting" 5 meters in 0.1 seconds), it writes and executes a lightweight Python script. Utilizing NumPy and a Kalman Filter, the agent recalculates the true trajectory and overrides the anomalous data points.
- Impactful Visualization: The cleaned, post-correction data is passed directly into Rerun.io, rendering a beautiful side-by-side 3D comparison of the original versus corrected trajectories.
Challenges we ran into
- Combating Marine Noise Dynamics: The signal-to-noise ratio ($SNR$) in maritime LiDAR is notoriously low due to water reflections. Factoring in external forces like waves ($W_{\text{waves}}$) and weather ($W_{\text{weather}}$) made it difficult to establish a ground truth.
- Taming AI Hallucination: Initially, our LLM agent would sometimes "hallucinate" corrections, modifying data that was actually correct. To fix this, we had to heavily constrain the LLM, restricting its action space purely to invoking verified mathematical tools (like ICP algorithms and Kalman filters) rather than guessing coordinates.
- Complex Data Formatting: Bridging the gap between raw point-cloud binary files, JSON structured data for the LLM, and the Rerun.io visualization engine required writing custom serialization parsers under immense time pressure.
Accomplishments that we're proud of
- Slashing Annotation Time: We successfully reduced the data annotation overhead by over 90%, proving that Agentic AI can effectively replace the most tedious parts of manual data engineering.
- Physics-Informed AI: We successfully demonstrated that LLMs can do much more than chat—they can perform rigorous physical reasoning when properly grounded with spatial constraints.
- End-to-End Execution: We didn't just build a concept; we built a fully functioning pipeline that takes raw point clouds and outputs a cleaned, visually verifiable trajectory.
What we learned
- Prompting for Spatial Logic: We learned how to translate complex 3D spatial problems into language-based prompts that an LLM can parse and act upon.
- The Power of Tool-Use (Agentic AI): We discovered that an LLM’s true potential in engineering tasks unlocks only when you give it the ability to write and execute its own code to solve the immediate problem at hand.
- Visualization is Key: Integrating Rerun.io taught us that an algorithm's value is best understood when stakeholders can see the "before" and "after" clearly.
What's next for captain
(What's next for AquaAgent, the digital "Captain" of marine perception):
- Human-in-the-Loop Integration: We plan to implement a UI where the Agent flags its corrections as 'Pending Review', allowing researchers a one-click approval process to ensure absolute data integrity.
- Generalization Beyond the Ocean: The physics priors we developed are scalable. Next, we want to apply AquaAgent to Autonomous Driving (AD) datasets, calibrating noisy sensor fusions for cars navigating in heavy rain or snow.
- Edge Deployment: Ultimately, we aim to optimize AquaAgent so it can run directly on edge-computing devices aboard USVs, calibrating perception data in real-time as the vessel navigates the open seas.
