About the Project & Inspiration
The Guardian is an intelligent multimodal AI inspection co-pilot designed to modernize heavy machinery maintenance. It acts as a conversational partner for field inspectors by merging the structured reliability of CAT Inspect with the advanced reasoning of the CAT AI Assistant. The system uses a FastAPI backend and React Native frontend to coordinate voice-first interaction, predictive telemetry analysis, and 3D spatial reconstruction. Our goal is to shift inspections from reactive checklists to proactive diagnostics that keep machines running and workers safe. Inspiration Our team was inspired by combining the wonder of the event’s space theme with the grounded, rugged reality of Caterpillar inspectors. We view these inspectors as the superheroes of the industrial world because they operate in extreme conditions of grease, dirt, and high heat. However, our research into the Status Quo revealed that their current technology often fails them. We found real-world reviews describing apps that are buggy, slow, and frequently lose valuable information during sync. We realized that these superheroes deserve supertech. We set out to build a tool that allows them to keep their hands on the machine and eyes on the task while an AI agent handles the documentation.
How We Built It
We engineered a layered AI pipeline to turn unstructured field data into high-quality insights. The system utilizes Google Gemini for vision analysis to detect external faults like cracks, leaks, or corrosion. For internal health, we integrated Actian Vector AI to analyze engine telemetry data. To identify internal failures, the system converts live telemetry snapshots into vectors and compares them against historical patterns. We used a vector similarity approach where the machine calculates the relationship between the current state and known failure cases. The logic follows a standard calculation: similarity = (Vector A * Vector B) / (Magnitude A * Magnitude B). If this "drift score" indicates an anomaly, the AI proactively warns the inspector via a voice summary. We also implemented an optional 3D reconstruction pipeline using Nerfstudio to allow for a deep-dive inspection of high-wear parts like Excavator 325 bucket teeth.
Challenges We Faced
The primary challenge was orchestrating multimodal data to ensure the voice, 3D models, and telemetry all worked together seamlessly. We had to ensure the 3D reconstruction remained an optional, best-effort pipeline so it would not block the critical telemetry analysis if a scan failed. Another hurdle was managing real-time voice interaction while ensuring the AI could synthesize findings into a structured CAT-standard report without losing context.
What We Learned
We learned that impactful innovation must be human-centered. Technical execution only matters if it solves a real-world problem for the hero on the front lines. By using Supermemory to fine-tune the assistant to specific user styles over time, we discovered how to build a deeper level of trust between the inspector and their digital co-pilot. Most importantly, we learned that the future of intelligent inspections lies in reducing human fatigue and providing superheroes with the supertech they need to succeed.
Our Key Takeaways and what we are proud of
Our team is proud of merging the wonder of the space theme with the real world challenges of field inspectors by treating these workers as the superheroes they truly are. We successfully built a hands-free co-pilot that serves as the intelligent "baby" of CAT Inspect and the CAT AI Assistant, ensuring workers can stay focused on the machine instead of a screen. A key takeaway is that innovation must be human-centered to solve the frustrations of the "status quo" where buggy apps and lost data currently hinder safety and efficiency. We are proud of our multimodal reasoning engine that fuses 3D reconstruction and telemetry drift analysis to shift the industry from reactive checklists to proactive diagnostics. Ultimately, we are proud to have created something that lived up to our initial, ambitious goal.
Log in or sign up for Devpost to join the conversation.