DAAP Developers - DejaView Glasses

Inspiration

We built DejaView, a smart-glasses assistant that gives machines memory and makes that knowledge accessible to technicians hands-free. Our target users are new or inexperienced field technicians, including workers unfamiliar with a machine or site and non-native English speakers. Technicians currently have to search through manuals, fault codes, service records, and scattered notes to diagnose unfamiliar problems. DejaView lets them simply ask questions. It uses the machine’s history and patterns to explain previous failures/logs and provide guidance in English or Spanish. After a repair, the technician tells the safety classes what happened, allowing the system to update its memory and improve future recommendations. Combining safety and efficiency.

What it does and How we built it

We built the MVP with Next.js, React, and TypeScript, using a JSON-based memory store organized into machine, episodic, human, procedural, and fleet memories with the help of Cursor. Cursor integrated Google Gemini 2.5 Flash for voice input, speech synthesis, and camera access. We encountered several challenges, including robotic voice output, incomplete logs/memory maps, inconsistent Spanish translations, and unreliable connectivity. We addressed these with improved speech handling, generalized parsing, a translation layer, an offline rule-based fallback, and responses grounded in stored machine knowledge.

Challenges we ran into

Had some challenges with the language barrier implementation. We had to iterate the language couple times to make sure the live translation was accurate. We also dealt with incomplete memory maps and inconsistent log formats, overcoming these by developing generalized parsing responses and being more specific.

Accomplishments that we're proud of

We successfully created an end to end, hands free assistant that bridges multi-modal inputs (voice, vision, and contextual memory) into a single functional smart-glasses interface. We are especially proud of designing a self-improving memory architecture that captures hands-free technician feedback after a job to automatically update machine histories for the next worker.

What we learned

Building DejaView showed us how multi-layered memory structures from balancing episodic site data with fleet-wide patterns, can significantly reduce diagnostic overhead and overcome language barriers.

What's next for DejaView Glasses - CAT Track

We would pilot DejaView with a dealer and new technicians, connecting it to real service records, manuals, and smart-glasses hardware. We would measure first-time-fix rate, diagnostic time, and technician feedback. Longer term, we would integrate live telematics and fault codes, support for more languages, connect the dealer’s full parts catalog, and add AR guidance that highlights the exact component a technician needs to inspect. DejaView directly addresses all required Cat Track points. It provides persistent machine memory through lifelong records of faults / repairs, personalized intelligence through language / experience based guidance, and self-improving systems that learn from every repair. It enables human - agent collaboration through natural language interaction that uses voice, vision, and smart glasses UI. Ultimately, DejaView lets Caterpillar machines remember what happened, reason about what is happening, and help the next person who works on them.