Landing page of website (mobile view)
Landing page of the website (desktop view)
april tags for reach part
Live camera with option to switch between operator/technician. Zoom features as well.
AR detection of each april tag.
When a tag is clicked on, more information about the part shows up in nodes.
Information the AI remembers about the part. Speak a note as well as type a note.
User talking with the AI chatbot.
Full card info regarding the part which actionable steps on what to do next.
Dashboard with all updates and parts.

The squeal that came back

A BMW M3 has a faint squeal from the left front on the first cold stop each morning. A new tech would bed the pads and move on. The tech who worked on it eighteen months ago would not, because they remember the inner pad wearing to 2 mm while the outer sat at 6, and that the cause was dry caliper guide pins. That memory lives in one person's head. Scale it to a Cat 320 with 6,240 hours and a pump seal that is weeping again three months after replacement, and the person at the machine has never seen the oil sample that flagged it a year earlier.

Cat Track asks how to give every machine a digital memory. Our answer: put the memory on the part, make it talk, and make it listen.

What it does

A printed AprilTag goes on each part worth remembering. Open the live view on any phone, and every tag in frame gets a node. Tap one and a card locks beside the part: what this part remembers, one next step, live readings, and an inspection checklist written for this part and this reader. Flip the role switch and the same card is rewritten for an operator or a technician.

Then the write-back. Tap Speak a note and say what you found. The model turns it into a structured event, reads any measurement out of the sentence and updates that reading, and freezes a snapshot of all readings onto the event. The card rewrites itself. The next person who points a phone at that part sees a card that already knows.

A mic button beside the card opens a part assistant that answers from this part's history, streaming and read aloud. An operator dashboard ranks every asset and part by its worst reading, shows activity across the fleet, and prints new tags.

How we built it

Next.js on Vercel, Supabase Postgres, the Anthropic API, and the browser's own camera, speech recognition and speech synthesis. No native app. The memory is assets → components → events, plus readings per part and a tag map. Three small model jobs, all grounded in the timeline: write the card and checklist, structure a note, extract measurements. The prompt carries unresolved flags forward and refuses to call a part healthy while a reading is flagged. Card text is cached by a hash of the facts, so the only thing that changes it is a new fact.

Perception is AprilTag 36h11 via js-aruco2. The four corners give a homography $\mathbf{x}' \sim H\,\mathbf{x}$ that places the card next to the tag.

What we learned

QR codes are the wrong marker for an engine bay. AprilTags read at steep angles and a quarter of the size, so we dropped QR entirely.
A card that tracks the tag is a worse card. Techs want to read, not aim. Now it locks where it opened until you close it.
Field notes are untrusted input to a prompt. Every prompt wraps history and notes in data blocks and treats them as records, never instructions.
Landscape broke detection: frames were downscaled by width, shrinking tags below the detector's minimum. Scaling by the short side fixed it.

Challenges and honesty

iPhone Safari will not share the mic while the page holds the camera, so the live view pauses the camera during a note. There are no accounts and deleting a note is behind a shared PIN. Readings are simulated and the card says so. The Cat 320 is in the data for the story; the live demo is an M3 because that is what we could tag in the room.

Where it goes

Readings from Product Link and fault codes instead of a seed file. Memory that spans machines of the same model, so one pump card can say this seal failed early on three other 320s. And an assistant that can open a work order, not just suggest one.