PinPoint

Inspiration

Computer science students not in ECE classes are hardly ever taught hardware programming. And getting into hardware is difficult for the uninitiated. This is a shame given how interesting hardware is.

What it does

It lets you learn and create hardware from scratch using audio and visual clues right on your desk.

How we built it

We built PinPoint using a multi-layer architecture:

FastAPI backend for handling voice/text lesson generation and device communication
MongoDB for persisting AI-generated lesson steps as JSON documents with visual elements (Text, Graphics, Line)
LightGuidePy SDK integration for AR projection control
WebSocket bridge for real-time communication between the host and LightGuide system

Challenges we ran into

Learning to use LightGuides software and SDK (it turned out to be a really well thought out and robust)
LLMs having no knowledge of LightGuide software or internal mechanisms (we had to resort to one short learning after exporting XML samples to the client)
LLMs producing brittle outputs that didn't quite work (used json_render's approach and wrote a json -> XML pipeline to create well structured templates)
Establishing connectivity to MongoDB on the client (had to create a vultr instance to create a reverse proxy to finally interface with MongoDB to data updates)

Accomplishments that we're proud of

Building a working multi tenant pipeline from AI generated Visualizations to actually displaying it on the remote LightGuide device. Having accurate, AI outputs mapped through the AR device Multi device connectivity even on OU wifi

What we learned

Working with AR hardware and its constraints Dealing with LLM output reliability in structured domains Real-time WebSocket communication between distributed systems How to bridge the gap between AI-generated content and physical hardware output

What's next for PinPoint

Expanding the lesson library to cover more hardware topics (sensors, actuators, microcontrollers) Supporting more AR projection hardware beyond LightGuide Building a community platform where users can share and remix lessons Exploring integration with physical hardware kits for a fully guided hands-on experience