BlueBook

Inspiration

Know-how is retiring. Every day more and more experienced workers leave the workforce due to America’s aging workforce, taking their wisdom and knowledge with them.

The solution? Bluebook.

We designed a way to preserve that invaluable knowledge and pass it on to the next generation. This system streamlines the onboarding process, especially when looking at physical, methodical jobs such as blue collar jobs, hence the name, Bluebook.

What it does

First, an expert records themselves doing a task once, on any phone. Our AI turns that video into a structured, step-by-step skill guide. A learner then puts on the glasses and is walked through the task hands-free:

Each step appears in their field of view as a short tooltip.
The exact tool or part they need is highlighted on the real object in front of them.
They can ask questions out loud with push-to-talk and get answers without putting anything down.

We’re starting with skilled trades, where know-how is retiring faster than it can be passed on. The same pipeline works for any hands-on task, from fixing a sink to changing a tire.

How we built it

Bluebook has three layers:

Perception: we run the source video through Whisper to transcribe the expert’s narration, and Gemini to read the video itself. Together they pull out every step, tool and object. That output is checked and packaged into a SkillPackage, a structured file with ordered steps, instructions, keyframes and the objects tied to each step.
Teaching: the SkillPackage streams to the G2 glasses as step-by-step tooltips in real time. Push-to-talk voice Q&A displays the user’s question to him through Gemini’s Live API and sends that question with the SkillPackage as context to the Gemini API for a response. This answer is then in turn displayed to the user and can be dismissed to return to the current step of the skill.
Interactive: the scene camera also tracks the wearer’s head pose using ARKit and uses a world view to plot all the objects in 3d space. This allows us to circle the object of interest for the current step

Challenges we ran into

The skill builder and skill display interface were built independently, so there were formatting issues when connecting the two parts together.

There were issues connecting the visual feed from the camera to the indicator on the glasses. The glasses have no built-in camera, so we had to implement our camera-centered features using an iPhone camera. Beyond that, most of the time sink in this step was finding a balance between overly jittery arrow movement and delay.

Accomplishments that we're proud of

A full pipeline that goes from one raw phone video to a guided lesson on real glasses.
A SkillPackage format general enough to describe almost any hands-on task, not just one demo.
Hands-free voice Q&A that answers in the context of the step the learner is on.
AR highlights that stay on physical objects as the wearer moves their head.

What we learned

How to make a trade in efficiency and accuracy by choosing between Gemini and Jev at different times in the process
How to take advantage of the iPhone’s built in technologies for object and face detection
Filtering is not always better. Our filters added ~450 milliseconds to the object detection/tracking time by trying to smooth out the tracking arrow, which was not worth the higher latency

What's next for Bluebook

A creator tool for experts: an easy app to record, review and publish a skill in minutes.
A Wikipedia for skills: an open library where anyone can upload a skill and any wearer can pull it up on demand, taking Bluebook beyond the job site into everyday life.
Smarter progress checks: use the scene camera to catch mistakes before they happen and adjust the lesson to each learner's pace.

Built to scale through quiet tech and privacy. Bluebook shares two core values with Even Realities, and both are key to scaling beyond a pilot:

Quiet tech: the glasses stay out of the way until you need a skill. Because they are so unintrusive, they look and feel like everyday eyewear, which creates a much lower barrier to entry for workers and learners who would never wear a bulky headset.
Privacy by design: we intentionally chose the Even Realities G2, which has no built-in camera. The camera is intentionally separate, and the AR highlighting feature can be disabled entirely. That means Bluebook can be worn on a job site, in a home or around coworkers without the worry of invasion of privacy, which makes it far easier to adopt at scale.