Explainer
GIF
Logo
Omnara

Glass Agent

A hands-free AI copilot for field technicians, on Meta AI glasses. Look at a machine, ask out loud, hear the answer, and say "wrap it up" to close the job.

Built with: Meta AI glasses · Anthropic Claude · Freestyle · Omnara

The problem

Field techs work with both hands busy, so they can't dig through laptops, customer records or parts catalogs.
Generic assistants (like the built-in Meta AI) can tell you "that's a coffee machine", but not whose machine it is, what was fixed last time, or what the parts cost.

What Glass Agent does

Sees: takes a photo through the glasses camera when the tech taps once.
Knows: matches the photo to this customer's installed machine and uses its service history and our parts catalog.
Answers: in one spoken sentence through the glasses speakers. > "That's your Jura E8 we descaled in March; it's scaled up again and the filter is overdue, so descale it and fit a new filter."
Acts: "Wrap it up: descaling kit, new filter, half an hour of labor."
- Logs the job and prices parts and labor ($123.50).
- Produces a job report on the phone.
- Hands the job to a back-office agent on Omnara.

Deployed on Freestyle

The backend runs on a Freestyle VM with a public HTTPS style.dev domain.
It's kept running under systemd, so it stays up when our laptops are closed.
Job data is stored on the VM's persistent disk and survives restarts and redeploys.
The VM pauses when idle and wakes on the next request, which keeps it cheap.
Deploying takes one command (scripts/deploy.sh), and keys stay in a root-only env file on the VM.

Back office on Omnara

Every wrap-up launches an Omnara agent with the finished job report.
That agent drafts the customer follow-up email and a next-visit maintenance plan (when the descale and filter are due next).
It runs in the background, so the tech never waits, and every run can be inspected in the Omnara dashboard.
Two agents, one workflow: the field agent in the glasses stays fast and focused, and the back-office agent on Omnara handles the paperwork.

How it's built

Meta AI glasses: camera, mic and speakers, connected through Meta's Wearables Device Access Toolkit.
iPhone app (Swift / SwiftUI): live glasses camera feed, on-device speech recognition, text-to-speech to the glasses, job and report screens.
Backend (TypeScript / Node): an agent on Anthropic Claude (vision plus tool use) over customers, equipment, service history and the parts catalog.
Web fallback: the same Freestyle-hosted backend serves a web page, so anyone can try the full flow without glasses.

Built for a live demo

Every external call (Claude, Omnara) has a timeout and a fallback, so nothing crashes on stage.
Wrap-up still works if Claude or Omnara is down.
Every request and tool call is logged and can be followed live from the Freestyle VM.

Demo data

A fictional "Northside Coffee Service" with 5 office customers, their machines, service history and 15 parts. The live demo customer is the hackathon venue and its Jura E8.