Empyrean

Empyrean is a smart glasses prototype that helps people who are blind or have low vision identify everyday objects. Touch an item and hear its name out loud.

Inspiration

For most of us on the team, living with a bleeding disorder means carrying a worry we can’t always set aside. Before we left for HackUMass, we were already asking ourselves when we’d take our medication, what we needed to pack, and what we’d do if our plans changed. We were excited to get to HackUMass, and we were already thinking about what we could build to make someone else’s day easier. Our own experiences taught us how much the right accessibility support can matter. We wanted to apply that lesson to a simple moment: reaching for something and knowing you’ve found the right item.

What it does

Empyrean uses a camera mounted on glasses to follow the wearer’s hand and nearby objects. When the wearer touches something, a Bluetooth speaker says its name. If the label is visible, the system can give a more specific answer. Hearing “Heinz ketchup” tells you which bottle you’re holding. The live dashboard shows what the camera sees, which object the wearer is touching, and a log of recent events. In our controlled demo, a hazard warning can also interrupt speech so the warning is heard first.

How we built it

We mounted a Logitech webcam on a pair of glasses and angled it down so it could see the wearer’s hands. On the laptop, YOLO detects and tracks objects, MediaPipe locates fingertips, and Grok Vision can refine an object’s name using details such as its label. We connected the camera, audio, FastAPI dashboard, and Tiger Data event log through a shared message system. That lets the dashboard and JBL speaker respond to the same touch event.

Challenges we ran into

After a delay at Boston Logan, we arrived late and had to find a place to work before we could get started. Once we were set up, our biggest challenge was getting object recognition to work when someone actually reached for an item. A hand could cover part of the object, and the camera angle changed what the model could see. We spent much of our testing tuning detection and fingertip tracking around that moment. Once touch detection was more consistent, we worked on how it handed off to audio. The speaker had to say the name at the right moment, and the alarm had to be able to interrupt it.

Accomplishments that we’re proud of

We got the camera, speaker, and dashboard working together. In one demo, the camera tracked a hand touching a toothpaste tube, and the same touch appeared in the dashboard’s event log. We’re also proud of how we worked as a team. Each of us owned a part of the project. We used branches and pull requests to review changes and test our modules together, which helped us bring four pieces of work into one working demo.

What we learned

Agreeing on what each event would contain made it easier to build our modules in parallel and connect them later. We also learned how much testing a prototype with a real hand and a real object can reveal. Detecting an object on a table is one problem. Keeping track of it while someone reaches for it is another.

What’s next for Empyrean

We want to put prototypes in the hands of people who are blind or have low vision and learn from them through guided trials. We’d ask which moments make spoken identification useful, where it gets in the way, and what the system misses. We’d use that feedback and testing data to improve it before trying more demanding settings, like telling similar products apart on a grocery shelf. We know what it feels like to carry extra questions through an ordinary day. We’d like someone to try Empyrean and tell us it took one of those questions off their mind. That’s what we’d like Empyrean to give people.