Inspiration

Out of the 430 million deaf people, tens of millions are born as or are profoundly deaf. We cannot imagine what it feels like to have never experienced something so fundamental as hearing a loved one's voice.

What it does

Def Leppard is a visionary device that augments speech and sound from the environment into visuals and vibrations through AR glasses and haptic motors. Through the AR glasses, the user can view a HUD showing live captioning per person and a coloured bar displaying the ambience of the noise levels. From the haptic motors, the user can feel the sensation of sounds around them and be alerted of any noise, speech or otherwise.

How we built it

We have major components from software, to hardware, and embedded software, driving our project:

Augmented Reality: XREAL One AR glasses and the XREAL Beam Pro phone paired together to create the viewport. The glasses have no camera of their own and need a USB-C DisplayPort source, so the Beam Pro is the display, running a fullscreen Chrome HUD into the XREAL in head-locked mode so captions stay fixed in the wearer’s view. The display is additive: black is transparent, which is why the HUD is a dark page with bright text. The Beam Pro also contains the microphone used for transcription; the Jetson never touches that audio.
Hardware The Nvidia Jetson Orin Nano Developer Kit, along with the OAK-1 camera, acted as the brain and eye of the project. The OAK-1 is a world-facing USB 3 camera at about 60 fps, filling in the camera the glasses do not have. The Jetson runs face tracking on its GPU, serves the HUD to the Beam Pro over the local network, and separately talks to the Arduino and haptic driver. The analog sound sensors cannot plug into the Jetson cleanly, which is why an Arduino Uno sits between the sensors and the board.
Software: We implemented Speechmatics speech-to-text API to transcribe conversations between multiple people with low latency. We used a face and mouthing tracker to overlay captions over the person who is speaking. On the Jetson, MediaPipe Face Landmarker finds faces in the OAK-1 feed and scores who is talking from how much each mouth opens and closes; a smile holds still, speech does not. The HUD itself is a web app on the Beam Pro: Speechmatics streams timestamped words with speaker labels, and we fuse those labels with lip-movement scores so each line of text sits on the person who said it, in their own colour, rather than as one shared subtitle bar. If a face drops out of view and comes back, we re-identify them so their captions and colour stay attached. Speech with no visible owner is kept off the glasses, and any nonhuman sounds, such as barking, honking, etc. are shown as emoji indicating sounds that the person should be aware of.
Embedded software: An Arduino Uno runs custom firmware that samples three analog sound sensors. The boards do not output comparable signals, so the firmware sends a versioned serial stream to the Jetson forty times a second, and the Jetson calibrates each side independently against silence and a known sound before deciding left, right, or centre.
Haptics: Those directional decisions drive a TITAN Core and three haptic motors outside the field of vision (behind the ears and behind the head). Short pulses scale with loudness so the wearer can feel where a sound came from and be alerted to speech, a knock, a honk, or anything in the environment. Captions and haptics run as separate paths, so a loose sensor never takes the HUD down.

Challenges we ran into

We ran into many challenges throughout the entire stack:

Lack of hardware: Originally, we planned to use lots of microphones to use time difference of arrival (TDOA) to locate where sounds are propagating from. However, we were unable to secure microphones. Therefore, we used visual processing to understand which person is speaking, and the analog sound sensors to determine the direction of loud sounds.
High latency and low accuracy/frame rate of captioning: When we originally built our prototype, we used a Raspberry Pi 5 along with the Raspberry Pi camera. Although the software worked well on a MacBook, the Raspberry Pi had high latency and the camera was low quality. We wanted to improve the latency and accuracy, so we pivoted to migrating to the Jetson platform instead. This ended up being a challenge in and of itself:
Improperly programmed Jetson: After migrating, we saw that instead of improving performance, the video feed was only 10 FPS and the captions were outdated. Through troubleshooting with Codex, which had built the migration in the first place, we discovered that the inference was running on the OAK-1 camera instead of the Jetson because it was unable to use the GPU. We resolved this by installing the necessary firmware and moving the accidentally misplaced computation from the camera to the Jetson.
Chaotic circuits: While soldering a broken connection on the haptic motor controller, a small resistor came off due to the heat. We did not have access to flux or soldering paste. We mostly resolved this by shorting the circuit with circuit analysis, imperative measurements, and soldering.

Accomplishments that we're proud of

We are proud of the work that we accomplished through great teamwork and willingness to keep trying. Completing such an ambitious and comprehensive hardware project was an amazing experience for all of us. Some technical accomplishments include:

Building novel circuit designs with many interlinked components
Debugging technologies which are new to us, such as the AR glasses and the software for the project
Working within the hardware constraints which were presented to us

What we learned

From building Def Leppard, we gained experience integrating the XREAL One AR glasses with multiple different circuit boards and elements. We also learned the importance of pivoting ideas to keep momentum.

What's next for Def Leppard

A technical consideration for the future of Def Leppard is the integration of 3D spatial sound into the AR glasses and haptic motors for features such as a locator bar of sounds around the user, similar to the feature in the video game Minecraft.

Moving forward with this project, getting in touch with people who are profoundly deaf or those who work with the deaf community would be excellent to gather feedback and have proper testing of our project.