auri | Devpost

final prototype
Hardware Map
figma mockup & prototype setup
GIF
selecting available sounds to filter into specific locations for precise filtering
GIF
#amazing ui/ux
GIF
sound filtering, tagging, and toggling; optimized user experience by sliding disabled sounds to the bottom of the sound list
GIF
ui with confirmation to allow for misclick prevention
hardware map of how our mic inputs information to be processed in the linux and arduino
how we connected the michrophone, usb hub, power source, and uno q together
neural network model visualization for training various sounds
neural network 2D architecture
confusion matrix for model results
hardware setup with screen, LED, and vibration motor
speech to text using gemini API for name detection

Inspiration

Millions of deaf and hard-of-hearing people can miss important sounds every day, from smoke alarms and emergency sirens to someone calling their name or a baby crying. Existing solutions often require carrying a smartphone, specific technology, installing home sensors, which cause safety barriers to those with poor hearing.

We wanted to create something that's always with the user: a wearable that instantly translates important sounds into intuitive vibrations, helping users stay aware of their surroundings without constantly checking a phone screen.

What it does

Auri is an AI-powered wearable wristband that listens for important environmental sounds and notifies the wearer using distinct vibration patterns and visual indicators.

Rather than simply detecting that a sound occurred, Auri identifies meaningful events such as:

Smoke alarms
Emergency sirens
Someone calling your name (custom to any input)
Car horns
Baby crying
Dog barking

Each sound triggers a unique vibration pattern, OLED display, and LED allowing users to quickly recognize what happened without hearing it.

Our web application lets users customize vibration patterns, prioritize alerts, and switch between modes such as Home, Outdoor, Sleep, and Custom modes.

How we built it

Frontend

Designed all the front end application with Figma
Coded it using React, Next.js, Typescript, and Base44
Added accessibility settings for customization for all to use

Backend

Created a model using Edge Impulse
Used open source data of dogs barking, ambient noise, firetruck, and baby crying
Trained a 1D neural network with custom layers (dropout, pooling, reshape) based on repeated trials to increase accuracy
Added Google Gemini API to be able to detect any custom name (without having to retrain a new model every time)

Hardware

Used the Arduino UNO Q, OLED screen, LED RBG, vibrato modulino to wire our hardware bracelet
Connected to a power source and microphone for audio input
Researched wiring diagram to complete the task with stripping wires and connecting to fit our prototype

Full-Stack Integration with Arudino AppLabs

Used socket-io to connect the frontend to the backend
Connected the hardware aspects to the backend
Made it wireless with a powerbank for better accessibility

Challenges we ran into

Building Auri required solving challenges across machine learning, embedded systems, backend development, and hardware engineering.

Improving AI accuracy: Developing a reliable sound classification model was challenging because environmental sounds vary with distance, background noise, and recording conditions. Using Edge Impulse, we iterated on our neural network architecture, feature extraction methods, training data, and model parameters to improve accuracy while maintaining low-latency inference.
Hardware integration: Combining components such as the Arduino Uno Q, OLED display, microphone, vibration motor, LEDs, and battery pack into a wearable prototype required extensive debugging. We worked through challenges involving power delivery, strip-board soldering, wire management, and component placement while keeping the device functional and wearable.
Learning new technologies: This was our first time developing with Arduino AppLabs, building a backend service, and integrating AI models with embedded hardware. Through experimentation, tutorials, and lots of debugging, we learned how to connect firmware, backend services, machine learning, and frontend development into one cohesive system.

Accomplishments that we're proud of

Built a fully functional AI-powered wearable prototype capable of recognizing important environmental sounds and delivering distinct haptic notifications.
Trained and optimized custom audio classification models using Edge Impulse for reliable real-time sound recognition.
Successfully integrated Google Gemini API, embedded hardware, backend services, machine learning, and a modern web application into one seamless end-to-end system.
Developed a companion dashboard using Next.js, React, and TypeScript that allows users to customize vibration patterns, configure listening modes, and manage device settings.
Designed, assembled, and debugged a complete wearable prototype, overcoming challenges in power management, wiring, strip-board construction, and physical packaging.
Learned and applied Arduino UNO Q development, with an MCU and MPU to be able to use machine learning with hardware

What we learned

This project taught us that accessibility technology requires much more than accurate AI. A model with excellent benchmark performance is only valuable if it delivers reliable, low-latency feedback in real-world environments.

We gained hands-on experience across the entire technology stack—from training custom audio classification models with Edge Impulse, integrating Google Gemini API into our inference pipeline, programming embedded systems with Arduino, and building a responsive companion application using Next.js, React, and TypeScript.

We also learned how important backend architecture is when connecting AI with embedded hardware. Designing a low-latency communication pipeline between the sound recognition system and the wearable was essential to creating a responsive user experience.

Perhaps most importantly, this was our first experience building with Arduino UNO Q, creating a backend service, and integrating AI with physical hardware. It challenged us to quickly learn new technologies, debug across software and hardware boundaries, and appreciate how every layer of the stack contributes to a polished product.

What's next for Auri

Our long-term goal is to make Auri completely standalone by moving audio inference directly onto dedicated edge AI hardware, eliminating the need for an external computer while reducing latency and improving battery life.

We're also excited to:

Deploy optimized neural networks directly onto embedded edge AI hardware for fully offline sound recognition.
Expand the library of recognizable environmental sounds.
Add personalized voice and name recognition so users can identify specific people calling them.
Improve battery life, reduce power consumption, and further miniaturize the wearable.
Design a custom PCB and enclosure to replace the current prototype.
Train adaptive models that learn from each user's environment over time.
Conduct user testing with members of the deaf and hard-of-hearing community to refine both the hardware and user experience.

Why Auri is different

Many existing accessibility solutions depend on smartphones, fixed home sensors, or smartwatches that require a connected mobile device. Auri takes a different approach by combining AI, embedded hardware, and customizable accessible feedback into a single wearable designed for all people! (no matter preference of led, vibration, size)

Unlike solutions that simply detect that a sound occurred, Auri identifies what happened and communicates that information through distinct vibration patterns, allowing users to understand their surroundings without looking at a screen.

Key advantages include:

Always available — a wearable that's with the user throughout the day.
AI-powered sound recognition — recognizes meaningful environmental sounds instead of generic noise.
Low-latency notifications — optimized communication between the AI pipeline and embedded hardware delivers rapid tactile feedback.
Highly customizable — users can personalize vibration patterns, alert priorities, and listening profiles through the companion web application.
Portable — designed to work at home, outdoors, at work, and while commuting instead of relying on fixed infrastructure.
Accessibility-first — every design decision, from haptic patterns to the companion app, prioritizes intuitive interaction for deaf and hard-of-hearing users.

Rather than replacing existing accessibility technologies, Auri complements them by providing an intelligent, portable, and customizable layer of environmental awareness that fits naturally into everyday life.