Inspiration

Hearing is one of the easiest of our five senses to lose, yet one of the hardest to live without. However, most people who need a hearing aid will never receive one. The World Health Organization finds that by 2050, nearly 2.5 billion people are projected to have some degree of hearing loss, and more than 700 million will require hearing rehabilitation.

We talked to a Certified Nursing Assistant who spent a summer working at The Villas at Saratoga, a skilled nursing and assisted living facility located in the Bay Area, CA.

We learned that even in a privileged area, getting a hearing aid is expensive, and keeping one means fittings, clinic visits, batteries and repairs. Something this crucial should not be this inaccessible, especially in care facilities that are already spread thin. We envision a world where hearing aids are cheap and reproducible, without sacrificing detailed customization and fit.

What it does

Users upload a photo of their ears and face, and enter results from a hearing test. Auris then reconstructs their ear in 3D, deploying research agents to better understand the user's needs based on their hearing profile and listening situations.

Once Auris completes the user profile, it outputs a 3D preview of the earpiece seated in the user's ear and a printable STL file. There are no electronics, no battery, and no clinic visit, and the user gets a passive in-ear acoustic horn custom fitted to them.

For the 1 billion young adults at risk, this dramatically increases their quality of life and baseline levels of hearing.

How we built it

Ear reconstruction: a photo becomes a 3D ear mesh (AudioEar [1] + SAM 2 [2]), and a front photo gives the head shape (MediaPipe [3]), served by FastAPI

Acoustics: a 1D transmission-matrix model predicts insertion gain at the eardrum across 32 sampled ear canals, checked against the linear NAL-R prescription [9] and scored with HASPI v2 [8] in quiet and in babble noise

Research agents: three agents run in parallel (hearing, use context, acoustic design) on LangGraph + Deep Agents with Claude, reading the literature and freezing one experiment plan

Guardrails: code, not the model, decides the numbers and the winner. A finding only counts if its quoted passage appears in the source that was read. A design must not be worse than the open ear in any scenario

Frontend: Next.js + three.js for the 3D viewer, Supabase for auth

LangChain Deep Agent Architecture

The core driver of our project and experimental design is LangChain's deep agent. It has the ability to search, open full papers, find exact sentences supporting certain claims, and keep going for dozens of steps.

We're using four agents in the preliminary research phase, orchestrated by one deep agent. Deep Agents gave us a file system that keeps full papers out of the agent's context, allowing an agent to search a paper for exact wording without having to remember it, and a middleware stack to control what each agent is allowed to do.

As for the design process, it starts with a photo. A photo of the ear becomes a 3D ear mesh, and then an earpiece is cast from the person's own concha with the horn carved inside of it. The earpiece follows the outline of the user's ear opening.

Every design is test-built and scored. The lead has tooling to run a fit check since each and every ear is completely unique, and a design can be within the boundaries of our given dimensions and still not fit. The earpiece has 16 adjustable parameters.

Agent Architecture

Acoustic and Auditory Modeling

We use a 1D acoustic model to quickly search earpiece designs. It treats the earpiece, ear canal, and eardrum as a series of tubular sections such that we are able to exactly calculate how each design affects sound across 200 frequencies. Each section, of length \(L\) and cross-sectional area \(S\), is described by a transmission matrix that maps the pressure \(p\) and volume velocity \(U\) at its input to those at its output, with \(k\) the wavenumber including viscothermal wall losses [4]:

$$ \begin{bmatrix} p_1 \cr U_1 \end{bmatrix} = \begin{bmatrix} \cos kL & j Z_c \sin kL \cr \frac{j}{Z_c} \sin kL & \cos kL \end{bmatrix} \begin{bmatrix} p_2 \cr U_2 \end{bmatrix}, \qquad Z_c = \frac{\rho c}{S} $$

Chaining these matrices from the earpiece's mouth to the eardrum yields the eardrum pressure for any design. Since the open ear already provides 10–20 dB of natural gain around 3 kHz [5], every design is scored by its insertion gain: the eardrum pressure with the earpiece relative to the same ear left open.

$$ \mathrm{IG}(f) = 20 \log_{10} \left| \frac{p_{eardrum}^{with}(f)}{p_{eardrum}^{open}(f)} \right| $$

We then further narrow down our choices by modeling across 32 realistic canals, keeping designs that work well in a majority or all of them.

We then use a 3D wave simulation to check what the 1D model doesn’t. We place the user’s reconstructed ears on a head shaped from their own photo, capturing how the head and outer ear themselves filter sound. The simulation solves the linearized acoustic equations for pressure \(p\) and particle velocity \(\mathbf{v}\) on a 1.5–3 mm finite-difference time-domain (FDTD) grid, with the head and ears as rigid boundaries:

$$ \frac{\partial p}{\partial t} = -\rho c^2 \nabla \cdot \mathbf{v}, \qquad \frac{\partial \mathbf{v}}{\partial t} = -\frac{1}{\rho} \nabla p $$

We utilize acoustic reciprocity and simulate a click from each eardrum which gives us each ear’s full directional response. Because the transfer function between two points is unchanged when source and receiver swap, the sound the click radiates toward a direction \(\hat{r}\) equals the sound that would reach the eardrum from \(\hat{r}\). We record pressure \(P\) on a closed surface around the head and project it to the far field with the Kirchhoff–Helmholtz integral:

$$ F(\hat{r}, f) = \oint_S e^{jk \hat{r} \cdot \mathbf{x}} \left[ jk (\hat{r} \cdot \mathbf{n}) P(\mathbf{x}) - \frac{\partial P}{\partial n} \right] dS, \qquad H(\hat{r}, f) = \frac{F(\hat{r}, f)}{C(f)} $$

Normalizing by the click spectrum \(C(f)\) gives the ear's directional transfer function \(H(\hat{r}, f)\). A single run per ear thus replaces hundreds of per-direction simulations, and agrees with them to within 0.5 dB.

Finally, we use auditory models to turn sound into something you can actually hear. We use scenes, such as a conversation or a rave, by placing sound sources around the head. Each source filters through the ear’s directional response and hearing test, so sounds at pitches they’ve lost count for less. In each one-third-octave band \(i\), a source of level \(S_i\) arrives at the eardrum at level \(E_i\), and must clear an effective threshold \(D_i\) set by the greater of masking from other sources \(Z_i\) and the user's hearing threshold, \(X_i + \mathrm{HL}_i\):

$$ E_i = S_i + 20 \log_{10} \left| H_i(\hat{r}) \right|, \qquad D_i = \max\left( Z_i, X_i + \mathrm{HL}_i \right) $$

The Speech Intelligibility Index [7] sums each band's audibility above that threshold, weighted by its importance to speech \(I_i\) and a level-distortion factor \(L_i\), giving a score from 0 (inaudible) to 1 (fully audible):

$$ \mathrm{SII} = \sum_i I_i L_i \frac{\min\left(\max(E_i - D_i, -15), 15\right) + 15}{30} $$

A design's benefit \(\Delta\) is its improvement over the open ear, averaged across the user's scenes \(s\) with weights \(w_s\) reflecting how much each matters to them:

$$ \Delta = \frac{\sum_s w_s \left( \mathrm{SII}_s^{with} - \mathrm{SII}_s^{open} \right)}{\sum_s w_s} $$

Speech is scored with the Speech Intelligibility Index and HASPI v2, and the result is an earpiece design tuned to one person’s ear, hearing, and daily life.

Accomplishments that we're proud of

  • We were able to successfully create an end-to-end pipeline from a phone photo to a printable, personalized earpiece
  • This proves there is an accessible alternative to hearing aids that can be mass-produced without the need for batteries and clinics
  • With an analog hearing aid, it's easy to make certain frequencies harder to hear, but our pipeline ensures that we thoroughly test each design and that every hearing aid boosts the user's hearing ability across every frequency

What's next for Auris

Help those in need

References

[1] X. Huang et al., "AudioEar: Single-View Ear Reconstruction for Personalized Spatial Audio," Proc. AAAI, 2023.

[2] N. Ravi et al., "SAM 2: Segment Anything in Images and Videos," arXiv:2408.00714, 2024.

[3] C. Lugaresi et al., "MediaPipe: A Framework for Building Perception Pipelines," arXiv:1906.08172, 2019.

[4] C. Zwikker and C. W. Kosten, Sound Absorbing Materials. Elsevier, 1949.

[5] E. A. G. Shaw, "Transformation of sound pressure level from the free field to the eardrum in the horizontal plane," J. Acoust. Soc. Am., 56(6), 1848–1861, 1974.

[6] C. P. Brown and R. O. Duda, "A structural model for binaural sound synthesis," IEEE Trans. Speech Audio Process., 6(5), 476–488, 1998.

[7] ANSI S3.5-1997, Methods for Calculation of the Speech Intelligibility Index. American National Standards Institute, 1997.

[8] J. M. Kates and K. H. Arehart, "The Hearing-Aid Speech Perception Index (HASPI) Version 2," Speech Communication, 131, 35–46, 2021.

[9] D. Byrne and H. Dillon, "The National Acoustic Laboratories' (NAL) new procedure for selecting the gain and frequency response of a hearing aid," Ear and Hearing, 7(4), 257–265, 2986.

Built With

Share this project:

Updates

Submission history