Inspiration

We were interested in the idea behind this hackathon that technology can strengthen the building blocks that make up our world. One of those building blocks is access to useful information about the physical world.

Specialized sensing and inspection equipment can be expensive and difficult for everyday people to access. At the same time, almost everyone already carries a device with a speaker and microphone.

That led us to a simple question:

What if a phone could be used to sense changes in the physical world instead of only showing us information about it?

That question became AURA.

What AURA Does

AURA turns a smartphone into a simple acoustic sensing tool.

It sends a controlled sound sweep through the phone's speaker and records the response through its microphone. First, it learns what a normal object sounds like and creates an acoustic fingerprint. When the object is scanned again, AURA compares the new response with that fingerprint.

If the response changes significantly, AURA reports an acoustic deviation.

The important part is that AURA does not need to know exactly what the object is or what changed. It simply detects when something no longer sounds like its previous baseline.

How We Built It

We built AURA as a browser-based application using Next.js and TypeScript.

The audio pipeline uses the Web Audio API and AudioWorklet to capture microphone data. We generate a controlled high-frequency sweep, process the captured signal, and use FFT-based frequency analysis to create the acoustic fingerprint.

During calibration, several measurements are collected and averaged to reduce noise. During scanning, the new frequency response is compared against the stored baseline using Mean Squared Error:

$$ MSE = \frac{1}{K}\sum_{k=1}^{K}(S_k-B_k)^2 $$

The result is converted into an easy-to-understand anomaly score.

We also built a scientific view that shows the actual measurements and frequency curves, so the user can see where the detected change comes from rather than simply being given a black-box result.

What We Learned

The biggest thing we learned was that building something that interacts with the physical world is very different from building a normal web application.

Browser audio hardware behaves differently across devices, microphones introduce noise, and high-frequency signals are not reproduced equally by every speaker. Because of this, we had to focus on relative measurements and calibration rather than assuming that one fixed threshold would work everywhere.

We also learned how powerful the hardware people already carry can be when software finds a new way to use it.

Challenges

The biggest challenge was making the system work with real microphone data rather than simulated numbers.

We had to deal with browser permissions, audio processing, microphone noise, different sampling rates, and keeping the interface responsive while processing audio.

Another challenge was making the technology understandable. The underlying signal processing is fairly technical, but the user should not need to understand FFTs or acoustic transfer functions to use AURA.

That is why the main interface is intentionally simple:

Calibrate → Scan → See the change.