Sonic Assist

Inspiration: Working at an MSP (a company that runs IT for other businesses), I once had to talk a business manager through a network failure over the phone. I could see the problem but couldn’t point at anything. We built a way for the expert to point at the exact part through the other person’s phone.

What it does:

An expert guides someone through a hardware fix. In our demo, that’s swapping the SSD in a Dell laptop.
Live help: the expert clicks on the phone’s live video and a mark appears on the real hardware.
3D guides: animated steps shown over the real laptop. The user taps Next to move on.
Works offline: the phone keeps the guide if the Wi-Fi drops and syncs when it’s back. Checks the work: a camera rig confirms the SSD is out or back in.

How we built it:

iPhone app (Salman): Swift with ARKit and RealityKit, aligned by a printed 180 mm marker. The live call uses WebRTC, and ElevenLabs reads steps aloud.
Dashboard (Max, Laura): React and TypeScript, with three.js and Spark to show Scaniverse scans. Max built the 3D view and calibration. Laura built the guide builder, the marker and the 3D models.
Relay (Ahmed): Node and web sockets over a phone hotspot. Every part uses one shared TypeScript message format.
Rig (Ahmed): an Arduino UNO Q with a Nicla Vision camera that spots a sticker on the SSD.
AI props (Max): Pixel3D on our DGX Spark turns one photo into a textured 3D model in a few minutes.

Challenges we ran into:

AR is mostly coordinate frames. Units, axes and “where is zero” caused more bugs than anything else.
Our phones have no LiDAR, so we align with a printed marker. Our first scan missed it and we had to rescan.
Guide marks landed across the room. Reading the positions straight from the relay showed the dashboard was right, so the bug was on the phone.

Accomplishments that we’re proud of:

Four people built four parts in parallel.
A guide made in a browser shows up as 3D steps on a real iPhone.
We ran an image-to-3D model on our own hardware.

What we learned:

Tech skills: IOS, X code, digital twin replica, AR live cam, live mic, live interaction.
Using Arduino hardware and different sensor modules, we used Necla vision for CV and then used the Uno Q to act as a hub for our Necla vision.
We spent two hours trying to debug why our code was crashing the phone camera only to find out it was a UI element.
Apparently you can restrict an AI tool by country. Hunyuan3D’s licence excludes the UK, so read licences first.

What’s next for Sonic Assist:

Cut-outs: lift the real screw out of the scan and animate it unscrewing.
Guides for more devices.
NFC tags that turn the physical location of a phone into co-ordinates in 3D space.