Spatial Synth

Inspiration

Spatial Synth turns the empty space in front of an iPhone into a musical instrument.

I wanted to make electronic music feel physical without adding a controller. AIR mode follows a hand and fingertip in three dimensions: horizontal movement changes pitch, vertical movement shapes the filter, depth changes the sound, and a pinch becomes Note On / Off. MASK mode uses the position and movement of the face, mouth, smile, eyebrows, cheeks, and head tilt to shape sound. The camera image is replaced by an abstract live mask.

What it does

The app includes FLOW and PULSE performance modes, arpeggiator patterns, scales and keys, 11 style presets, BPM sync, motion-triggered ARP FILL, depth-triggered PITCH FREEZE, and a four-second gesture loop. AIR and MASK never run their camera sessions at the same time. Losing tracking, switching modes, or leaving the app stops every active note so the instrument cannot remain stuck playing.

Privacy is part of the instrument design. Spatial Synth does not use Face ID enrollment data, and it does not save or send camera images, depth maps, face meshes, or raw expression values. Only normalized face calibration values selected by the user stay on the device.

How I built it

I collaborated with Codex throughout the Build Week extension. GPT-5.6 inspected the inherited Swift project, kept sensor-specific code behind a shared performance frame, fixed camera and Note Off lifecycle risks, implemented the gesture tools and musical styles in small commits, and repeatedly checked the project with the iPhone SDK, strict concurrency settings, static analysis, signed builds, installation, and launch tests.

I chose the product direction, interaction mappings, musical styles, privacy boundary, and safety priorities. The app was built, installed, launched, and given basic interaction checks on an iPhone 13; final listening and endurance tuning remains hands-on work.

Build Week work

The pre-existing v0.2 baseline is preserved as commit 79bb1fe. The later dated commits document the functionality added during the Build Week submission period: safer camera lifecycle, stable Note Off behavior, FLOW / PULSE, gesture tools, 11 AIR / MASK styles, and BPM sync.

What's next

The next steps are hands-on tuning of sensitivity and mappings, longer musical loops, Core MIDI output, preset saving, and recording/export.

Built With

• arkit
• avfaudio
• avfoundation
• codex
• gpt-5.6
• swift
• swiftui
• vision