Kinetix

Inspiration

Apple built the iPhone Duo's outer display so the person being photographed can see themselves. Duo Preview. It is a mirror.

A mirror does not tell you anything. If you are copying someone's movement, seeing yourself does not tell you which joint is wrong or how far behind you are. And the person teaching you cannot see what you see.

"Watch me, now you try" is how every physical skill on earth gets taught. Nobody has ever measured what happens in between, because you need a camera pointing in both directions at the same time. Until this month, no phone had one.

What it does

Two people stand facing each other with the phone between them. One demonstrates a movement, the other copies it. Both are tracked at once.

The demonstrator's display shows how closely the copy matches, joint by joint, in degrees. It names the worst limb and shows how far behind the learner is in seconds.

The learner's display, facing the other way, shows their own body with the demonstrator's pose ghosted over it and a single cue. No numbers. They cannot see the demonstrator's screen and they do not need to.

The hinge scrubs both clips together. Fold the phone to step through the movement and stop on the exact frame where the form breaks.

The same engine also runs single person assessments as profiles: squat form, with knee angle, depth, valgus detection and rep counting, and facial symmetry for post stroke or Bell's palsy retraining, with baseline calibration and per side deviation. Each profile is a small set of rules about what to measure and what to show. Swap the profile and you have a different vertical.

How we built it

Vision's VNDetectHumanBodyPoseRequest gives 15 tracked joints per frame. The comparison layer is the actual work:

Angles, not positions. Two different people at two different distances have different body proportions, so joint coordinates are meaningless to compare. Everything is normalized to the hip midpoint, scaled by torso length, and reduced to nine limb angles:

$$\theta_i = \angle\left(\vec{b} - \vec{a},\quad \vec{c} - \vec{b}\right)$$

for each tracked limb \( i \), where \( \vec{a}, \vec{b}, \vec{c} \) are that limb's three joint positions after hip-midpoint normalization and torso-length scaling. An elbow angle is the same whether you are tall or close to the lens.

Mirroring. Two people facing each other are mirrored, so the demonstrator's left limbs compare against the learner's right. Get that backwards and every frame reads as maximum error, which looks exactly like a broken app.

Lag search. The learner is always behind. Comparing frame \( N \) to frame \( N \) reports constant error even on a perfect copy. For each learner frame we search the last second of the demonstrator's motion for the best match:

$$\tau^\star = \underset{0 \le \tau \le 1\text{s}}{\arg\min} \left\lVert \theta^{\text{learner}}(N) - \theta^{\text{demo}}(N - \tau) \right\rVert$$

and the offset \( \tau^\star \) of that best match is itself the useful number.

The outer display is driven through a SwiftUI CameraCaptureAccessory inside .sceneAccessory:

.sceneAccessory(.camera) {
    CameraCaptureAccessory { /* outer-display content */ }
}

The inner display splits across the fold with ArrangementView and reservedRegions(kind: .division). Hinge angle comes from onHingeChange.

Challenges

VNDetectHumanBodyPoseRequest does not work in the iOS simulator at all. It fails to initialize with the following, and no frame delivery approach fixes it:

Vision error 9: "unable to setup request"

We lost an hour finding this because the failure is silent: frames decode fine, Vision just returns nothing.

The same request on macOS detected a body in 20 of 21 frames of the same footage in 0.7 seconds. So we wrote a small macOS command line tool that runs the identical request over each clip and writes a pose sidecar as JSON, which the app loads instead of running Vision at runtime. On real hardware it runs live from the cameras.

The fold on the inner display runs vertically, a 40pt band down the middle, active only in book pose. So "above and below the fold" is really left and right, and .split.axes(.vertical) silently hides the secondary pane with no error.

The scene accessory presents on pose alone in the simulator. Apple's documentation and the Group Labs session both state it requires an active rear camera capture session. With no capture session in the process, it was unavailable in closed pose and available in open pose, and a --display=primary screenshot captured real rendered SwiftUI off the outer framebuffer.

Accomplishments

Three findings about the iPhone Duo platform that are not in any documentation: the accessory presenting without a capture session, the vertical fold breaking a vertical split, and Vision body pose being dead in the simulator.

A comparison layer that is invariant to body proportion, distance from camera, and mirroring, with 62 unit tests covering all of it.

As far as we can find, no third party app has shipped or demoed anything using CameraCaptureAccessory yet. In six years of foldable phones, exactly one app has ever put different content in front of two people on opposite sides of one device, and it was text translation on the Pixel Fold.

What we learned

The outer display is the least explored surface on the device and the most tightly constrained. Everything else about the Duo is a bigger canvas. The outer display is the only part that creates a shape that did not exist before, and Apple's own framing of it, a mirror for the subject, is the smallest thing you can do with it.

What's next

Live capture from both cameras with AVCaptureMultiCamSession when hardware ships on October 23.

Reaction time between two people, which is the same two streams measured on the time axis instead of the joint axis. Measuring how fast one person reacts to another has always needed wired lab equipment.

Letting the expert describe the movement in plain language and having the app generate the measurement profile, instead of hand coding one per exercise.

Built With

• avfoundation
• ios-27.1
• iphone-duo-simulator
• swiftui
• vision
• xcode-27.1-beta