Reframe

Inspiration

Every AR furniture app can put a virtual couch in your room. Not one of them removes the couch that's already there, and that's the whole problem. You can't evaluate a new armchair when the old one is still sitting in the shot. Compositing a posed mesh over the feed is nothing, we've all done it. The hard part is the inverse: reconstructing the floor, wall, and skirting behind an object that is physically present and occluding those exact pixels in every frame you receive. We wanted to hold up a phone, point it at a chair, say "replace this with something warmer and red," and have it resolve on a live feed we never touch.

What it does

Reframe is a live diminished/augmented reality room editor for iPhone. Place a catalog asset on the floor, swap a real object for one whose projected silhouette covers it, pull a real object out and reveal reconstructed background, undo instantly, offline included. Retrieval sits on a Qdrant vector DB with 20k+ furniture items, content-addressed. When you say "warmer and red" we embed it and run the ANN query under a hard pre-filter on the target region's measured footprint and clearance, so nothing that can't physically fit ever comes back, and we re-rank whatever survives on fit residual. Voice (GPT Realtime 2.1 over WebRTC), reticle, and tap all lower to the same intent struct and hit one resolver, because maintaining three of anything is how you get three sets of bugs. Sessions serialize to a durable .rfcap file so you can replay and re-edit from a Next.js web twin. Readiness is per-capability and we don't lie about it: replace lights up the moment a coarse proxy exists, remove holds until reveal coverage is good enough, and there's a visible "Healing" state while inpainting is still converging.

How we built it

The rule the whole thing hangs on: models are advisory, deterministic code owns identity, geometry, revision, commit, restore. GPT-5.6 Sol plans through five read-only tools and hands back a proposed edit. It cannot commit anything. A deterministic validator does that, and only that. It runs collision and clearance in the metric scene frame, checks the support relation against the detected supporting plane, and computes a rasterized cover score by projecting the candidate mesh under the current pose against the target mask, \( c = |S_{\text{new}} \cap S_{\text{target}}| / |S_{\text{target}}| \). The edit lands only if \( c \geq \tau \). Commit is one compare-and-swap on a monotonic revision counter behind an idempotency key, so a stale or replayed proposal is a no-op instead of a corruption. Nothing a model returns ever writes scene state.

Geometry runs on two tracks, and the split wasn't a design choice so much as something the latency forced on us. First-edit time cannot wait on dense reconstruction. So the fast track fuses ARKit VIO poses and plane anchors with a SAM 3.1 video track, carving the mask sequence into a voxel occupancy volume that we deliberately dilate rather than erode. That gives you an editable proxy in a couple of seconds that is guaranteed to over-cover the real object, which is exactly the failure direction you want. The dense track runs DA3 monocular metric depth and solves a per-frame affine alignment \( \hat{d} = a\,d_{\text{pred}} + b \) against sparse VIO depth with robust least squares, throwing out any frame whose residual is too high, then fuses the accepted frames into an Open3D CUDA TSDF with truncation a few voxels wide. Occlusion masks and metric extents tighten asynchronously as the SDF firms up. Reveal texturing follows a strict precedence and we never break it: reprojected observed pixels from other views first, deterministic planar/homography fills second, LaMa only for texels no geometry can account for. A synthesized texel is never permitted to paint over an observed one.

On device it's SwiftUI, ARKit, RealityKit, with a Metal compositing fallback for the diminished-reality passes RealityKit won't express without a fight. The gateway is Bun and TypeScript over SQLite in WAL mode, fronting an in-process GPU lane scheduler. Vision workers are Python 3.12 in isolated Docker containers behind that scheduler. One JSON Schema contract defines the binary FramePacket and every message on the wire, and we implemented it three times, Swift, TypeScript, Python, so the bytes mean the same thing everywhere. Sol was good at taking loose constraints and turning them into schema-ready field definitions and leaning on the contract boundaries until they held. Codex wrote a large chunk of the cross-runtime implementation and its tests. Four people don't get this far in a week without both.

Challenges we ran into

Base iPhone 17, no LiDAR. Every metric millimetre is VIO plus learned depth, and learned depth is affine-ambiguous in scale, full stop. So the alignment stage will drop a frame before it fuses a plausible-but-wrong one, because a single bad scale estimate doesn't degrade the TSDF locally, it contaminates it globally. The thing that ate the most hours, embarrassingly, was coordinate conventions. ARKit is right-handed looking down \( -Z \), OpenCV looks down \( +Z \), and the flip \( \text{diag}(1,-1,-1) \) is its own inverse. Apply it an even number of times and everything looks almost right, which is the worst kind of wrong because it survives eyeballing. We now carry the handedness tag next to every matrix and gate it in CI with a known-ray reprojection test that fails on any parity mismatch. Keeping the first edit off the serial depth-to-TSDF-to-mesh chain is what forced the two-track design in the first place. And one GPU multiplexing segmentation, depth, fusion, reveal, and naming will absolutely starve itself if you let it, so there's an explicit priority-lane coordinator with preemption in front of it that keeps a reveal job from ever blocking the interactive segmentation lane. Reveal isn't instant and we refused to pretend otherwise, so we pinned it to real gates: \( p_{10} \) and median reveal coverage across eight sampled viewpoints.

Accomplishments that we're proud of

The deterministic core is tested code, not a slide. The binary FramePacket contract, the coordinate conventions, the CAS transaction and undo model with idempotency keys, the cover-score math with its exact thresholds, crash-safe .rfcap capture on device, HMAC room-scoped session auth, the GPU lane coordinator, the 20k+ item catalog pipeline. Unit-tested across all three runtimes. And the invariants actually held: the camera feed is never punched per pixel, a model never writes scene state, synthesized texels never overwrite observed ones, and the phone never claims a capability is ready before the artifact behind it is verified and live.

What we learned

Put every model behind a typed provider boundary. You will change your mind about which model you're using. You will not get to change your mind about what identity means, so don't couple them. Readiness is per-capability, "editable" was never one bit, and people trust the system more when replace unlocks before remove and the UI just tells them why. A dilated mask is invisible, a leaked chair leg is not, so conservative geometry wins over clever geometry every single time. Contracts-first is the only reason four people and two AI collaborators could work across three languages without the implementations quietly drifting apart. And in a fusion pipeline a dropped frame costs you nothing, you get the next one, but a confidently wrong frame poisons everything downstream of it.

What's next for Reframe

Closing the gates we set. A full replace on a live SAM 3.1 track, end to end. LaMa and the CUDA TSDF running in production at a budgeted cadence. Live voice turns and Qdrant retrieval over the full 20k+ catalog under real session load. FPS, thermal, and visual-quality validation on actual hardware, not a bench. After that, catalog ingestion at IKEA scale, the Mode B1 photoreal Gaussian-splat twin in the web viewer, tabletop and shelf support relations, multi-object scenes, and eventually multi-room capture. None of it worth giving up an invariant to get.