Frame

home page

Inspiration

The best photographers aren't born. The rest of us just never got better.

Our phones carry cameras more capable than the ones that produced the most celebrated images of the last century. But most of what we shoot disappears into a forgotten iCloud folder before it's even seen.

We built Frame because we found that no amount of post-processing can remedy a poorly composed photo. Every moment is temporary and we only have one shot to capture it. We combine a personal multimodal photography teacher with a open social ecosystem to challenge you daily and help you learn from the best.

What Frame does

Gives real-time composition feedback while you frame, including score updates and directional suggestions.
Runs edge AI vision model pipelines to help you find better framing for selected subjects.
Adds live coaching overlays for blur, shake, level, and exposure to improve technical quality before capture.
Scores photos on various handpicked metrics like aesthetics, composition and more.
Includes a challenge and social layer with global active/upcoming challenges, submissions, podium stats, and the ability to follow admired photographers.

How we built it

Mobile app (Expo + React Native): Vision Camera powers live capture, frame processing, and camera controls.
On device intelligence: iPhone captures frames and runs sensor coaching on-device through CoreML; a local Python server handles all ML inference over a USB-tunneled HTTP connection
Serialized GPU pipeline: Serialized PyTorch models through a threading lock to prevent Metal command queue deadlocks
Image and coaching pipeline: EXIF-aware decode, blur rejection, sensor fusion (motion/orientation), exposure analysis, and visual overlays.
JS thread budget management: Sensor hooks (level, shake, exposure) fire at 20-30Hz but only trigger React re-renders on state transitions to avoid saturating the native bridge.
Social network: Supabase-backed auth and cron competition scheduling, challenge submissions, and social interactions.

Challenges we ran into

iOS version/device compatibility: behaviour wasn’t always consistent across iOS versions and physical devices, especially around camera permissions, build tooling, and native module behavior.
Swift + native plugin learning curve: we had to quickly ramp up on Swift/Objective-C interop to connect CoreML inference to the React Native camera pipeline.
Expo + native integration friction: combining an Expo-managed app with custom iOS/CoreML plugins required careful config/plugin setup and repeated iteration to keep builds stable.
Real-time performance constraints: we had to balance model quality with latency so scoring felt instant and didn’t block the camera experience.
Sensor and networking overload: Handling real-time streams from multiple sensors (gyroscope, accelerometer, exposure, etc.) while sending frequent HTTP requests for inference was a major challenge. It was tough to do all this in a single-threaded React Native environment, without overwhelming the JS thread or starving network requests.
Paper inconsistencies: We discovered inconsistencies in a prior paper related to composition scoring. To address this, we obtained the original dataset used in their work and built our own training pipeline from scratch, developing a model better suited to our specific task and requirements.

Accomplishments that we're proud of

Built a working end-to-end system that combines on-device detection + server-side aesthetic scoring.
Shipped a practical coaching experience that helps users improve composition and shot stability in real time.
Implemented a complete challenge loop (discover → submit → browse) with social engagement features.

What we learned

Art is really hard to quantify. We're still bad at it, and models aren't perfect.
UX for AI camera tools depends as much on feedback timing and clarity as model accuracy.
Hybrid architecture (mobile + local server) is powerful, but requires disciplined handling of networking, orientation, and caching.
Instrumentation and debug artifacts are essential for diagnosing vision/model issues quickly.

What's next for Frame

Bring more scoring and guidance fully on-device to reduce dependency on a local server. (quantize?)
Activate richer composition explainability (pattern/attribute-level coaching) in the user-facing flow.
Expand challenge mechanics with better ranking, curation, and progression.
Improve personalization so coaching adapts to a user’s style and historical submissions.
Continue optimizing inference/runtime performance for smoother real-time feedback.

isaac