Fit & Frame

view your frames in your own room
upload photo and edit frame
icon

Note: The application is available via TestFlight, only. For access, please send me your email address.

Inspiration

A few months ago, my wife and I paid a lot of money for some custom picture frames. They came out looking great, thankfully, but what if I could have previewed them before buying? That was the inspiration for Fit & Frame.

What it does

Fit & Frame is an AR application which lets you customize your own picture frame, and preview a model of it in your own home.

How we built it

Mobile interface created using Expo/React Native. I'm mostly a frontend engineer with little experience developing mobile applications, so React Native is a life saver in this regard.

React Native is not typically suited to creating AR Applications, but using React Viro, an open-source AR/VR library, I was able to leverage some of the native APIs in a way that is more familiar to me as a frontend engineer.

Custom model generation using Three.js, hosted in the cloud using Netlify Functions.

Challenges we ran into

Generating a custom 3D model posed many challenges. It's a lot of complex mathematics. When you're dealing with angles and 3d space, the trigonometric functions are essential.

Finding the ground plane in an AR space was another challenge. Using React Viro which exposes some of the underlying ARKit/ARCore functionality, the developer gets a list of detected planes called anchors. However, it's difficult to be sure which one of those planes is likely to be the floor.

Finally, integration was a big challenge. I chose to do model generation on the server so it would be compatible with many platforms; Running that code in Netlify Functions required some frustrating changes. Viro refused to work with the latest versions of the Expo SDK. A lot of fiddling with libraries and frameworks to make them play nicely together.

What we learned

Programmatically generating 3d models is hard. You have to define the 3d position of each vertex, which vertices each triangle consists of (indices), which way the triangle is facing (normal vectors), then where a texture should be applied to each vertex (UV mapping). It's a complex, math-intensive process.

I learned how to use AR on mobile devices, all the concepts were totally new to me. I'll likely be making many more AR projects in the future.