Maxwell

Inspiration

Reaching new heights can be really daunting. Instead of something that overwhelmed users with goals or change, we wanted to focus on the idea of familiar technologies. As we sat around the room brainstorming ways we could help people, we realized that growth isn't necessarily related to traditional productivity. Sometimes, it's just learning to appreciate yourself and your interests more. So, we decided to build Maxwell an app that guides users to take better photos with their mobile camera using prompts like move camera to the left and get closer to the subject.

What it does

Maxwell is an app that reflects your traditional camera app. The only difference is that, as you take photos, Maxwell gives live feedback on small changes you could make to improve your photo composition using Machine Learning. In addition, Maxwell also generates a photo quality score in the form of a bar at the bottom of user screens, enabling users to figure out if they're improving as they move around the space. And, to enable more learning, Maxwell includes a gallery that shows users a breakdown of their score and the criteria for a good aesthetic score.

How we built it

Maxwell runs on Expo with a React-Native front end and a Flask back end connected by a Swift module. Using the Apple Vision framework, we extracted features from photos such as contours, faces, objects, symmetry, rule of thirds, and salient heat maps. Then we used the AADB database, which is a collection of photos professionally rated on a set of aesthetic criteria to train our neural network by relating features to the criteria with OpenCV, training our model on existing data, and applying a regression on the live feedback from our camera integration.

Challenges we ran into

No one on our team had ever worked with an ML model, Computer Vision, or iOS Development. Additionally, we had very little full-stack experience aside from React, SQL, and Node. We struggled a lot with determining the direction we should take, like our software architecture, because we knew very little about our subject, which resulted in a switch from Expo Go to React Native, Google ViT to Apple Vision, and Expo Camera to React Native Vision Camera. Additionally, we almost gave up on our project while trying to integrate our Swift module into the codebase.

Accomplishments that we're proud of

Migrating between Expo Go and Expo (Apple Developer)
Implementing Swift
Understanding and implementing a Neural Network with an R^2 of 0.8
Live Photo Analysis
Implementing Heat Maps, Object Recognition, Facial Recognition, Text Recognition, Contour Extraction, Symmetry/Pattern Recognition

What we learned

Going forward, we feel a lot more comfortable with full-stack development and software architecture. We learned a lot about Computer Vision and training ML Models through our hours of research, and we now understand how to train a neural network from the ground up.

What's next for Maxwell

As we hacked, we realized our focus wasn't developing a perfect model during the Hackathon, but a working model. In the future, we'd like to train our model on more data, integrate other data sources, and export other features from Apple Vision. We'd also like to try using other CV frameworks.