We wanted to do a fun and lighthearted project for LAHacks, so we decided to make an application that ostensibly measures how much a person can lift based on frames captured from a webcam.
How it works
The application uses OpenCV to perform SURF feature extraction on images from the webcam. These features are then matched against features extracted from several pictures of Arnold Schwarzenegger. These matches are then used to create a projection matrix from the reference image to the scene. We then analyze the projection matrix to establish a confidence in the match using a custom algorithm developed at LAHacks. If confidence is established, the prominence of the reference image in the scene is used to "score" the lifting capability of the scene. Notably, this algorithm doesn't actually solve the problem of determining a user's strength. It is, however, a fun and entertaining program to use with friends.
Challenges We ran into
We ran into performance problems from the start. As we began to tune the OpenCV functions and increase the performance, we added more reference images to match against. With these additional reference images, the potential for false positives increased dramatically. Another surprising difficulty we encountered was that several standard methods for estimating confidence, such as the relative difference of the point descriptors, were ineffective in this context. We developed a novel algorithm to reject false positives in response to this.
Accomplishments that We're proud of
The most interesting thing we accomplished at LAHacks was creating an effective and computationally efficient way to reject false positive matches. After previous methods failed, we explored a variety of heuristics and methods for reducing the noise. As we were looking for heuristics to reject noise, we looked at many things. What ended up being the most effective was examining the eigenvalues of the projection matrix. The eigenvalues were typically one large positive value, its negative and a small value, typically in the range of 0 to 1. We noticed that whenever we found a good match, this small eigenvalue was very close to 0.5. When a false positive is found, it typically varies wildly. We're still not exactly sure why this is the case, but it does function as a good heuristic.
What We learned
We obviously learned a lot about computer vision, but we also learned the value of experimentation in solving and understanding technical problems. If we had approached the problem from a exclusively theoretical point of view, we never would have found the eigenvalue heuristic.
What's next for Project Lift
We'd really love to get this application on iOS and Android, but due to the computationally intensive nature of computer vision, we're not sure how successful that will be.