We wanted a fun, lighthearted project for LAHacks, so we decided to make an application that would use computer vision to ostensibly tell you have much you lift.
How it works
We used OpenCV to capture frames from a webcam and perform SURF feature extraction. The extracted features are then compared to features extracted from pictures of Arnold Schwarzenegger with a FLANN based matching algorithm. The matches are then used to create a projection matrix from the keypoints in the reference image to the corresponding keypoints in the scene captured from the webcam. We then establish a confidence in this projection matrix based on an algorithm we developed experimentally. Once confidence has been established, we use the area occupied by the reference image in the scene to score the scene's lifting capabilities. Notably, it doesn't do what it advertises at all. But it does solve the problem of finding images of Arnold Schwarzenegger in frames captured by the webcam.
Challenges We ran into
We struggled with performance and matching accuracy from the start. As we improved performance by tuning the OpenCV functions, we added more reference images to search for. This caused the number of false positive matches to increase dramatically, so we were forced to develop a novel way to reject false positives in a computationally efficient manner.
Accomplishments that We're proud of
As mentioned previously, the false positive rejection algorithm is our most notable accomplishment. We spent an extended period of time searching for acceptable heuristics by which we could filter the images. We found several imperfect metrics by which to judge the match.
The most interesting of these is examining one of the eigenvalues of the projection matrix associated with a match. The eigenvalues of the projections tend to include a large positive value, that value's negative, then a value much closer to zero. We noticed that for the projection matrix associated with any true match, the small eigenvalue is extremely close to 0.5. It varies wildly the projections associated with false positives. We're not completely sure why this is true, and we did notice this metric is not translation or rotation invariant. It is, however, good enough to reject almost all false positives.
Another accomplishment we're proud of is how well we managed to balance performance and accuracy. We're currently matching against 16 reference images of Arnold Schwarzenegger, with relatively low rates of false positives and very reasonable latency times.
What We learned
We obviously learned a lot about computer vision but we also learned the value of exploring a problem experimentally. We never would have come up with the eigenvalue heuristic without staring at the projection matrices and their eigenvalues side-by-side for a while and trying to find patterns.
What's next for Project Lift
We would absolutely love to get this application working on Android and iOS, but because of the computationally-intensive nature of computer vision, we're unsure how effective that will be.