After working on an uninspiring twitter sentiment vs stock fluctuation study (it's the weekend...so no stock data), I just went project to project, trying to figure out what to build. I had initially come up with the idea of using machine learning to track what a professor is saying and how I wouldn't have to touch my laptop or anything, just listen. And it got me thinking, what if I can't touch my laptop? What if I had no control over it? Windows Hello is an amazing service but I feel like it's time to bring facial recognition to *nix!
What it does
The application activates on login and holds everything else up until the user is scanned and recognised by the system. Simple and clean
How I built it
I used Microsoft's Face API to identify the owner through machine learning. Alongside that, I ran some shell scripts that activated on login and threw the system in single-application mode. Then, while it was in single user mode, I disabled the keyboard and mouse by unloading kexts. I used Imagesnap to capture images at a timed interval to funnel into Face API's identification engine. And, finally, I called Photo Booth as the single foreground application to let the user that an interaction is needed (particularly indicating that a camera would be the key). The entire system would be held in single-application / kext-less mode until Face API is able to recognise the user
Challenges I ran into
My original plan was to implement this on Linux due to the straightforward nature of using pm-utils and dbus-monitoring to have the facial recognition activate before unlocking the device. But my laptop's webcam wouldn't passthrough onto my VM and I had no USB A-USB C adapter to try and dualboot so I had to temporarily abandon the idea. But, by that point, my heart was set on using Face API and was debating between four ideas: 1) Go back to my sentiment analysis and turn that in, despite my heart not really being in it at that point. 2) Design Tony Stark's helmet (maybe in the winter!). 3) Try and grab a Raspberry Pi with a camera from the hardware shop but spend one of my ten hours just figuring out my approach and it could still break at any given moment or 4) try to make facial recognition work on MacOS!
The project in itself had two difficult problems. Firstly, I couldn't connect to Face API since my was being rejected and almost made me consider jumping ship until a mentor and I realised that either the API had changed or something under the hood was changed. After experimenting and failing multiple times alongside three amazingly patient mentors, we had sorted out the computational aspect of the project
Secondly, and more conceptually challenging, was figuring out how to design the application actions in a thoughtful manner that not only achieved the proof-of-concept security aspects, but also engaged the user without compromising the functionality of the system. I initially thought about thought about the core function of the software: security. Since I couldn't replace the lock screen in OS X as easily as I theoretically could've in Linux, I had to design a system that activates after on login. Then, I tried to think how to make the user's data inaccessible to intruders so I initially showed the wallpaper with a "breathing" effect. With that came difficulties in UX since users wouldn't necessarily understand what would be expected of them, and thus revelation of using single application mode to open Photo Booth and lock up all functionality until Face API had decided to unlock the system.
Accomplishments that I'm proud of
I'm very happy that I kept working for the whole 36 hours and came up with a product that, at least in my head, would excite people and add a whole new dimension of what can be. I'm also pleased with my general conceptual design for this product and I feel like, at least in theory, it's a very well designed system, especially in 14 hours I had to work on this (along with the other technical setbacks). A lot of time and energy went into just planning and conceptualising this product and it's an overall personal pride for me to design something that I felt overwhelmingly exciting about and I hope everybody else is excited about too!
What I learned
Soylent is not for me. When I'm exceptionally exhausted (like I have been from working nonstop for hours on end), I forget what I'm doing and feel really tired. But, with those out of the way, I learned that I can do it. These past ten hours might have been filled with exhaustion and general "loopiness", but it's also been filled with inexplicable joy from knowing that I did it! I built an amazing product that I'd use every day, multiple times a day, and it'll still take my breath away
What's next for Azure-Based Facial Recognition on MacOS
I want to explore replacing the general lock screen and designing a more tried and tested solution. Along with that, I also want to design this system with local data, rather than external data calls. Then, finally, implement this on Windows (not all laptops have Hello), and Linux. I'm so excited for everyone to see it!