Inspiration
Us developers have experienced firsthand the joy of customizing your workspace, and configuring it to your liking. I myself like using Neovim and Amethyst (tilling window manager for Mac) and one thing I've noticed a lot is that when I am using Amethyst, I don't switch virtual desktops. Amethyst has a custom screen implementation that doesn't actually use the virtual desktop API provided by macOS. The inspiration of this project comes from boosting developer productivity by introducing a new method to interact with your applications, open window windows.
What it does
Pawductivity changes how you work with your computer, it allows, maximum gesture customization, unlimited gestures, notifications, a gesture editor, and a neat handy Menu Bar Icon to access the applications settings.
How we built it
We built Pawductivity by utilizing the private API's macOS provides developers to call user and system events. We used Apple's hand landmark detection (VNHumanHandPoseObservation) found here for identifying the hand wrist, joints, and finger tips. We also used a custom Neural Net to process the data points for both hands and try to predict the movement a user did.
Challenges we ran into
We ran into a lot of problems and challenges when creating this application, one of the big ones would be differentiating between one hand, gestures, and two hand gestures. Because there is two models that Apple provide provides (one of them being for one-handed gestures and the other being for two handed) so we had to work outside the box and combine these data points into specific dimensional arrays that can be used by the model.
Accomplishments that we're proud of
Our greatest accomplishment would be having the Gestures actually do an action, this application is meant to be a prototype of something bigger, something larger, something even more innovative than what the judges will see this Sunday, but doing a gesture and watching it do an action on your laptop that can boost your productivity is the right step in the correct direction for a revolutionary product like Pawductivity
What we learned
A lot about the macOS ecosystem, and a lot more about the DispatchQueue. We didn't settle on our project idea until Saturday afternoon and didn't start coding it till six in the evening, so we definitely learned a lot about managing our time! We also learned about hand landmark points, and how they function in classification models.
What's next for Pawductivity
The future for Pawductivity is a bright one, me and my partner will be developing this application further and refining, the gesture, detection, and training procedure, to hopefully put this out on the App Store and create a product that not only I and my partner will be using, but potentially the whole world!
Log in or sign up for Devpost to join the conversation.