We wanted to build a piece of software that was as interesting as it was useful. We began to think about accessibility and realized that a user interface with inputs completely determined by the user would be a novel system as well as one that is helpful to those who may be unable to interface with tech via traditional means.
What it does
In essence, our project utilizes a convolutional neural network in order to decode gestures defined by the user. This allows for an individual to create custom controls to interface with technology of their choosing (We plan to develop various output modules which allow for seamless integration with various services). Our clever network design, data pre-processing, and guided training procedures allow for training to be run on low-end machines (Again, we plan to offload this training onto the cloud in the future). At the moment, we've set up the system not as an interface with any specific software, but as a demonstration of its ability to differentiate gestures. By virtue of that, we hope that its usability as a universal user interface is acknowledged.
In our application, the user starts at a simple menu and can choose to define a new set of gestures or test recognition of their existing gestures. *If they choose to define new gestures, they will be asked how many gestures they want as well as what they should be called. The system will then observe their hand motions through a connected camera, training itself to recognize each one. *If the user decides to test their current gestures, they will arrive at a screen that shows them the camera feed as well as an indication of which gesture it has identified their input as.
How we built it
We built nearly the entire program in Python, using various libraries that were necessary along the way. In order to generate usable data, we built a process that takes raw image data and converts it to a more useful form. This includes automatically cropping, gray-scaling, and resizing the images to lower resolutions. We then designed a convolutional neural-network which is able to classify those processed images. The functionality of the whole system is then tied to the native python GUI that we built for ease of use.
Challenges we ran into
Just as with any development, we encountered a few hurdles along our path. Although we established a clear plan before beginning our work, we often had to change course when our initial ideas were leading us in the wrong direction. Addressing this involved lots of re-working and further planning. Additionally, some of our prototype ideas had difficulty meshing with our final development environment, not to mention the struggle of keeping said environment consistent amongst our team members.
The most obvious challenge for our project was of course the time constraints. As a team of fairly ambitious individuals, we regularly had to decompose our ideas into more simplistic and easily achievable tasks. While this might seem like a downside at first, it actually proved beneficial as we were to make mild, yet consistent progress.
Accomplishments that we're proud of
This project was quite ambitious and difficult to execute. Given our limited amount of time, we're with how much we've been able to complete. We also had a great time!
What we learned
One of the biggest things we've learned was how to properly coordinate a team, as well as how to mesh our work together in a meaningful manner. We also learned that a proper planning phase not only considers the viability of a single approach, but rather a broad analysis of all available avenues and their feasibility.
What's next for U.G.I.
We have plans to expand this project into a much more comprehensive and complete product. Such plans include developing multiple input and output modules which would allow for users to further refine U.G.I. to suit their needs. Ideally we would like available input methods to range from voice, and even EEG. Such a robust system would allow for individuals from all over the globe to be able to interface with the world of technology regardless of their ability.