We got inspiration for the Glove as a way to create and use multiple personalized interfaces within our VR app. The applications of the Glove are not limited to one sector or app, but rather it can be utilized for a variety of fields, ranging from video games to agriculture, to even potentially be used for medical procedures. The potential of the Glove is limitless.
What it does
The Glove sends hand motions into Unity, where it is mapped onto a 3D model of the hand, and thereby giving a precise and accurate representation of the human hand in a 3-dimensional environment. This plays into the versatility of the Glove because the applications for a "virtual" hand are "virtually limitless"! We accomplished this by using an accelerometer to track the motion and velocity of the hand, a gyroscope for tracking the rotation of the hand, and a magnetometer to track the direction in which the hand is pointing. By combining data from these three sensors and putting it through a filter, we are able to track the movements, tilts, and changes in the position of the hand in real-life and transfer into a model in Unity.
How we built it
Over the course of this Hackathon, we worked on the integration of the Glove with external applications such as the drone. By utilizing the 3D movement-tracking provided by the IMU, we were able to convert the movements of the hand into controls understandable commands for roll, yaw, pitch, and elevation for the drone. This was one of the most difficult challenges of this Hackathon as we needed to develop a PID: a proportional-integral-derivative controller. Through the magic of calculus and statistics, we were able to use the PID to continuously calculate error values and desired setpoints to determine the margin for differences in desired and actual values. A combination with a Kalman filter to determine standard deviations of data entry and we get a stable signal. The result: a continuous flow of live data with little to no extra noise to interfere with usable movement tracking. In addition, we needed to find a workaround to trace packets to and from the drone through the drone controller. In the end, we were unable to successfully and reliably intercept packets, forcing us to look for other means. We began to develop a solution with our own drone, aiming to develop computer controls through Unity and VR. Unfortunately, in the testing process throughout the night, we had fried the Raspberry Pi that we tested with, severely handicapping our ability to fully develop the product.
Challenges we ran into
Throughout the course of the Hackathon, we had to continuously adapt and improvise according to the limitations of hacking the drone and packet tracing with up to three different iterations of the drone-control design. Ultimately, we needed to develop our own drone to prevent problems between computer-drone signals. This, however, came with its own problems as we needed to build every part of our drone's hardware while developing the software for the gyroscope that would keep it stable and the camera that would enable VR capabilities. In the end, during the testing process, we fried the Raspberry Pi that would send and receive signals to and from the computer controlling it through Unity, hampering our ability to fully develop the drone.
Accomplishments that we're proud of
We are proud of the progress we made for the Glove and the applications that go with it. The experience we gained through working with a Kalman filter and the PID gave valuable experience with hardware-software integration and provided the opportunity for many of our team members to learn about the basics of electrical engineering. Though we had lost most of our progress made through the night after out Raspberry Pi fried, the experiences that we gained from working on the Glove's software, hardware, and electrical system are pieces of knowledge we will all take on to use in our futures.
What we learned
We learned a lot about integration on both the software and hardware end of product development. Our software team was constantly working to develop a working system of sensors to control and fly the drone stably through the use of Unity. Our hardware team was able to learn more about electrical system design and work (i.e. soldering, crimping, and testing).
What's next for The Glove
In the future, we hope to develop the Glove further with more compatible applications while refining the software to create a more polished user experience. Specifically, our goals with the integration with the drone are to transfer our progress to a new Raspberry Pi and create a working drone to be controlled with our Unity program and VR controls.