Inspiration
Our inspiration for this came from our members reminiscing on how much time they spent on their computers. So we developed a student version to remind students to take breaks. The educator version was inspired by our fascination with computer vision.
What it does
This provides a student and educator program, which aid with learning. The student program monitors the student and reminds them to take breaks or wake up depending on their facial expressions. The educator version allows presenters to swipe to move during presentations and to share files during it as well.
The file sharing functionality is done by pinching to enter file select mode, then up and down hand movements to select a file, and then pinching again to prepare to send. After choosing a file to send, moving your hand up sends it all users connected to your network, and moving down cancels the action.
How we built it
Student Version: We used mediapipe for face recognition and implemented a timer that detects how long a student has been present at their computer. Then, we utilized the os library to determine whether the student is running a windows or linux computer. Our pop ups were created using tkinter and opencv2.
Educator Version: We used mediapipe to enable hand recognition and facial recognition. We then used sklearn and machine learning techniques to train our program to recognize left and right hand swipes. For pinching gestures we used geometric data received from mediapipe to detect a pinch. Up and down motions were implemented with geometric data as well, tracking the general displacement of the hand over time.
Challenges we ran into
Student Version: We first ran into issues when we had to download the mediapipe and opencv2 libraries. With these packages, older versions of python and other packages (like numpy and tensorflow) had to be utilized so that all packages could work on the same venv. After we had the face recognition pop up window, we realized that Unix and Windows operating systems were not both compatible with tkinter, and the way the camera was chosen for Macs utilized a certain api from Apple. When looking at how test participants were seen falling asleep, the threshold had to be lowered so that the computer would be less sensitive to users squinting.
Educator Version: We had troubles with detecting gestures using ML, the general issue being the amount of data it needed to learn all possible left and right swipes. We ended up deciding with general gestures to follow to reduce the amount of frame data we would need to feed it.
Accomplishments that we're proud of
During this Hackaton, we created a gesture tracker, that can tell apart when a user is doing a hand swipe and not doing one. The left and right swipe recognition is very accurate for the amount of data we got during the duration of this. We also created a fully functioning file dropper that can send decently large files efficiently.
What we learned
During this time, we learned how to work with computer vision and media pipe. Along with this, we learned the importance of environment management, as we had to often switch environments to work with old and incompatible interfaces across our programs.
What's next for J_Viision
Our next goals are to expand to real world classrooms by improving user hand recognition. We also want to aim this toward a business setting where users will need to present and transfer data between many members at once.
Log in or sign up for Devpost to join the conversation.