As we slowly start to make our way back into in-person learning, we enter an awkward phase of trying to teach some students in the classroom and some virtually. In our experience, instructors are (understandably) eager to get back writing on the whiteboard in front of a class full of students.
This doesn't translate well to the Zoom students at home.
We imagine an affordable smartboard that can digitize the instructors annotations on the board and make sure that the students at home can read it. Preferably, the board would be a robust, plug-and-play unit that takes the pressure off of instructors to teach two classes, in-person and virtual, at once.
What it does
G.R.I.P. Board is a Grasp Responsive Interactive Projector Board that is able to digitally draw on the projector using a simple glove. The glove is extremely simple to use. Pointing with your index finger turns on the red light, and draws on the screen. Pointing with your thumb turns on the green light, and clears the screen.
That's all the user has to know! And with soft, conductive knit pads on the glove, it is comfortable without the bulk of make other human-computer interface devices. As an added bonus, it is powered by a single watch battery (and we still haven't found the limit of the number of hours it will run for)!
Our standalone camera/projector system watches for the LEDs on the glove to activate, and captures their coordinates and color appropriately. With a little extra processing, the projector draws at that location, making it feel like you are drawing on the board with your finger!
How we built it
This project has two build components, the glove and the base.
- The glove is constructed with LEDs connected by wires routed on the outside of a standard commercial glove. Conductive pads are stitched into the glove at the index and middle fingers and make contact with the palm during different gestures. Pointing the index finger naturally completes the circuit through the conductive pads to a watch battery in the wrist strap. All of the components were first soldered together, and then stitched to the outside of the glove.
- The base of the G.R.I.P. Board is comprised of a projector, webcam, and a Raspberry Pi 4 to do all of our computing. The base simply needs to be plugged into any wall outlet and it will take care of the rest. All of the computer vision and post-processing is done onboard and displayed via HDMI to the projector. From a software standpoint, the webcam captures a triggered LED, OpenCV, after applying various camera calibration transformations, determines the LED coordinates and color, which are sent to a PyGame instance which draws these coordinates back onto the projected display.
Challenges we ran into
So many. First off, our favorite camera broke and luckily we brought a spare webcam just in case. Calibrating the camera is extremely important, because the display surface is never optimal and your internal parameters vary wildly. We tried to get a reliable, automated calibration system to run every time we moved the G.R.I.P. Board to a new display surface, but, in the end, had to give up due to time. Our projector is also pretty weak, so we had a lot of trouble finding a suitable place to project to without getting washed out in the lights. Also, we were unfamiliar with running a raspberry pi headlessly, so we were often tweaking and developing our program using our projector or using Git to synchronize our program developed in our own computers.
Accomplishments that we're proud of
The secret sauce for reliable LED detection is our custom camera filter that we crafted from electrical shielding bags. It is able to darken the incoming light enough to not max out the webcam sensor, and also allows for us to project the same color as the glove (red). Our camera + filter can only detect light emitting sources, not simply reflected light.
What we learned
Camera calibration is extremely hard, and necessary! We also wish we had more software experience, poor Dylan really lead the charge on so much of this project.
What's next for G.R.I.P. Board
We think it could be useful for a lot more than just teaching! It's a portable smartboard at less than 5% of the cost (seriously, most smartboards are $3000+). Imagine business meetings, fieldwork, military, you name it.
On a technical level, we would love to get it working with automated calibration! Other cool features would be PowerPoint integration for annotating slides, and a simple way to stream your board online without Zoom. Additionally, from a hardware standpoint, there could be some hardware improvements such as a faster processor, a better quality camera, stronger LEDs on the gloves that are less directionally dependent. A better projector would also help.