Inspiration
We were inspired by the Apple pencil. While it does touch the screen, it provides a whole new experience to writing. We sought to raise this experience to the next level, without using anything but our hands! We see that our accessible technology can be used by those who are injured or disabled. For example, it could be used to help the communication of those with arm/hand injuries as well as the deaf.
What it does
Penpal is a web application that helps people who are injured and disabled spell. Penpal utilizes an Arduino Uno, accelerometer, gyroscope, force sensor, and software to write letters. The user wears a glove with the hardware attached to it and writes with his/her finger while putting pressure on the force sensor. Then, the hardware and software work to accurately predict the letter written using a trained Neural Network on the EMNIST dataset and displays it on the frontend React/Next.js web application.
How we built it
Using the measurements derived from the hardware, we utilize Riemann sums to approximate the position of the finger in space. We then utilize OpenCV to preprocess the data and draw the air hand-drawn letters to a pixelated grid.
We employ WebSockets in Socket.io to connect our Flask backend and React.js/Next.js frontend in order to get real-time updates in statistics.
Challenges we ran into
In the beginning, we initially planned to utilize a microcontroller that included bluetooth access so that we could make API requests within the Arduino IDE. However, we were unable to connect the hardware accurately to display the data measurements, so we transitioned to attempting to use a Raspberry Pi.
However, we also had issues with the Raspberry Pi with what we believe to be password hashing issues while trying to SSH into the Raspberry Pi. By that time, there were no more Bluetooth modules left, and so we were left with using an Arduino Uno along with using the Serial monitor.
This proved to work very well, as we were able to read using Python and the PySerial library the gyroscope and accelerometer measurements from the serial monitor. However, that also came with numerous challenges which involved a lot of debugging until a mentor came to the rescue.
Also, we had issues with producing an accurate 2D representation of the hand written letters in OpenCV, which also came with a lot of trial and error.
Accomplishments that we're proud of
We are very proud to have never given up, even though it seemed hopeless in the beginning as we had trouble just connecting the hardware to the computer. Especially, we were proud of being able to read the serial monitor from Python. Although it seems simple now, it was very difficult to debug because we did not know that we could not have 2 simultaneous programs reading the output at the same time.
What we learned
Every one of us have learned a lot from this hackathon. Some of us have very little hardware experience and thus learned to learn the tools and skills necessary to integrate hardware and software. A great example of this is debugging the Python reading of the serial monitor. Some of us did not have much frontend experience, so it was nice to learn how to integrate backend and hardware with a frontend that displays real-time data with websockets. We all learned valuable debugging skills and how to work effectively in a team in a time-pressure situation.
What's next for Penpal
We hope to increase the accuracy of the hand measurements of letters as there is definitely room for improvement. Also, we could add user authentication to store personalized data for different users. These are some of the ideas that come to mind, but the possibilites are honestly endless.
Log in or sign up for Devpost to join the conversation.