Mr. Narayan's Lab Assistant

As Mr. Narayan ages, we want to help him stay safe and productive in the workplace.

Comment

Inspiration

My grandfather, Ram Narayan, moved to this country, with a couple of dollars in his pocket so that he could complete his PhD, and get his family out of poverty. While here, he worked 3 jobs amidst his PhD so that he had enough food for his wife and himself. Since then, he has sacrificed everything for our family, and he continues to work so that we are comfortable. After all he has done for us, we wanted to do something for him. He has been passionate for Electrical and mechanical engineering and that is where he currently works, but as his health worsens, things are getting more and more difficult. So, we took it upon ourselves to create something that helped him continue his passion.

The first thing we did was ask him about his health problems, and the kinds of things he would like to see in a device. A couple of things became apparent after this call. As he ages, he feels that he is becoming more and more forgetful. He sometimes forgets where he puts tools and ends up wasting time looking for where he put them. In the past year he has fallen once, and he fears that he will fall again, when no one is around in the office. In the past couple of years, he has had health episodes at the office, and because he did not have his phone on him, he had to drive home by himself before getting a ride to the hospital from his wife.

We also asked him what kind of things are important for such a device. Here is what he said. If data is collected at the office, it must stay confidential. Because he works on government confidential information, things must stay secure. He uses an android, so the user interface cannot be an Apple app. It must be android compatible. He wants his wife to know about any episodes that he has, he doesn't want just emergency services to be called

Our Solution

We decided to build Mr. Narayan, a personal Lab assistant to promote greater independence. This would allow him to continue to work at the office, safely and productively. We incorporated 3 main systems that achieve this goal. The Heart Rate Sensor: This sensor emits Infrared and detects how much is reflected. By sensing changes in the IR value (which correspond to pulse), a HR can be achieved. Because Mr. Narayan has no reason for his heart rate to get really high unless a health episode is occurring, we set a threshold. When this Value exceeds a certain threshold, a warning is set off to alert both his Wife and the user interface. We used the MAX30102EFD module. 6 axis Gyro and Accelerometer: Sudden changes in accelerometer data and Gyroscope data can be used to detect falls. Using the magnitude of all 3 acceleration vectors, a threshold can be set that when surpassed, indicates a fall. We used the MPU6050 module. Lastly the onboard Camera (an ESP32-CAM) uses Computer vision algorithms to detect when objects have been placed down. If an object is placed where it is not supposed to, the algorithm detects this and saves the image into a database. This way instead of having to search the office, he just needs to search a couple images to see where the object was placed.

How we built it

We started by prototyping our hardware on a breadboard using modules we had on hand. We then turned our focus our attention onto the software to read basic sensor data and transmit this to our user interface. Once our initial prototype was complete, we started putting everything into a smaller form factor that could actually be used by Mr. Narayan. During this phase, we had to figure out a viable power delivery system using LiPo batteries. We chose to go with a pendant, and a bicep mounted heart rate monitor. Using a 3D printed case we arranged all the modules within the pendant and soldered all the electrical connections. After the entire pendant was finished, we then tested the full assembly.

Challenges we ran into

We initially thought of using a machine learning model, trained by images of the hand. However, it was very hard to find data, and the training model often overfitted due to the low quantity. We tried to use videos instead of photos to get a more accurate depiction of each action, but we simply did not have enough data to create an accurate model. We transitioned to CV software, which was already pre-trained on human hand movements, and it showed far more promising results, though a big challenge was using the best proxy to determine what the hand would do when placing down an object. We eventually found that a curled-up hand often meant grabbing while a straight hand meant that no object was being held. If an object is released, an object drop is detected.

Accomplishments that we're proud of

We’re proud of being able to get a reliable method of detecting when an object is placed, despite our initial difficulties, and coming up with an optimized method to figure out what proxy to use for the release. After thinking of various methods such as image-based machine learning or using the acceleration of the hand, we are proud of our final solution of monitoring hand releases. We are also proud of getting a website that can show results from the accelerometer, gyro sensor, heart rate monitor, and camera live, all in one place, which could really help Mr. Narayan be able to see the outputs from the wearable all in one place.

What we learned

Initially, we believed that machine learning was simpler and was a ‘dream solution’. We believed that we just needed to feed the neural network images, and it would return the correct result like a ‘black box’. However, the ambiguity and difficulty of understanding what is happening from a simple image alone showed us that a ML solution would not always be the best. We used what we thought was a ‘simpler route’ via the CV and pre-trained models, but it proved to be more useful due to how it was specifically optimized for our use-case and to analyze the motion of a human hand.

What's next for Mr. Narayan's Lab Assistant

Despite producing a working prototype, we still have bugs to fix and improvements to make. To improve the detection for putting down an object, we plan to observe the object being placed itself using another custom library and functions (similar to mediapipe), which could process if the object was actually placed. This would allow us to be clear that the person is holding on to something before stating that a drop was made. Additionally, we want to fix the issue that an offscreen hand is not well detected and then immediately renders once back on camera, which would help avoid false positives of detection. Overall, we hope to turn Mr. Narayan’s Lab Assistant into a more optimized wearable that can help people function very well in their day-to-day life.

Furthermore, such a device can be expanded to promote independence in older individuals, across a wide range of environments.

Built With

Share this project:

Updates

Submission history