SeniorShield

Inspiration

In the United States, over 12 million senior citizens (people over the age of 65) live in their homes alone and 4.3 million of these individuals face some sort of cognitive deficiency or limitation. And these numbers most definitely aren’t stagnant. The number of senior citizens is expected to more than double by 2060, and with it, the number of them living alone with cognitive issues. A serious issue exists because of this: cognitively limited individuals, typically senior citizens, oftentimes have difficulty identifying visitors who come to their homes unannounced. Cognitive decline, something that becomes more and more pronounced as you age due to a decrease in executive control of the prefrontal cortex, affects your working memory, causing seniors to struggle to remember people who have visited their homes, and sometimes open the door for complete strangers. This could worry caregivers who truly don’t know whom the seniors might let into the house. There have been many horrible examples that really show the danger of this, ranging from theft to even, sadly, death. This issue is one that deeply impacts some of our own members and we felt that HackTJ would be a great opportunity to come together, use our innovative minds, and create something that could really address this problem that has been slowly getting out of control. This was the inspiration for SeniorShield.

What it does

SeniorShield is a mobile application that allows cognitively impaired individuals, mainly senior citizens, to stay safe within their homes. The mobile application connects to a Raspberry Pi device and uses facial recognition to alert the cognitively impaired individual whether the person outside is safe or is a stranger. First, via the mobile app, the user adds a certain amount of images of the face of safe, known people to a database that stores the images of the faces of those that are verified to see the cognitively impaired individual. When a camera that is attached to the Raspberry Pi device recognizes a face in the frame, the system attempts to match the face to an individual within the database. If the person outside the home is recognized as a verified individual, an alert is sent via the application to the cognitively impaired individual inside the home, telling them to let the outside person into the home, along with their name. If they are not recognized, then a different alert is sent, telling the person in the home not to let the outside person into the home. Our SeniorShield application ensures that those who are cognitively limited stay safe within the confounds of their own home, in an innovative and secure way.

How we built it

The creation of SeniorShield involved numerous steps: some short, some long, but all necessary. The first thing we focused on was the setup of the Raspberry Pi. We used a Raspberry Pi B+, a model that has significant limitations when it comes to, among other things, computing power. After a bit of struggling with the setup of a laptop display, we switched to an external monitor, and the process went a bit more smoothly. We then connected the Raspberry Pi to a PiCamera and, with that all configured, we began focusing on the facial recognition system using Python and several libraries, most particularly OpenCV. This process was tiring and involved a lot of research, but eventually, we were able to get an initial version of the system and uploaded it to the Raspberry Pi. Obviously, the system didn’t work off the bat and many additions and edits had to be made. After putting in some time, though, the system was able to recognize individuals and update the appropriate fields in our Firebase database. The mobile app has a listener attached to the data on Firebase and any updates to it trigger the app to reassess whether the name is a safe visitor. The UI updates according to a base state with no visitor, safe visitor, and unsafe visitor. Once we had completed all of that, we had a fully functioning application!

Challenges we ran into

Practically every step of the journey, we had some issues. None of us can really remember an instance in which a step was smooth-sailing from start to finish. That being said, here are some of the most difficult issues we ran into: A few challenges we ran into, especially fairly early on, involved the setup of the Raspberry Pi. We had initially planned to use a laptop display to work with the Raspberry OS, but after a great deal of struggle configuring the wifi settings, we decided to switch to an external monitor. Though the issue persisted here, after a ton of time searching StackOverflow and other different forums, we were able to solve the issue. Once we had dealt with the issues of actually setting up the Raspberry Pi, now came the really hard part: we had to create the facial recognition software and run it on the Raspberry Pi. The biggest issues of utilizing a Raspberry Pi were the limited computing power present and the pure longevity of installing each package. To combat this, we decided to employ lightweight Haar cascade classifiers which had a limited amount of packages needed to install. The downside of using a Haar cascade classifier combined with a relatively low-quality camera were the sensitive light settings needed to produce results. To resolve this, we put a lamp behind the camera to highlight the definitive features on the user’s face. Essentially, like everything in a hackathon, with enough time and manpower, we were able to get a facial recognition system that was genuinely able to work within the Raspberry Pi. The final big “stage” of issues involved the creation of the iOS app and the process of connecting the app with the Raspberry Pi and its facial recognition results. It was challenging figuring out how to structure the app in order to communicate with our Raspberry Pi. We first went through writing server-side Swift code using vapor, but the HTTP protocol was blocking requests to the app. Then we did substantial research and found that Firebase would allow us to connect the two platforms. This project was a tough one. It’s the most ambitious one most of our group members have ever worked on. But we honestly were glad for the opportunity and were excited to problem-solve (and Google search) our way through all the issues we had.

Accomplishments that we're proud of

We were extremely happy with basically everything we worked on during the event. From the start, we knew what we were working on was really ambitious and are excited by how everything turned out. Here were some of the bigger accomplishments that we had as we worked on SeniorShield: The Raspberry Pi was definitely the first thing we were glad to get working. After some struggles mentioned in the last section, we were able to get the Raspberry Pi OS to open up on our monitor, connect it to the wifi, and get the camera fully functional. This step was super motivating in getting the rest of the project done. The next thing we were glad to finish was the facial recognition system and getting it to work with the Raspberry Pi. The facial recognition system involved a great deal of research and debugging of code. Furthermore, we weren’t able to test it until we had the Raspberry Pi working, and once we did have it working, the installation of different libraries and packages took quite a while. And after that process, we still had to write a bit more code and fix some issues so that the system worked properly within the Raspberry Pi OS and functioned with the PiCamera. But at last, the system seemed to be functioning properly. The final big accomplishment we want to mention is the process of creating the iOS application and connecting it to the Raspberry Pi and the facial recognition system. We were able to take advantage of Firebase’s dynamic capabilities and communicate with a database from the Raspberry Pi. From there, the iOS app could pull from the database and automatically update the UI based on the pulled data. We had a ton of stuff we worked on and were pleased with how the app turned out.

What we learned

We learned a lot throughout the creation of SeniorShield. Raspberry Pi was not something that a lot of us had experience with so the process of experimenting with that was, though a bit painstaking, very informative and fulfilling. We also learned how to use Raspberry Pi with things like Python to use different features like PiCamera. Creating the facial recognition system meant we had to familiarize ourselves with libraries such as OpenCV and their AI-based tools and by the end, we felt much more comfortable with our ability to use these types of technologies within future programs. Lastly, we learned about building a multi-platform program and how to transfer data across these platforms. This project was an amazing learning experience for us and we will be sure to use the features we learned this year within our future projects.

What's next for SeniorShield

We are really excited about the potential applications and overall future for SeniorShield. We hope to expand our application to alert people other than the user indoors (such as caregivers and relatives) that someone has shown up to the home and who it is. We hope that by doing this, it becomes easier to keep track of who is entering the home, leading to the cognitively limited individual being safer. Another component we wish to have is the addition of a newer microcontroller, which would grant us much greater computing power and improve the capabilities of the facial recognition software. The final main feature we wish to add is a web application to go along with the mobile application. This would allow users multiple forms of connection to the application and make accessibility much easier. Of course, these are not the only plans we’ve had for SeniorShield. The possibilities are endless for SeniorShield and there’s always some idea we’ll be trying to make a reality.

Thanks! - Trishal, Shrey, Aryan, Atin