ImmuniPass

A Standardised and Digitized Immunization Verification System to make public areas safe, post COVID-19 immunization.

Comment 1

Integrated video submssion:

https://docs.google.com/document/d/1DSdIxdrZcJgPvmX8qjXTLevgPowt8BOWQYNQ2kuJLO8/edit?usp=sharing

Inspiration

The COVID-19 vaccines are estimated to arrive around one to two years from now, when the world will be completely shattered by the pandemic. When public areas and transportation systems start going back to normal, there will be complete chaos as the vaccines would have just started shipping. There would be no way to know who is immunized and who is not. To solve this problem, we propose ImmuniPass: A standardized digital verification system for identifying immunized individuals in transport systems such as buses, metro, cabs and even airplanes, and public areas such as malls, restaurants, airports etc.

What it does

Our mobile app enables immunized individuals to access public vehicles areas by verifying that they are immunized by a unique two-step process:

  • First, the owner or manager of a public area or vehicle (such as the cab driver or a restaurant owner) scans a unique QR code automatically generated by the app.
  • Next, they take a photo of the immunized person, and our backend server runs a secure face ID using AES encrypted, prior collected face data of the person. This ensures that the account represented by the QR code belongs to the same person.
  • This now registers this person into that vehicle/area and they can get notifications on the potential risks detected, as explained below.

We have developed a security surveillance system that uses security cameras, microphones, and other sensors to do the following:

  • Runs advanced camera vision models on real time video to detect whether the people in a certain area or vehicle are wearing face masks or not.
  • Runs machine learning models on real time audio data from microphones to detect if anyone is coughing in that area/vehicle.
  • Uses various sensors to detect air quality.
  • If any security risk is seen in a certain area/vehicle (for example if multiple instances of people coughing were detected in a bus), our backend server automatically sends a notification to alert all the people who are registered in that area/vehicle at the time, as well as the manager of the area (such as the bus driver) about the potential risks. This feature also comes with a system specifically for vehicles, where the user can scan an image of a license plate, and get the past data pertaining to that vehicle.

Lastly, we have developed a web app for doctors and hospital agencies, where verified immunization centers can register new people into the system, who were previously un-immunized, along with their face data, after giving them the vaccine. This web app also provides data visualization of the surveillance data for doctors to analyze trends. To make this system more user friendly, we have added a feature in our mobile app, by which un-immunized individuals can find nearby verified immunization centers, via the google maps API.

How we built it

In order to build our system, we used an assortment of technologies and platforms:

  • React native to build the frontend interface for both users and managers of public areas.
  • React js to build a web app for doctors and hospitals to register immunized people into the system
  • Auth0, which we integrated with our react native app and react web frontend, for login, registration, and authentication.
  • Flask Python for our backend server, mainly for integrating the front-end with the database.
  • MongoDB Atlas to store user data, hashed passwords, registered immunization centers, and environment data such as Air Quality
  • Python OpenCV + Keras for mask detection from video feeds.
  • Google’s teachable machine for cough detection, which takes into account background noise from real time audio data in a noisy public area or in traffic.
  • Dlib + OpenCV for facial recognition and SHA256 + AES encryption for secure storage of face data.
  • Vercel to deploy our react web app.
  • Raspberry pi with a webcam, microphone and air quality sensor to simulate a surveillance system.
  • Google Maps API for nearby immunization centers feature.
  • Microsoft Azure to store the encrypted image embeddings
  • AWS for the lambda functions and server/backend services
  • Digital Ocean to host the flask backend for the nearby immunization centers retrieval We store encrypted face embeddings (not face images) in order to comply with HIPAA and other privacy requirements

Relevance in transport

Public transport and vehicles provide an ideal medium for the spread of COVID-19, as there is close contact between people, in a closed area. Our system can be integrated with apps such as FaceDrive or HiRide to verify immunized individuals, and ensure safety of both customers and employees. This system can also be used to regulate social distancing practices such as wearing masks in vehicles.

Business model

This app is targeted towards:

  • Private Organizations such as FaceDrive, HiRide or Walmart - We will provide Immunipass as a service to such private companies. Our business model cost/benefit analysis is given below:

Cost

Companies will have an initial cost of the hardware (cameras, microphone etc.) required in each shop/vehicle, as well as an ongoing maintenance cost of this hardware for the surveillance system. Along with this, we will charge a small cost of around 10 cents on each transaction in the shop/ each ride in a vehicle, to fund our cloud servers and have a profit margin.

Benefit

Our system will provide an incentive for customers to go to the shop, or use the ride sharing app that uses Immunipass, as they are guaranteed further safety in these ‘immunized’ areas. This incentive will significantly increase with competition between companies to be the safest one. This makes Immunipass profitable for both us, and the company using it. We will also create a branding for ‘Immunipass certified’ so that companies can advertise their safety.

  • Government authorities - In this case, our system will be funded by the government and mandated by the law in every public area and vehicle.

Challenges we ran into

Our first great challenge was with integrating Auth0 with the expo front-end, due to lack of documentation specifically for expo-built react native apps. However, this problem was resolved by help from our auth0 mentors, as well as Auth0 forums. Our second challenge was in getting the mask detection model light enough to run on a raspberry pi. Thirdly, since some of our teammates had never worked with encryption, performing encryption and hashing of the facial images and embeddings was a challenge. Lastly, integrating the many parts of this complex system and making it work coherently was a challenge in and of itself.

Accomplishments that we're proud of

We are extremely proud to have created a very sophisticated system spanning multiple components, from hardware to a mobile application to a web application, in a short time. We are also proud to have successfully integrated Auth0, considering none of us had prior experience. Thirdly, in addition to creating 3 different clean user interfaces in react js and react native, we are proud to have designed and trained effective machine learning models using limited training data. Lastly, we are proud to have managed to coordinate and complete the hardware (raspberry pi) aspect while working remotely. We would like to highlight that we are extremely proud of designing and creating our system in a way that allows for the widespread adoption of face recognition technology with regards to privacy concerns. This posed a major challenge but we are proud to have found a solution. We don't store any face images on our database or in the cloud. Instead we calculate an embedded vector representation of a face and then encrypt this vector and store it. There is essentially no way to reconstruct a face from the vector, and it is also now encrypted. This allows us to comply with all privacy concerns and expand our system internationally.

What we learned

First and foremost, we learned how to use Auth0 for user authentication with MongoDB in our react native and react js apps. Furthermore, we learned how to use teachable-machine to create effective audio classification models. Given that our team members had limited experience, we learned how to encrypt data for secure storage, specifically facial data and facial embeddings. We also learned how to implement facial recognition, cough detection, and mask detection using python. Lastly, in addition to using Vercel for the first time to deploy our react web application, we learned how to integrate webcams, microphones, and air quality sensors with a raspberry pi, and integrate it into our entire system as a whole.

What's next for ImmuniPass

In the future, we would like to implement a custom user data system, integrating other medical health conditions, and, making use of MongoDB’s key value structure, in order to give custom notifications (for example giving more frequent and sensitive risk notifications to at-risk individuals such as senior citizens and people with respiratory disorders). We would also like to make a more sophisticated system to detect coughs, by estimating the position based on the readings from various microphones placed around the area. We are aware of a logical flaw in our system, that in order for face ID to work, it requires the user to remove their mask briefly to scan their face, which is not favorable. In the future, we would like to explore new and innovative methods of bio metric scanning such as iris scanning and IR imaging.

Share this project:

Updates