OpenDoAR

AR support for returning to physical spaces safely, affordably and effortlessly

Comment

Inspiration

The mission of OpenDoAR is to empower universities and small businesses in a safe return to physical spaces for their people. We aim to improve outcomes for our users in affordable health compliance and overall monitoring.

In this year and the next, the return to physical spaces for organisations and educational institutions are in progress.

For big companies, advanced and expensive methods are employed to ensure healthy employees are entering the office. These methods are not always affordable or convenient for smaller organisations.

Looking at educational institutions, there is an ambiguity in compliance to the return-to-campus system.

Currently, UC Berkeley uses a badge system for COVID health monitoring and safety. Everyday for a student on campus, a questionnaire is meant to be filled out via mobile which grants the student a badge status. E.g. Green badge granted (assuming student doesn’t have symptoms, are fully vaccinated, etc). There are yellow and red badges too from less ideal answers.

Before entering a lab or classroom, a Teaching Assistant (TA) is meant to check if the student has a green badge, and if not, entry is not granted. The effort from the student needing to pull out and display the green badge on their phone, and showing it to the TA at the door is higher than students are willing for. Hence, for over 90% of the time, this practice is ignored. This becomes more common as the reality and local-impact of COVID fades from people's minds (as infection rates drop).

We decided to use Face-Id now for our implementation since it is very commonplace with recent iPhones and simplifies ease of use. Within the last few years, it has become normal for people to look at their phone to unlock private information. Due it is simplicity of access, we hope to extend this to our AR application to allow for quick identification.

Core Features

  • AR based on text overlay and image query for mobile. Low effort scanning of students entering classroom.
  • Mobile Dashboard to analyze core statistics about number of green/red badges
  • User accounts and Authentication for multiple events and for authorized access
  • Face detection of students ONLY in the TA's class
  • Desktop Power BI Dashboard for higher management oversight on compliance and monitoring.

MobileApp Implementation

On the admin side, the camera aims towards the door of a classroom or space, and as students walk in, the app processes students’ faces against the pictures they provided to their organization (or uploaded via the app from their user side) and tags them as their respective colored badge (green for compliant, yellow for not having filled out a daily screening survey, and red for not compliant). The data is recorded for future reference on the compliance of the selected person. This data was kept in an anonymous fashion to prevent HIPAA issues.

WebApp Implementation

Primarily for use by admins, they can view data visualization of their attendees/students’ badge statuses and vaccination statuses and the proportion of badge statuses relative to the entire class/event to better plan future events with regards to health regulations and safety.

In order to support the backend, we created a flask server hosted on Azure. The face detection model uses dlib landmark dataset, which labels important features of the face and tries to detect similarity between faces with a 99% accuracy. The backend also supports users and groups and dynamically indexes sets of face databases into memory as needed. We also encrypt on transit and don’t store images that are processed to prevent issues with storage of possible personally identifiable information, breaking HIPPA policies.

For the website, we used React to build out the UI and NextJS for server-side rendering, and for the mobile app, we used Flutter and its ARKit framework to develop the frontend and the primary AR features of our solution.

Power BI was used to create a visualisation dashboard for higher management to understand their people's compliance and health in their return to physical spaces. An Azure Virtual machine with datascience configuration was used to build this.

Challenges we ran into

  • Initially implemented a trained keras model to predict face detection, but a raw dlib based face landmark detection worked better.
  • Implementing AR functionality and sending Image had a different byte encoding, causing issues when sending it over to the flask server for processing.
  • Struggled initially figuring out the scope, problem statement, and user cases for our idea that addressed a real problem and also targeted the primary categories of the hackathon.
  • Used Azure VM for first time to build on Power BI.
  • Publishing our Power BI dashboard to the the PBI service and then embedding it into our website was blocked as none of our members' school/work accounts permitted Power BI sign up. This is honestly a 5 min step if we gained permissions from our organisation accounts. File is attached in Github to allow running in PBI Desktop.

What's next for OpenDoAR

For events, the world is returning to physical events. And during this transition period, the need for easy tools for health and safety is critical to keep re-emergence chances of the pandemic low. We want to put an app into the hands of event organisers that easily allow them to check their attendees without making it take ages for people to get in.

A side pivot, in the context of networking events, our infrastructure could allow for people at networking events to figure out who to talk to. At many networking events, people try to find others with similar interests, but might end up talking to the wrong people. In order to simplify this process, we can do a quick scan of the people’s face to see if the interests align (show AR overlay of profession, interest, company). This also reduces the social awkwardness/time spent in talking with someone you realise you're not interested in. The faces of the people at the event can be integrated into the database as attendees are there to be public and meet people. In terms of actual work, there isn’t too much involved in extending it to the networking space. Additionally, pronouns are something we can overlay (like LinkedIn) to clear any ambiguity in a socially conscious society.

For small and medium sized companies while people are re-integrating to the workplace, an automatic system to detect compliance people entering the property could be difficult and expensive. Even with security guards, which could be expensive, they are limited by very manual checking methods. With our system, we can help employees in their re-integrate into their work, by simply downloading a new app to the phones security guards already have.

Additionally, we are taking UC Berkeley as a proving ground in a simple effective tool. We want to roll this out to other universities in the US and beyond. Making it the solution that enables smoother exchanges and visits from people outside the university.

Try it yourself at

https://github.com/vikranth22446/greenhelth

See our prototype

https://www.figma.com/proto/6k9jPCZPsMdHJeWmx7Jwlt/CalHacks-Wireframes?node-id=9%3A102&scaling=scale-down&page-id=0%3A1&starting-point-node-id=30%3A91

Website demo

https://drive.google.com/file/d/1Fctu4RkF0ecVfJsftB7z6G8HFwNnv0DI/view?usp=sharing http://opendoar.tech

Username: test@gmail.com Password: test

Built With

Share this project:

Updates