Arduino Hardware Wiring Diagram
Constructed Arduino Hardware Circuit
Mobile App Signup Page
Mobile App Bus Line Without Filters
Mobile App Seating Chart with Recommendation
Mobile App RFID Setup Page
Mask Detection Analysis (High Confidence True)
Mask Detection Analysis (High Confidence False)
Audio Spectrogram for Cough Detection
Admin Website View + Add Bus Lines
Public transportation is a necessity to society. However, with the rapid spread of COVID-19 through crowded areas, especially in lines like city metros and busses, public transportation and travel have taken a massive hit. In fact, since the beginning of the pandemic, it is estimated that usage of public transportation has dropped between 70-80%. We set out to create a project that would not only make public transportation safer and more informed, but also directly reduce the threat of disease transmission through public transportation, thus restoring confidence in safe public transportation.
What it does
SafeTravels improves safety in public transportation by enabling users to see the aggregated risk score associated with each transportation line and optimize their seating to minimize the risk of disease transfer. A unique RFID tag is tied to each user and is used to scan users into a seat and transportation line. By linking previous user history on other transportation rides, we can calculate the overall user risk and subsequently predict the transportation line risk. Based on this data, our software can recommend the safest times to travel. Furthermore, based on seating arrangements and user data, a euclidean based algorithm is utilized to calculate the safest seat to sit in within the transportation vehicle. Video analysis for mask detection and audio analysis for cough detection are also used to contribute to overall risk scores.
How we built it
A mobile app was created with Flutter using the Dart programming language. Users begin by signing up or logging in and linking their RFID tag to their account. Users are able to view public transportation schedules optimized for safety risk analysis. Seat recommendations are given within each ride based on the seat with the lowest disease transfer risk. All user and transportation data is encrypted with industry-level BCrypt protocol and transferred through a secure backend server.
The Hardware was created with Arduino and programmed in C++. An MFRC522 RFID reader is used to scan user RFID tags. An ESP8266 WiFi module is utilized to cross reference the RFID tag with user IDs to fill seat charts and update risk scores for transportation lines and users. If a user does not scan an RFID tag, an ultrasonic sensor is used to update the attendance without linking the specific user information. Get requests are made with the server to securely communicate data and receive the success status to display as feedback to the user.
Video Analysis (Mask Detection)
Video analysis is conducted at the end of every vehicle route by taking a picture of the inside and running it through a modified Mobile Net network. Our system uses OpenCV and Tensorflow to first use the Res10 net to detect faces and create a bounding box around the face that is then fed into our modified and trained Mobile Net network to output 2 classes, whether something is a mask or not a mask. The number of masks are counted and sent back to the server, which also triggers the recalculating of risks for all users
Audio Analysis (Cough Detection)
We also conduct constant local audio analysis of the bus to detect coughs and count them as another data point into our risk calculation for that ride. Our audio analysis works by splitting each audio sample into windows, conducting STFT or Short Time Fourier Transform on that to create a 2D spectrogram of size 64 x 16. This is then fed into a custom convolutional neural network created with Tensorflow that calculates the probability of a cough (using the sigmoid activator). We pulled audio and trimmed it from Youtube according to the Google AudioSet, by getting audio labeled with cough and audio labeled as speech and background noise as non_cough. We also implemented silence detection using the root mean square of the audio and a threshold to filter out silence and noise. This works in realtime and automatically increments the number on the server for each cough so the data is ready when the server recalculates risk.
The backend was created with Node.js hosted on Amazon Web Services. The backend handles POST and GET requests from the app, hardware, and Raspberry Pi to enable full functionality and integrate each system component with one another for data transfer. All sensitive data is encrypted with BCrypt and stored on Google Firebase.
A novel algorithm was developed to predict the risk associated with each transportation line and user. Transportation line risk aggregates each rider’s risk, mask percentage, and the duration multiplied by a standard figure for transmission. User risk uses the number of rides and risk of each ride within the last 14 days. Because transportation line risk and user risk are connection, they create a conditional probability tree (Markov chain) that continually updates with each ride
Optimal Transportation Line and Seat
After the risk is calculated for each transportation line and user, algorithms were developed to pinpoint the optimal line/seat to minimize disease transmission risk. For optimal transportation lines, the lowest risk score for lines within user filters is highlighted. For optimal seat, the euclidean distance between other riders and their associated risk levels is summed for each empty seat, yielding the seat with the optimal score
Challenges we ran into
One challenge that we ran into when doing the audio analysis was generating the correct size of spectrogram for input into the first layer of the neural network as well as experimenting with the correct window size and first layer size to determine the best accuracy. We also ran into problems when connecting our hardware to the server through http requests. Once the RFID tag could be read using the MFRC522 reader, we needed to transfer the tag id to the server to cross reference with the user id. Connecting to a WiFi network, connecting to the server, and sending the request was challenging, but we eventually figured out the libraries to use and timing sequence to successfully send a request and parse the response.
Accomplishments that we're proud of
Within the 24 hour time period, we programmed over 3000 total lines of code and achieved full functionality in all components of the system. We are especially proud that we were able to complete the video/audio analysis for mask and cough detection. We implemented various machine learning models and analysis frameworks in python to analyze images and audio samples. We were also able to find and train the model on large data sets, yielding an accuracy of over 70%, a figure that can definitely increase with a larger data set. Lastly, we are also proud that we were able to integrate 5 distinct components of the system with one another through a central server despite working remotely with one another.
What we learned
One skill we really learned was how to work well as a team despite being apart. We all have experience working together in person at hackathons, but working apart was challenging, especially when we are working on so many distinct components and tying them together. We also learned how to implement machine learning and neural network models for video and audio analysis. While we specifically looked for masks and coughs, we can edit the code and train with different data sets to accomplish other tasks.
What's next for SafeTravels
We hope to touch up on our hardware design, improve our user experience, and strengthen our algorithms to the point where SafeTravels is commercially viable. While the core functionalities are fully functional, we still have work to do until it can be used by the public. However, we feel that SafeTravels can have massive implications in society today, especially during these challenging times. We hope to make an impact with our software and help people who truly need it.