As frequent flyers, we often witness situations in which passengers unknowingly bring aboard carry-on items just as overhead storage is reaching its capacity. As a result, both the passenger and crew have to scramble to get the luggage back out of the plane and checked to the passenger's final destination. Not only does this cause the poor passenger some embarrassment but it can also lead to untimely departures and feelings of discontent among other passengers. However, this is neither the passenger's nor the gate agent's fault: there currently is no method or technology to help the gate agent estimate when he/she should start checking larger bags in at the gate.
What it does
As boarding starts, a Delta gate agent can use To minimize boarding time, we use image recognition to identify the types of luggage (bag, backpack, suitcase, etc.) each passenger possesses. Multiple camera streams can be used and processed in real time in order to accurately identify and account for each passenger's luggage. Once the type of luggage is known, an estimation of its dimension is used to calculate the amount of space it will take in the overhead compartments. As the overhead capacity fills up, the gate agent can monitor the capacity status on his/her monitor and make smart decisions about whether or not a bag should be checked in.
How we built it
We have a Flask web app in which a Delta gate agent can enter the flight number and see a progress bar of the overhead bin capacity. As the gate agent scans the boarding passenger's ticket, the scanned customer's info (name, seat, destination) will appear on the ticket agent's screen and the computer vision system (utilizing OpenCV and Microsoft's Cognitive Services Computer Vision API) will count and categorize and count the passenger's luggage in real-time. Our Flask back-end communicates back and forth with our front-end with Socket.io. In addition, multiple camera feeds are supported through the use of parallelized threads. We also used UDP message sending to send a "ticket scanned" message from the gate agent (web app) to the cameras and computer vision system (flask backend). We created a Rest API post method to pass the types of luggages back to the webpage for visualization.
Challenges we ran into
One of the biggest challenges was figuring out a reasonable method for identifying and categorizing the carry-on luggage. At first, we were going to use blob tracking and potentially background removal techniques order to define regions of interest and minimize noise. However, it soon became apparent that there were limitations Window's Cognitive Services Computer Vision API and that enhancing the pictures did little to boost the accuracy of the tags and descriptions generated by the API. We also had some trouble figuring out how to send messages back and forth between the flask socket and the computer vision system. Eventually we settled for using UDP message sending and a Rest API to transfer the data.
Accomplishments that we're proud of
We were able to run live streams of videos from our phones, onto the web and into our computer vision system for real-time analysis. We had a lot of fun wandering around the building and calling the Computer Vision API to describe all kinds of situations found at the Hackathon. In addition, our program is only bottlenecked by the Window's API call and doesn't add any additional time to the boarding process. If each ticket takes about 3 seconds to process and display on the gate agent's monitor, by the time the ticket information is ready, our program will have already categorized their luggage and made an appropriate suggestion to the gate agent regarding checking-in the passenger's luggage.
What's next for EasyGate
The next big step for EasyGate involves machine learning and running a k-means clustering algorithm to categorize harder and more complex luggages. While the Window's API is capable of amazing object detection and environmental descriptions, it is not able to differentiate between closely related items and it often over-tags the picture with unnecessary descriptions. Running a machine learning model on the incoming data would allow the system to accurately differentiate between small hand bags and bulky duffel bags and thus, a better estimate of the items size. In addition, figuring out the optimal way of positioning the bags in the overhead bins would also help cut down on unused space on the plane. similar situations and has trouble