Inspiration
We were inspired to create this project by the problem of sanitation for flight passengers. However, the project quickly evolved into a vision of comfort and luxury for airline passengers. We decided to utilise bleeding-edge technology to provide a unique experience that completely changes the way passengers interact inside of the plane.
What it does
Our project is an in-flight terminal system for American Airlines. It provides the full media experience of existing terminals, with a better UI, but it provides some unique improvements on the current paradigm. First, we use gesture controls rather than capacitive touch displays to control the device. These screens are often difficult to use, so the advanced gesture control feature helps make the screens more fun and accessible. Second, we provide an AI-powered assistant to help the passenger learn about their destination and prepare for travel. Third, we implemented a seat-checking system that can identify items left behind on the aircraft and assist flight crew in identifying and locating the lost items.
How we built it
The in-flight assistant is powered by OpenAI’s unreleased GPT-4-Turbo model, which combines inference speed with extremely high quality and reasoning ability. With some prompt engineering, we were able to create a chatbot that had an amazing ability to guide passengers in-flight.
Our image recognition software utilizes a web camera to provide images, which were then base 64 encoded and passed through the not yet publically available GPT-vision model, which is capable of distinguishing objects. We ensured the prompts worked around its weaknesses of location image processing by creating the proper query environments. This is how we detect what kinds of objects are present on the chair, and then websockets were utilized to pass this data to our Attendant page.
Our system utilizes custom gestures including pinching and closing your fist to interact with the inflight entertainment system. To enable these features, we brought back to life the long abandoned handsfree.js library. The advantage of our design is that all gesture recognition is privacy-preserving being done locally in the browser with minimal computational overhead for the features and experiences it enables.
Challenges we ran into
Our initial hardware plan for the camera was a raspberry pi, which unfortunately did not properly connect. We did pivot our plan into a web-camera, and learning how to ensure the camera quality fit the needs of image recognition was new.
The library we decided to use for gesture recognition was broken, so we had to entirely reimplement. We also weren’t able to locate the source code, so we had to reverse-engineer the library. This proved to be a challenge because we ended up having to reimplement many features of a browser that are taken for granted, such as scrolling and clicking.
Accomplishments that we're proud of
We are proud of our ability to tackle the enormous amount of features we wanted to put into the application. We also had lots of trouble traveling, so we were not able to rest very much. This wasted a lot of time and put us at a disadvantage since we arrived sleep deprived before writing a single line of code, but we were able to work well as a team and pull through in the end. We’re also proud that these ideas truly do keep the well-being of people in mind, as we always strive to blend humanity with technology in a way beneficial to all.
What we learned
Our team heavily modified a library for the gesture recognition software, and in the process learned the ins and outs of library creation, beyond using existing documentation code of libraries. We also learned a lot about UI/UX design as we had some clashing opinions to make the best interactive interface for the customer.
What's next for CloudCare
We want to create a more practical and compact design, including embedded cameras, that could be installed into a plane en-masse. This would allow large scale implementation of our system, since it is already scalable and well-architected.
Log in or sign up for Devpost to join the conversation.