Across several industries logging of maintenance actions is a laborious and time consuming process. However logging of data generates significant amounts of unstructured and structured data which plays a vital role in enabling systems to have predictive capabilities in the maintenance pipeline. Our goal was to capture this data most efficiently leveraging existing standard operating procedures and AI technologies.

What it does

SCRIBE stands for Systematic Contextual Recorder with Intelligent Blending of Elements. In short, it is an AI video and audio assistant for maintainers. The system eliminates the need for maintainters to fill out forms by recording them as they work, and populating a database with this information. The system will save time for maintainers as well as improve the data quality they provide.

SCRIBE is only active when a user says the keyword, “SCRIBE”, after which it will record both audio and video for a preset amount of time and then turn off. Putting the maintainers in control provides privacy and also allows the system to only see and store the most important information. A typical session may start with the user saying “SCRIBE, begin procedure 101.” Using standard operating procedures and error codes, our system has great context of what it will see in both the audio and video feeds to follow. For most of the procedure, a user will likely not talk to the system, especially when everything is going as planned. However, if an anomaly occurs, this can be easily recorded with SCRIBE as it is happening. Finally, after the work is done, a user can say “SCRIBE, end procedure 101.” At this point, our system will perform multi-modal data fusion to populate a maintainer action database as well determine a summary of the work done. SCRIBE will relay the summary verbally to the maintainer, who can either affirm the maintenance log is correct, or provide corrections to the system. These corrections allow our system to learn over time, in order to improve model accuracy and technicians trust in the system.

How we built it

The goal was to develop the concept of SCRIBE within a day, however, we wanted to keep our concept grounded to existing technologies. Hence, we based our concept on Amazon APIs for voice and image recognition. And plan to use them in future for prototype development.

Challenges we ran into

The challenge was to consolidate all the feedback obtained from the mentors and propose a feasible concept.

Accomplishments that we're proud of

Learning the existing challenges faced by maintainers and their process needs and providing a solution fitting to their requirements. Also , we were one of three winners of the Into The Dataverse Hackathon and are very excited to be able to pursue our idea further with the $15,000 prize.

What's next for SCRIBE

We are planning on refining our concept and work for a minimal viable product.

Share this project: