Contextere Branded Header Image

Inspiration

At Contextere we are transforming the future of work using AI to deliver actionable intelligence to the last tactical mile. We empower manufacturing, maintenance, repair, and overhaul personnel to make better decisions and do their jobs faster, safer, and more effectively.

Front line industrial technicians and operators do not have timely, contextual access to the information they need to perform their jobs. As a result, frequently incorrect or incompletely performed tasks lead to waste, inefficiencies and frequent re-work.

To address this problem, Contextere has developed an insight engine and Advanced Virtual Assistant (AVA) that extracts, structures and fuses previously inaccessible industrial data and equipment sensor information and uses machine learning (ML) and natural language processing (NLP) to determine and provide immediate, contextually relevant insights for analysts, and real-time AI-enabled work instruction for technicians.

Over 60% of an industrial worker’s day is characterized as non-productive time (Deloitte Insights); the Contextere AVA increases productivity by 30% and virtually eliminates rework by providing the right insight at the right time. The converging forces of a global demographic shift and the digitization of work have placed the focus on unprecedented skills gaps - OECD estimates 1.1B jobs will be radically transformed by technology in the next decade. The Contextere AVA provides companies with the opportunity to put AI where its need most – in the hands of its industrial workers to improve their knowledge, skills, and performance.

COVID-19 has created a unique opportunity and urgency to accelerate productivity, skills development, and digital transformation. The workforce requires access to the information from across the enterprise, from the edge to the centre, when they need it and where they need it. The Contextere AVA provides the opportunity to close the digital gap with a high return on investment.

'what it does' section header image - picture of a phone on a table surrounded with tools

What it does

The Contextere Advanced Virtual Assistant (AVA) combines automated data extraction with an ML-based insight engine, using NLP and neural networks to extract meaning from industrial enterprise data and determine the appropriate contextually relevant micro-guidance or insights. The Contextere AVA is an ML question-answering recommendation platform designed specifically for industrial technicians and operators; it:

  • automatically extracts information, meaning, and semantic relationships from structured, unstructured, and live industrial enterprise data including manuals, work orders, forms, and equipment sensor data.

  • understands context - user location and role, and equipment location/status - to provide intelligent guidance and curated insights to industrial users.

  • uses a natural language conversational interface to provide users with augmented analytics insights and AI-enabled work instructions on desktop, mobile and wearable devices.

The Contextere AVA is designed to be adapted to industrial data from any domain and the ML algorithms are typically retrained and tested on customer or domain-specific data as part of the deployment process. The Contextere AVA ‘bot app’ for Microsoft Teams is a standardized front-end app experience that enables the user to ask questions and receive answers in real-time using the Microsoft Teams chat environment. A technical question entered into the Contextere AVA bot app connects to the Contextere insight engine inference service running as a dedicated tenant on Azure to find the most relevant answer from the domain information available within that tenant. The inference service returns the relevant answer to the bot app and displays the specific answer in the chat, together with any additional contextually relevant information or metadata that can enhance understanding (such as figures, tables, or files) and delivers this content to user is the form of media rich adaptive card.

The Contextere AVA submitted to the Teams Challenge uses demonstration data from the industrial HVAC domain including extracted text, figures, and images from technical manuals, service bulletins, user submitted notes, and domain-specific documentation. In this live use case, the Contextere AVA would provide commercial HVAC technicians with information to improve their understanding while reducing cognitive load when operating and maintaining complex HVAC systems.

'how we built it' section header image - picture of an engine room

How we built it

Contextere is unique in its application of ML within an insight engine that proactively and predictively delivers curated guidance to a technician or analyst in an industrial setting based on their evolving local real-time context and interests. The focus of our algorithms is to determine and deliver just the right piece of information – a reductionist approach to curating the vast amount of available enterprise data.

The Contextere AVA uses an innovative combination of NLP, neural networks, and ML question-answering (Q-A) algorithms to extract meaning from structured and unstructured industrial data, match that to the meaning determined by a user question, and identify the appropriate micro-guidance or insight relevant to the user and work requirement. The user interacts with the Contextere AVA through a conversational user interface on a range of mobile and wearable technologies and via Microsoft Teams.

Since relevant last mile data such as manuals, work instructions, and maintenance records exists largely in PDF, Excel forms, and handwritten notes, automated techniques were developed and optimized to extract text, tables, and figures from those formats. This extraction and fusion process can process a data file in less than 0.5s with over 95% accuracy using a single CPU. The extraction process has been parallelized to be scaled across multiple CPUs using cloud resources.

The Contextere NLP pipeline uses customized word embedding and named entity recognition techniques to extract information (including text, images, and tables) from that structured and unstructured technical data into a digital format that can be used by ML algorithms for Q-A recommendation training and inference. NLP techniques were refined to extract meaning from the technical text and establish semantic relationships with associated information within files and across data artifacts.

Industrial use case training datasets consisting of hundreds of industrial manuals were used to define, train, test, and optimize the Contextere ML inference capability using NLP and transformer neural networks. Development and testing focused on the trade-off of inference speed, accuracy, and model size. The Contextere AVA returns answers to users in less than 0.5s with greater than 90% accuracy.

ML performance has been optimized for small, highly technical datasets to mirror the incremental and iterative data access process anticipated in our target customers. Together, this dynamic metadata and information structure enables the Contextere AVA to identify primary answers and supporting correlated information with high relevancy to users in domains with highly technical information and terminology. The Contextere AVA has also been refined to provide inference directly on modern mobile phones when cloud services are not available.

'challenges we ran into' section header image - picture of workers' hands pointing at a tablet

Challenges we ran into

Data environments of large industrial organizations are very complex and relevant last mile industrial data such as manuals, work instructions, and maintenance records exists largely in PDF documents, Excel forms, and handwritten notes. As a result, Contextere developed and optimized automated techniques to extract text, tables, and figures from those formats. This extraction and fusion process can process a data file in less than 0.5s with over 95% accuracy using a single CPU. The extraction process has been parallelized to be scaled across multiple CPUs using Azure cloud resources.

The difficulties in extracting relevant information from these dense data environments is compounded by domain-specific language. Existing NLP and Q-A algorithms pre-trained on a corpus of standard language such as Wikipedia frequently failed to understand important and relevant details of technical industrial terminology. ML inference techniques were refined to extract meaning from the text and establish semantic relationships with associated information within files and across data artifacts.

Unlike traditional deep learning techniques that require vast amounts of training data, Contextere has developed a unique neural network and transformer-based ML training and inference implementation optimized for smaller domain-specific technical datasets and emergent data that reflects the incremental and iterative data access process anticipated in our target customers. Together, this dynamic metadata and information structure enables the Contextere AVA to identify primary answers and supporting correlated information with high relevancy to users in domains with highly technical information and terminology. The Contextere AVA has also been refined to provide inference directly on modern mobile phones when cloud services are not available.

The evolving nature of Microsoft Teams presented a number of challenges adapting our user experience to the Teams capability. We were unable to provide initial support for a variety of rich and dynamic media interaction types that would increase the overall contextual awareness and understanding of the user. However, we expect the underlying support for these capabilities to evolve within Teams and anticipate being able to continue to enhance the app functionality in future releases.

'accomplishments we're proud of' section header image - picture of a group on industrial workers

Accomplishments that we're proud of

Contextere has successfully developed a data extraction pipeline that enables industrial companies to extract new value from their data and improve the performance, safety, and knowledge development of technicians and operators on the job.

Contextere has been profiled in Forbes, selected as a Gartner 2020 Cool Vendor in the Digital Workplace and chosen by the World Economic Forum as a member of the WEF Global Innovator Community of the world’s most promising start-ups and scale-ups.

Contextere is the first company to use ML to automatically determine, extract, and curate contextually relevant technical guidance and information semantically correlated from enterprise and equipment sensor datasets for a technical workforce. As Gartner noted in their Cool Vendors for the Digital Workplace report (May 2020), Contextere is “focused on augmented knowledge which is deeper and more substantial than augmented reality”. Our machine learning algorithms are optimized for small datasets with technical domain-specific language and constantly determine and deliver curated guidance and insights on a variety of user device types.

Contextere is also the first company to optimize technical ML Q-A inference for the CPU, memory, and user interface constraints of the smartphone. Our inference engine can function directly on the edge-based mobile device or in the cloud. In either implementation, the Contextere AVA returns answers to users in less than 0.5s with greater than 90% accuracy. And now, we’re leveraging the collaborative power of Teams to bring that capability directly within the corporate workflows people already use.

'what we learned' section header image - picture of coloured ink in water

What we learned

NLP and ML are not one-size-fits-all solutions and extensive effort is necessary to build or adapt models for highly technical domain-specific language. Existing pre-trained models perform well on a general language corpus but do not return good results on the types of industrial data that may be extracted from operations and maintenance manuals, work orders, and form-based information tracking and capture. While this represents a challenge, it also provides Contextere with a tremendous opportunity to develop unique intellectual property that provides value to our customers. In the process of developing this innovative solution, Contextere developed new optimization, training, re-training, and model reduction techniques that work well with the available data and provide excellent performance in the targeted use cases.

Traditional deep learning techniques require vast amounts of training data to increase model performance and accuracy. Our typical customer domain datasets are much smaller and are often made available incrementally over time; in short, the dataset ‘emerges’. Testing existing pre-trained ML models on our datasets resulted in sub-optimal accuracy, requiring Contextere to develop a unique neural network and transformer-based ML training and inference implementation optimized for smaller domain-specific technical datasets and emergent data. This solution enables Q-A recommendations for technical equipment domains and is optimized to rapidly return consistently correct answers to a user’s technical questions and real-time recommendations.

'what''s next for contextere insight engine and AVA' section header image - picture of timelapse train

What's next for Contextere Insight Engine and AVA

The next step of the evolution of the Contextere insight engine and AVA will focus on (1) the development of automatic domain ontology creation techniques to streamline the capture of semantic relationships across data; and (2) the integration of real-time equipment sensor data as part of the context determination capability.

Contextere will develop techniques to automatically extract and build specialized domain ontologies that enable semantic correlation and fusion of extracted data. Valid ontologies are important for establishing and managing relationships between files and objects based on noun phrase metadata extracted from within files. These relationships are, in turn, important for identifying new, contextually relevant insights across the available data. While ontologies can be defined manually, this is an intensive process that does not scale and may not encompass new emerging information from within the data. Automation techniques are critical to removing these constraints.

The current and historical operating status of equipment is an important determinant of a user’s human-machine context when conducting maintenance or operations. Integration of that dynamic real-time equipment sensor data into the Contextere insight engine, using that data to refine the context of a user question or to drive a new insight, and delivering sensor data to the user through the integrated AVA user experience are all critical next steps of our product evolution. This activity will include 5G IOT integration as well as design, training, and integration of ML algorithms that support streaming data as well as the data curation process to identify relevant ‘small data’ from within the broader real-time data stream.

Built With

Share this project:

Updates