We built a virtual training platform with live IoT visualization and simulations via a Digital Twin.
We were inspired by UA's idea to provide an immersive experience for skilled trades training, and the potential for improving facility management by using reality capture.
We want to simplify data management by centralizing access to IoT data, service records data for training and maintenance, as well as student knowledge - for reality capture on Matterport, 3D models on Forge and standalone 3D files etc.
What it does
Our virtual training center recreates in person training and monitoring through rich interactive content in a digital twin, live IoT data visualization and simulations.
We combine , Matterport reality capture with sensor data (Fluke and other data sources), 3D objects, quizzes and educational content.
Admins or instructors can:
- link their matterport scans
- upload any gltf/glb files
- link BIM360 / forge (not implemented)
Admins can then:
- annotate the scans with rich content, embed videos, links, streaming video, etc.
- create virtual manuals,
- set up IoT sensors and data sources
- run simulations on IoT data, play and pause for teachable moments
- insert 3D objects to communicate complex information
- Authorized viewers or students can access the virtual training center via a web app and experience the digital twin in VR.
This prototype uses an existing UA mobile training center. The end goal is to network 300 training centers into a virtual super classroom that provides training on any type of system relating to the plumbing and pipefitting trades.
This solution is extensible to other use cases that can benefit from reality capture, IoT, data visualization and training. Examples include: facilities management and other skilled trades.
How we built it
We built a React web app to view and manage Reality capture files and associated IoT data.
It begins with OAuth2 for Admin and Viewer access. On both Admin and Viewer dashboards, users have the options to view their matterport scans, or upload any gltf/ glb files - which we show in A-frame.
Admins can insert 3D objects in both viewers, implemented with A-frame and the Matterport SDK.
The application runs on Google cloud compute, uses a realtime database (Firebase) to store IoT sensor data, virtual manuals, educational content and quizzes for students.
We are using both a simple graph library to simulate live IoT data visualization from historical data (CSV files).
For all other data sources, we used Grafana (open standards) for IoT data visualization of big data (this prototype only used public datasets).
We tried to add the app to a Knox container for additional security, but couldn't get approved for the partner program in time.
Challenges I ran into
There was no API for for Fluke sensor data and the data was exported as excel sheets. We loop through xls data to simulate a live scenario. Matterport scan when exported as an .obj and converted to gltf/ glb lost a fair amount of resolution -- limiting our usage of a-frame Applied for the Knox platform for biometric authentication 3 days before the deadline and didn't get approved in time. Definitely tried to add a lot of features. We are primarily backend engineers so front end work was a bit slow and caused some delays and limited fidelity of the final solution. Being new to web 3D, it took us some time to come up with an idea, and it took finding UA and understanding their needs to get going.
Accomplishments that we are proud of
Our team was very new to web 3D until 3 weeks ago and we were proud to have picked up the technologies, industry needs and direction in time for the hackathon. Support for multiple reality capture formats Works in web and VR Integrated Big data visualization libraries to help make sense of IoT data Ability to visualize historical data Ability to include educational content
What we learned
3D web is learnable and fascinating. Mashup approach is a fast path to generating value
What's next for Twin Peeks
Knox and other secure containers -Biometric and badge authentication (RFID/QR)
Access controlled areas in the scan
Visualize IoT data in the 3D scene with heatmaps
Work with UA to round out the educational/training platform
- LMS solution
Understand the needs and infrastructure of niches like facilities management
Focus on providing a simple solution to expand adoption of digital twins
Peter, Arpita, Dane