Intelligent Augmented Reality Guide for Yard Personnel

Inspiration

Understanding shunting personnel's challenges in vast, high-pressure environments drove our initiative. We discovered that workers face dangerous situations monthly, sometimes with tragic outcomes. Amidst the yard's noise, vital alerts like an approaching train can go unnoticed. Such incidents aren't just statistics; they are harrowing reminders of the challenges yard personnel face daily. This drove us to create an AR assistance system with GenAI, ensuring every move they make is safe, efficient, and informed.

What it does

The Intelligent Augmented Reality Guide for Yard Personnel, powered by GenAI, offers a multifaceted AR system. Through visual cues on HMDs and mobile devices, it provides real-time guidance for shunting operations. That means our system can be used in both AR glasses and AR-functional mobile phones. It shows the best way to reach the target destination with a colored safety system. Beyond guidance, the system doubles as an interactive assistant, addressing on-the-spot queries regarding the shunting yard's current status. Meanwhile, the integrated earphones ensure that workers receive instant auditory warnings in case of dangerous situations.

How we built it

Using AR technology and GenAI, we overlayed real-time data onto a virtual representation of the yard, created in Unreal Engine. In order to simulate realistic scenarios, the visual interface was developed in VR to represent AR, ensuring it's intuitive for users. Real-time data is coming from TrackOps Depot API, which helps match the tracks in the simulated environment to the real-life tracks. This system is also supported by Generative AI, which assists the yard personnel in daily work and maintenance tasks. For that purpose, we used Vertex AI supported by Google, and fine-tuned the pre-trained LLM model to create a chatbot that is specifically designed for our case.

Challenges we ran into

First and the most important challenge was not to sleep and we accomplished it. Other than that, coming up with a good idea was cumbersome in our case since we are not experienced in this field. Thus, we could not easily understand the problems that shunting yard personnel might face. Also, there were a wealth of data so choosing the right data for our case was a bit challenging. Technically, working on the different sets of technologies and balancing user experience with these technologies took most of our development time.

Accomplishments that we're proud of

Successfully creating an integrated AR system that not only guides but actively seeks to protect workers is our crowning achievement. Furthermore, our auditory warning mechanism has the potential to significantly reduce yard accidents, marking a pioneering step in shunting yard safety. The incorporation of cutting-edge tech, like Generative AI, in our design process further amplifies our sense of accomplishment.

What we learned

The importance of real-time data in high-pressure environments became clear. We also recognized the critical role of intuitive design in ensuring that technology truly aids its users. Above all, we learned that with the right tools, we could make a tangible difference in workers' safety and efficiency. Even though it looks unnecessary at first sight, it might create a massive impact.

What's next for Intelligent Augmented Reality Guide for Personnel

Our roadmap involves refining the system through feedback garnered from real-world applications, ensuring continuous improvement in intuitiveness and utility. There's potential for integrating more advanced AI analytics for predictive warnings and expanding our solution to other high-risk industrial environments, revolutionizing workplace safety globally. It can also help maintenance workers in their daily work. Maintenance personnel can greatly benefit from our system, with future iterations slated to integrate microphone capabilities, fostering seamless interaction between the yard staff and our GenAI-powered guide.

Built With

• blender
• c++
• data-science
• generative-ai
• http
• python
• socket.io
• unreal-engine
• vertex-ai