Inspiration - Our idea was inspired by people in the neurodiverse and disabled community. Through our inclusive mission we collaborate and train people from this community. This project was birthed from conversations around inclusion.
What it does - The purpose of the app is to be a utility for people with a hearing impairment. It works through the Meta XR Passthrough and Convai voice recognition. These tools transcribe spoken words into visible text that can be seen in the environment near the individual speaking. The font size, colour and type can also be changed to ensure a comfortable and personalised viewing experience.
How we built it - We built the app by using the Unreal Engine 5 VR template as a base and removed all of the objects within the scene. The Meta XR passthrough layer was then added as a component to enable passthrough allowing the player's environment to be seen. We used blueprints to design the controls around a push-to-talk system, that allows the player more control in their environment. The text is controlled using Convai, a free AI LLM that allows transcription in real-time. We then push the output of the transcription into a UI widget that is rendered in the 3D space.
Challenges we ran into - As we started developing the project we encountered more and more software related problems. There's a lack of in depth documentation on using the tools and functions surrounding our theme. We were limited in our options and tested many different AI recognition and software that fit within the rules of the hack to get the core functionality needed.
Accomplishments that we're proud of - After several hours of trial and error getting the core functionality we finally overcome the biggest hurdle to get our core mechanic implemented. The custom sounds and images we used to polish our product as well as the dedication our team members made to make sure everything works.
What we learned - More confidence in general using Unreal Engine. Greater exposure using AI and Meta XR passthrough in projects. Greater depth of control scheme using Unreal Engine Blueprints.
What's next for See Hear - The intention for the app is to be a broad utility app not just for individuals that have partial or no hearing. We intend the app to be a general platform for vulnerable individuals or non-native speakers. Examples of this are using image recognition for the partially sighted, a calm meditation space for neurodiverse individuals in overwhelming situations or real-time translators for non-native speakers.
Log in or sign up for Devpost to join the conversation.