Inspiration
We all know how easy it is to miscommunicate. We have all had the experience of trying to explain something and just getting nowhere. This is always harder when you're separated by a screen. XpertView was our solution to this, an attempt to bring experience closer to you.
What it does
A real-time visual coaching platform that bridges students and mentors across distances. It allows a coach to draw directly onto the live video feed from a student’s AR glasses, so the student can instantly see the coach’s visual guidance as an overlay in their view. Both can speak and interact live, creating a shared visual and auditory training space.
How we built it
WebSockets to handle connections and signalling protocol. Includes ICE negotiation, and a TURN fallback server sourced from Twilio. A custom WebRTC implementation for real-time communication of audio, video and annotation data flowing peer-to-peer. Uses a deep learning speech recognition model for dynamic caption generation.
Challenges we ran into
A variety of computer vision and augmented reality experiments and caption generation proved much more tricky than we anticipated!
Accomplishments that we're proud of
Live video, audio, and annotation streaming between devices.
What we learned
WebSockets and WebRTC. Computer vision techniques including object detection and tracking. As always, we have improved our debugging skills.
What's next for XpertView
A move to Meta Glasses or VR headsets.
Built With
- assembly-ai
- computer-vision
- deep-learning
- javascript
- python
Log in or sign up for Devpost to join the conversation.