Macrohard Teams

Homepage of site
Video call with 7 participants
7 person video call after zombie apocalypse

Inspiration

From the impressive, real time 3d tracking projects that came out from MediaPipe over the past few years, and VRChat (as an application), We landed on a "video"-calling SPA after an unsuccessful debugging session on making a first person character controller in threlte

What it does

Provides a realtime chat interface, with 3d model, webcam animated, avatars between people.

ALL APPLICATION DATA is routed via "text message" through talk.js (more precisely over wss through ConversationRef.set(custom: {...})) )

How we built it

For the database and API layer we used talkjs, hacking their conversation messages, to rapidly pass pose data between clients. For the fronted, I used my favorite framework svelte and tried out threlte as a threejs drop-in replacement for svelte. And finally, for the pose estimation, mediapipe's solutions provided a perfect foundation to approximate client poses.

Challenges we ran into

The primary challenge we had were threlte's documentation becoming moderately overwhelming for me. To the point of causing a pivot away from the original premise of a VRChat as I couldn't reliably get a first person character controller to work in our scene. Then secondly the teams unfamiliarity with the tech stack.

Accomplishments that we're proud of

It makes us laugh

What we learned

Sleep should have been aquired
A lack of a rate limit is fun

What's next for Macrohard Teams

With the structure and modularity of this project, we could go absolutely anywhere, from implementing custom gltf uploads through TalkJS's attachment handling to improving the functionality of the chat.